Connectionists: Best practices in model publication

[Brad Wyble]

Dear connectionists,

I wanted to get some feedback regarding some recent ideas concerning the
publication of models because I think that our current practices are
slowing down the progress of theory.  At present, at least in many
psychology journals, it is often expected that a computational modelling
paper includes experimental evidence in favor of  a small handful of its
own predictions.  While I am certainly in favor of  model testing, I have
come to the suspicion that the practice of including empirical validation
within the same paper as the initial model is problematic for several
reasons:

It encourages the creation only of predictions that are easy to test with
the techniques available to the modeller.

It strongly encourages a practice of running an experiment, designing a
model to fit those results, and then claiming this as a bona fide
prediction.

It encourages a practice of running a battery of experiments and reporting
only those that match the model's output.

It encourages the creation of predictions which cannot fail, and are
therefore less informative

It encourages a mindset that a model is a failure if all of its predictions
are not validated, when in fact we actually learn more from a failed
prediction than a successful one.

It makes it easier for experimentalists to ignore models, since such
modelling papers are "self contained".

I was thinking that, instead of the current practice, it should be
permissible and even encouraged that a modelling paper should not include
empirical validation, but instead include a broader array of predictions.
 Thus instead of 3 successfully tested predictions from the PI's own lab, a
model might include 10 untested predictions for a variety of different
experimental techniques. This practice will, I suspect, lead to the
development of bolder theories, stronger tests, and most importantly,
tighter ties between empiricists and theoreticians.

I am certainly not advocating that modellers shouldn't test their own
models, but rather that it should be permissible to publish a model without
testing it first. The testing paper could come later.

I also realize that this shift in publication expectations  wouldn't
prevent the problems described above, but it would at least not reward
them.

I also think that modellers should make a concerted effort to target
empirical journals to increase the visibility of models.  This effort
should coincide with a shift in writing style to make such models more
accessible to non modellers.

What do people think of this? If there is broad agreement, what would be
the best way to communicate this desire to journal editors?

Any advice welcome!

-Brad



-- 
Brad Wyble
Assistant Professor
Psychology Department
Penn State University

http://wyblelab.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20140127/94700716/attachment.html>