Connectionists: Best practices in model publication

[Brad Wyble]

Thanks Randal, that's a great suggestion.  I'll ask my colleagues in
physics for their perspective as well.

-Brad




On Mon, Jan 27, 2014 at 11:54 PM, Randal Koene <randal.a.koene at gmail.com>wrote:

> Hi Brad,
> This reminds me of theoretical physics, where proposed models are
> expounded in papers, often without the ability to immediately carry out
> empirical tests of all the predictions. Subsequently, experiments are often
> designed to compare and contrast different models.
> Perhaps a way to advance this is indeed to make the analogy with physics?
> Cheers,
> Randal
>
> Dr. Randal A. Koene
> Randal.A.Koene at gmail.com - Randal.A.Koene at carboncopies.org
> http://randalkoene.com - http://carboncopies.org
>
>
> On Mon, Jan 27, 2014 at 8:29 PM, Brad Wyble <bwyble at gmail.com> wrote:
>
>> Thank you Mark, I hadn't seen this paper.  She includes this other point
>> that should have been in my list:
>>
>> "From a practical point of view, as noted the time required to build
>> and analyze a computational model is quite substantial and validation may
>> require teams. To delay model presentation until validation has occurred
>> retards the development of the scientific field. "  ----Carley (1999)
>>
>>
>> And here is a citation for this paper.
>> Carley, Kathleen M., 1999. Validating Computational Models. CASOS Working
>> Paper, CMU
>>
>> -Brad
>>
>>
>>
>>
>> On Mon, Jan 27, 2014 at 9:48 PM, Mark Orr <mo2259 at columbia.edu> wrote:
>>
>>> Brad,
>>> Kathleen Carley, at CMU, has a paper on this idea (from the 1990s),
>>> suggesting the same practice. See
>>> http://www2.econ.iastate.edu/tesfatsi/EmpValid.Carley.pdf
>>>
>>> Mark
>>>
>>> On Jan 27, 2014, at 9:39 PM, Brad Wyble wrote:
>>>
>>> Dear connectionists,
>>>
>>> I wanted to get some feedback regarding some recent ideas concerning the
>>> publication of models because I think that our current practices are
>>> slowing down the progress of theory.  At present, at least in many
>>> psychology journals, it is often expected that a computational modelling
>>> paper includes experimental evidence in favor of  a small handful of its
>>> own predictions.  While I am certainly in favor of  model testing, I have
>>> come to the suspicion that the practice of including empirical validation
>>> within the same paper as the initial model is problematic for several
>>> reasons:
>>>
>>> It encourages the creation only of predictions that are easy to test
>>> with the techniques available to the modeller.
>>>
>>> It strongly encourages a practice of running an experiment, designing a
>>> model to fit those results, and then claiming this as a bona fide
>>> prediction.
>>>
>>> It encourages a practice of running a battery of experiments and
>>> reporting only those that match the model's output.
>>>
>>> It encourages the creation of predictions which cannot fail, and are
>>> therefore less informative
>>>
>>> It encourages a mindset that a model is a failure if all of its
>>> predictions are not validated, when in fact we actually learn more from a
>>> failed prediction than a successful one.
>>>
>>> It makes it easier for experimentalists to ignore models, since such
>>> modelling papers are "self contained".
>>>
>>> I was thinking that, instead of the current practice, it should be
>>> permissible and even encouraged that a modelling paper should not include
>>> empirical validation, but instead include a broader array of predictions.
>>>  Thus instead of 3 successfully tested predictions from the PI's own lab, a
>>> model might include 10 untested predictions for a variety of different
>>> experimental techniques. This practice will, I suspect, lead to the
>>> development of bolder theories, stronger tests, and most importantly,
>>> tighter ties between empiricists and theoreticians.
>>>
>>> I am certainly not advocating that modellers shouldn't test their own
>>> models, but rather that it should be permissible to publish a model without
>>> testing it first. The testing paper could come later.
>>>
>>> I also realize that this shift in publication expectations  wouldn't
>>> prevent the problems described above, but it would at least not reward
>>> them.
>>>
>>> I also think that modellers should make a concerted effort to target
>>> empirical journals to increase the visibility of models.  This effort
>>> should coincide with a shift in writing style to make such models more
>>> accessible to non modellers.
>>>
>>> What do people think of this? If there is broad agreement, what would be
>>> the best way to communicate this desire to journal editors?
>>>
>>> Any advice welcome!
>>>
>>> -Brad
>>>
>>>
>>>
>>> --
>>> Brad Wyble
>>> Assistant Professor
>>> Psychology Department
>>> Penn State University
>>>
>>> http://wyblelab.com
>>>
>>>
>>>
>>
>>
>> --
>> Brad Wyble
>> Assistant Professor
>> Psychology Department
>> Penn State University
>>
>> http://wyblelab.com
>>
>
>


-- 
Brad Wyble
Assistant Professor
Psychology Department
Penn State University

http://wyblelab.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20140128/5ee75078/attachment.html>