<div dir="ltr">Thanks Randal, that's a great suggestion. I'll ask my colleagues in physics for their perspective as well. <div><br></div><div>-Brad</div><div><br></div><div><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Mon, Jan 27, 2014 at 11:54 PM, Randal Koene <span dir="ltr"><<a href="mailto:randal.a.koene@gmail.com" target="_blank">randal.a.koene@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div style="font-family:arial,sans-serif;font-size:13px">Hi Brad,</div><span style="font-family:arial,sans-serif;font-size:13px">This reminds me of theoretical physics, where proposed models are expounded in papers, often without the ability to immediately carry out empirical tests of all the predictions. Subsequently, experiments are often designed to compare and contrast different models.</span><div style="font-family:arial,sans-serif;font-size:13px">
Perhaps a way to advance this is indeed to make the analogy with physics?</div><div style="font-family:arial,sans-serif;font-size:13px">Cheers,</div><div style="font-family:arial,sans-serif;font-size:13px">Randal</div></div>
<div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div>Dr. Randal A. Koene</div><div><a href="mailto:Randal.A.Koene@gmail.com" target="_blank">Randal.A.Koene@gmail.com</a> - <a href="mailto:Randal.A.Koene@carboncopies.org" target="_blank">Randal.A.Koene@carboncopies.org</a></div>
<div><a href="http://randalkoene.com" target="_blank">http://randalkoene.com</a> - <a href="http://carboncopies.org" target="_blank">http://carboncopies.org</a></div></div></div><div><div class="h5">
<br><br><div class="gmail_quote">On Mon, Jan 27, 2014 at 8:29 PM, Brad Wyble <span dir="ltr"><<a href="mailto:bwyble@gmail.com" target="_blank">bwyble@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Thank you Mark, I hadn't seen this paper. She includes this other point that should have been in my list:<div><br></div><div><div>"From a practical point of view, as noted the time required to build </div>
<div>and analyze a computational model is quite substantial and validation may </div><div>require teams. To delay model presentation until validation has occurred </div><div>retards the development of the scientific field. " ----Carley (1999)</div>
</div><div> </div><div><br></div><div>And here is a citation for this paper.</div><div>Carley, Kathleen M., 1999. Validating Computational Models. CASOS Working Paper, CMU<span><font color="#888888"><br></font></span></div>
<span><font color="#888888"><div><br></div><div>-Brad</div><div><br>
</div><div><br></div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jan 27, 2014 at 9:48 PM, Mark Orr <span dir="ltr"><<a href="mailto:mo2259@columbia.edu" target="_blank">mo2259@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Brad, <div>Kathleen Carley, at CMU, has a paper on this idea (from the 1990s), suggesting the same practice. See <a href="http://www2.econ.iastate.edu/tesfatsi/EmpValid.Carley.pdf" target="_blank">http://www2.econ.iastate.edu/tesfatsi/EmpValid.Carley.pdf</a></div>
<span><font color="#888888"><div><br></div><div>Mark</div></font></span><div><div><div><br><div><div>On Jan 27, 2014, at 9:39 PM, Brad Wyble wrote:</div><br><blockquote type="cite"><div dir="ltr">
Dear connectionists, <div><br></div><div>I wanted to get some feedback regarding some recent ideas concerning the publication of models because I think that our current practices are slowing down the progress of theory. At present, at least in many psychology journals, it is often expected that a computational modelling paper includes experimental evidence in favor of a small handful of its own predictions. While I am certainly in favor of model testing, I have come to the suspicion that the practice of including empirical validation within the same paper as the initial model is problematic for several reasons:</div>
<div><br></div><div><div>It encourages the creation only of predictions that are easy to test with the techniques available to the modeller.</div></div><div><br></div><div>It strongly encourages a practice of running an experiment, designing a model to fit those results, and then claiming this as a bona fide prediction. </div>
<div><br></div><div>It encourages a practice of running a battery of experiments and reporting only those that match the model's output. </div><div><br></div><div>It encourages the creation of predictions which cannot fail, and are therefore less informative</div>
<div><br></div><div>It encourages a mindset that a model is a failure if all of its predictions are not validated, when in fact we actually learn more from a failed prediction than a successful one.</div><div><br></div><div>
It makes it easier for experimentalists to ignore models, since such modelling papers are "self contained". </div><div><br></div><div>I was thinking that, instead of the current practice, it should be permissible and even encouraged that a modelling paper should not include empirical validation, but instead include a broader array of predictions. Thus instead of 3 successfully tested predictions from the PI's own lab, a model might include 10 untested predictions for a variety of different experimental techniques. This practice will, I suspect, lead to the development of bolder theories, stronger tests, and most importantly, tighter ties between empiricists and theoreticians. </div>
<div><br></div><div>I am certainly not advocating that modellers shouldn't test their own models, but rather that it should be permissible to publish a model without testing it first. The testing paper could come later. </div>
<div><br></div><div>I also realize that this shift in publication expectations wouldn't prevent the problems described above, but it would at least not reward them. </div><div><br></div><div>I also think that modellers should make a concerted effort to target empirical journals to increase the visibility of models. This effort should coincide with a shift in writing style to make such models more accessible to non modellers.</div>
<div><br></div><div>What do people think of this? If there is broad agreement, what would be the best way to communicate this desire to journal editors?</div><div><br></div><div>Any advice welcome!</div><div><br></div><div>
-Brad<br></div><div><br></div><div><br></div><div><br></div><div>-- <br><div dir="ltr">Brad Wyble<br>Assistant Professor<br>Psychology Department<br>Penn State University<div><br></div><div><a href="http://wyblelab.com/" target="_blank">http://wyblelab.com</a></div>
</div>
</div></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Brad Wyble<br>Assistant Professor<br>Psychology Department<br>Penn State University<div><br></div>
<div><a href="http://wyblelab.com" target="_blank">http://wyblelab.com</a></div></div>
</div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Brad Wyble<br>Assistant Professor<br>Psychology Department<br>Penn State University<div><br></div><div><a href="http://wyblelab.com" target="_blank">http://wyblelab.com</a></div>
</div>
</div>