[ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty

Tue Apr 29 13:21:24 EDT 2014

The Kieras & Bovair work I remember dealt with learning to fire the Starship Enterprise phaser bank. Here are a couple of references.

Kieras, D. E., & Bovair, S. (1984). The role of a mental model in learning to operate a device. Cognitive Science, 8, 255-273.

Kieras, D. E., & Bovair, S. (1986). The acquisition of procedures from text: A production system analysis of transfer of training. Journal of Memory and Language, 25, 507-524.

Those papers may have initiated the thread of "identical elements" production system models of transfer that included the Singley & Anderson book

Singley, M. K., & Anderson, J. R. (1989). Transfer of cognitive skill. Cambridge, MA: Harvard University Press.

I
I

I did a paper based specifically on predicting error rates based on such models, and somewhat inspired by the Kieras & Bovair work on mental models

Pirolli, P. (1991). Effects of examples and their explanations in a lesson on recursion:  A production system analysis. Cognition and Instruction, 8, 207-259.

I'm not sure how you want to draw the line between measurement and prediction, but here's a paper on combining objective measurement with production system representations to objectively measure difficulty and ability for individual skills in the Lisp Tutor

Pirolli, P., & Wilson, M. (1998). A theory of the measurement of knowledge content, access, and learning. Psychological Review, 105, 58-82.

On Apr 29, 2014, at 9:46 AM, Susan Chipman <susan.chipman at gmail.com<mailto:susan.chipman at gmail.com>>
 wrote:

         A name has come to mind that is probably the name of the ETS researcher I mentioned to Ken re efforts to predict the difficulty of math test items -- Isaac Bejar (pronounced Behar).  Probably there were other such efforts at ETS for obvious reasons.  However, that work did not involve cognitive modeling.
          Frank Ritter's response reminded me that David Kieras (quite possibly in collaboration with Bovair) did some very impressive work on transfer of training operational procedures for a mythical machine.  Very precise prediction of learning times in relation to order of training and overlapping production rules was achieved.  I used graphs showing these results in briefings at ONR to impress the physical scientists and engineers that we really were doing SCIENCE.

Susan Chipman

On Tue, Apr 29, 2014 at 6:30 AM, GUNZELMANN, GLENN F DR-03 USAF AFMC 711 HPW/RHAC <glenn.gunzelmann at us.af.mil<mailto:glenn.gunzelmann at us.af.mil>> wrote:

Here's a paper that looks at different types of problems in the Tower of
Hanoi that vary in difficulty, with a model that predicts exploratory/final
path behavior:
http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2003-gunzelmann_an
derson.pdf<http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2003-gunzelmann_anderson.pdf>

Here's a second, with a model with three different strategies, one of which
accounts for performance across seven different instrument-flight maneuvers
of varying difficulty in a UAV simulation:
http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2010-myers_gluck_g
unzelmann_krusmark.pdf<http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2010-myers_gluck_gunzelmann_krusmark.pdf>

Finally, in case it is relevant when task difficulty varies as a function of
the individual's state (i.e., well-rested versus sleep-deprived), this one
looks at errors (and RT) in a simple addition/subtraction task:
http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2012-gunzelmann_mo
ore_gluck_dinges.pdf<http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2012-gunzelmann_moore_gluck_dinges.pdf>

-Glenn

-----Original Message-----
From: ACT-R-users [mailto:act-r-users-bounces at ACTR-SERVER.HPC1.CS.cmu.edu<mailto:act-r-users-bounces at ACTR-SERVER.HPC1.CS.cmu.edu>]
On Behalf Of Ken Koedinger
Sent: Monday, April 28, 2014 9:50 AM
To: act-r-users at ACTR-SERVER.HPC1.CS.cmu.edu<mailto:act-r-users at ACTR-SERVER.HPC1.CS.cmu.edu>
Subject: [ACT-R-users] Ref for empirical tests of cognitive models by
predicting task difficulty

I'm looking for references for the following statement and figure the ACT-R
community may have some:

"One way to empirically evaluate the quality of a cognitive model is to test
whether it can be used to accurately predict differences in task
difficulty."

I'm particularly interested in references to models that predict error rates
(but reaction time prediction is ok too) across a number of related tasks.
Models that predict errors at steps in tasks and/or specific strategy or
error differences are even better.  One such reference is our own tech
report below -- see constraint C3 in Table 1.

Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain
theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF
<http://pact.cs.cmu.edu/koedinger/pubs/Koedinger,%20McLaren%20.pdf>]

Others?  Including your own work?

Thanks!
Ken

_______________________________________________
ACT-R-users mailing list
ACT-R-users at act-r.psy.cmu.edu<mailto:ACT-R-users at act-r.psy.cmu.edu>
https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users

_______________________________________________
ACT-R-users mailing list
ACT-R-users at act-r.psy.cmu.edu<mailto:ACT-R-users at act-r.psy.cmu.edu>
https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users

_______________________________________________
ACT-R-users mailing list
ACT-R-users at act-r.psy.cmu.edu<mailto:ACT-R-users at act-r.psy.cmu.edu>
https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/act-r-users/attachments/20140429/1f1125ba/attachment.html>