[ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty (UNCLASSIFIED)

Tue Apr 29 09:06:17 EDT 2014

Classification: UNCLASSIFIED
Caveats: NONE

Ken,

I did some work in predicting error rates awhile ago, in another life ;-).  

BTW, I think this is a money maker for cognitive modeling.  It is one of the
best applied uses of cognitive modeling.

Kelley, Troy, D.; Patton, D.; Allender, L. (2001). Error rates in mental
manipulation of spatial images.  Perceptual and Motor skills, 92, 985-992.

Kelley, Troy, D., Patton, Debra, J. Allender, L. (2001). Predicting
Situation Awareness Errors using Cognitive Modeling.  In: M. J. Smith, G.
Salvendy, D. Harris, R.J. Koubek. (Eds.) Proceedings of Human-Computer
Interaction International 2001 Conference: (Vol. 1)  Usability Evaluation
and Interface Design: Cognitive Engineering, Intelligent Agents and Virtual
Reality. (pp. 1455 - 1459). Lawrence Erlbaum Associates, Mahwah, New Jersey.

Kelley, T. D., Scribner, D. R., (2003). Developing a Predictive Model of
Dual Task Performance.  ARL Technical Report ARL-MR-0556.

Kelley, Troy, D.; Lee, Frank., Wiley, Patrick.; (2000). Developing and ACT-R
model of Spatial Manipulation.  ARL-Technical report. (ARL-TR-2179).
Aberdeen Proving Ground, Aberdeen, Maryland.

Troy Kelley
Cognitive Robotics Team Leader
Human Research and Engineering Directorate
U.S. Army Research Laboratory
Aberdeen, MD
21005
Voice :410-278-5869

-----Original Message-----
From: ACT-R-users [mailto:act-r-users-bounces at ACTR-SERVER.HPC1.CS.cmu.edu]
On Behalf Of Bonnie E John
Sent: Monday, April 28, 2014 12:00 PM
To: Ken Koedinger
Cc: act-r-users at ACTR-SERVER.HPC1.CS.cmu.edu; ACT-R-users
Subject: Re: [ACT-R-users] Ref for empirical tests of cognitive models by
predicting task difficulty

Leonghwee Teo's thesis and, more directly, the subsequent CHI paper had
great predictive power for new users of a website. In 36 tasks, it predicted
93% of the easy tasks (95% of the users would finish successfully within 3
minutes and 93% of the hard tasks (>75% of humans couldn't complete within 3
minutes) with 14% false alarms (incorrectly identifying easy or medium tasks
as hard).
Looks like I neglected to put it in the ACT-R repository, so I'll send it to
you in a separate email.
Bonnie

Inactive hide details for Ken Koedinger ---04/28/2014 09:51:49 AM---I'm
looking for references for the following statement and Ken Koedinger
---04/28/2014 09:51:49 AM---I'm looking for references for the following
statement and figure the  ACT-R community may have some

From: Ken Koedinger <koedinger at cmu.edu>
To: act-r-users at actr-server.hpc1.cs.cmu.edu
Date: 04/28/2014 09:51 AM
Subject: [ACT-R-users] Ref for empirical tests of cognitive models by
predicting task difficulty Sent by: "ACT-R-users"
<act-r-users-bounces at actr-server.hpc1.cs.cmu.edu>

________________________________

I'm looking for references for the following statement and figure the ACT-R
community may have some:

"One way to empirically evaluate the quality of a cognitive model is to test
whether it can be used to accurately predict differences in task
difficulty."

I'm particularly interested in references to models that predict error rates
(but reaction time prediction is ok too) across a number of related tasks.
Models that predict errors at steps in tasks and/or specific strategy or
error differences are even better.  One such reference is our own tech
report below -- see constraint C3 in Table 1.

Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain
theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF
<http://pact.cs.cmu.edu/koedinger/pubs/Koedinger,%20McLaren%20.pdf>]

Others?  Including your own work?

Thanks!
Ken

_______________________________________________
ACT-R-users mailing list
ACT-R-users at act-r.psy.cmu.edu
https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users

Classification: UNCLASSIFIED
Caveats: NONE

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5619 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/act-r-users/attachments/20140429/2b8fbe5e/attachment.bin>