From grayw at rpi.edu Tue Apr 1 21:32:52 2014 From: grayw at rpi.edu (Gray, Wayne) Date: Wed, 2 Apr 2014 01:32:52 +0000 Subject: [ACT-R-users] Lecturer, Neuroscience Message-ID: <29BA228E-74BE-42F2-BE72-256FD5B7AFE7@rpi.edu> All, The Cognitive Science Department of Rensselaer Polytechnic Institute is looking for a Lecturer in Neuroscience to begin in the Fall. Computational Neuroscience is one of the three specific areas of competency we are looking for. A pdf of the official ad (just released) is attached. This is the first day that the ad is going out so no one should worry too much about the not submitting your materials BEFORE April 1st. I am chairing the search committee and would be happy to take short questions from anyone in this community. Cheers, Wayne -------------- next part -------------- A non-text attachment was scrubbed... Name: Cogs_005461_Neuroscience_Lecturer_FINAL.pdf Type: application/pdf Size: 33802 bytes Desc: Cogs_005461_Neuroscience_Lecturer_FINAL.pdf URL: From timothy.halverson.1 at us.af.mil Thu Apr 17 15:18:56 2014 From: timothy.halverson.1 at us.af.mil (HALVERSON, TIMOTHY E DR-02 USAF AFMC 711 HPW/RHCPA) Date: Thu, 17 Apr 2014 19:18:56 +0000 Subject: [ACT-R-users] Programmer Undergraduate/Graduate Student Internship at AFRL - Summer 2014 Message-ID: <7B9AEFC5D5CB6943B0C092374F8B4CFE2142878A@52ZHTX-D08-04D.area52.afnoapps.usaf.mil> My apologies if you receive multiple copies of this email. (Please note that individuals must be U.S. citizens or permanent legal residents of the United States to be eligible for these positions.) Dear Colleagues, Please broadly disseminate this opportunity for a summer internship with the Air Force Research Laboratory to your students. Programmer for Research Integrating Cognitive Agents with Web Interfaces SOCHE Supporting 711th Human Performance Wing, Air Force Research Laboratory, Wright-Patterson AFB, Ohio Available Summer 2014 The Warfighter Interface Division at the Air Force Research Laboratory seeks a talented programmer for a full-time research internship appointment supporting the development of a unified environment for cognitive agents that use web-based visualizations. This work contributes to a larger research goal automating the evaluation of visualization tools for warfighter decision making. The position offers experience working on a multi-disciplinary research team in an exciting, fast-paced environment advancing the cutting edge of models and metrics for human-machine teaming in complex environments. Specifically, the intern will work with mentors to develop the bridge between the cognitive agents and the web-based tasks/displays. Tasks will include writing client-server code and developing the communication protocol. The intern will develop or enhance skills in scripting languages (e.g. JavaScript, Python), communication protocol standards, and client-server paradigms. S/he will gain invaluable experience for working on a research and engineering team. Desired qualifications: Currently junior level or above seeking degree in Software Engineering, Computer Science, Information Technology, or related, relevant field; Solid background in JavaScript or Python; and Solid background in Java or C/C++; Experience using HTML or JavaScript D3 library helpful; Experience with client-server programming helpful; Familiarity with cognitive architectures or agent-based modeling helpful; Enthusiasm and curiosity for research helpful; Previous experience in a laboratory setting helpful; This position is a contractor appointment through the Southern Ohio Council for Higher Education (SOCHE) program. Appointment is for up to 10 weeks full-time with potential for follow-on appointment. This internship is located at Wright-Patterson AFB; internship requires appointee to relocate to Dayton, Ohio for the duration of the internship. Applicant must be a US Citizen and a degree-seeking student in good academic standing. The SOCHE program is open to all qualified US Citizens without regard to race, sex, religion, color, age, physical or mental disability, national origin, or status as a Vietnam era or disabled veteran. Applicants will need to apply at the SOCHE Internship web site: http://access.socheintern.org/ Applicants are encouraged to email their CV/resume and brief statement of research interests (1 page max), to Dr. Tim Halverson, 711 HPW/RHCP, Timothy.Halverson.1 at us.af.mil and Dr. Leslie Blaha, 711 HPW/RHCV, Leslie.Blaha at us.af.mil. Inquiries are welcome. Please be sure to include relevant programming coursework in your CV/resume. Please submit your application by May 15, 2014. From haltingstate at gmail.com Fri Apr 18 15:00:24 2014 From: haltingstate at gmail.com ( haltingstate@gmail.com) Date: Fri, 18 Apr 2014 12:00:24 -0700 Subject: [ACT-R-users] ACT-R modeling/user groups in Los Angeles? Message-ID: Someone suggested that the best way to become familiar with ACT-R was to see it used in an interaction study and that it was often used in eye tracking and human interface studies. Is anyone aware of any projects or users of ACT-R in the Los Angeles area? I do work in quantitative finance, natural language processing and am trying to understand ACT-R in practice better. I read the book and went through the code examples for the Python implementation but do not understand the framework and how it is used in practice well enough to do an independent implementation. Sincerely, Brandon Smietana -------------- next part -------------- An HTML attachment was scrubbed... URL: From db30 at andrew.cmu.edu Fri Apr 18 16:28:12 2014 From: db30 at andrew.cmu.edu (db30 at andrew.cmu.edu) Date: Fri, 18 Apr 2014 16:28:12 -0400 Subject: [ACT-R-users] ACT-R modeling/user groups in Los Angeles? In-Reply-To: References: Message-ID: <3238EEA1703C78AED08337CD@actr6b.cmu.edu> --On Friday, April 18, 2014 12:00 PM -0700 " haltingstate at gmail.com" wrote: > > > Someone suggested that the best way to become familiar with ACT-R was to see > it used in an interaction study and that it was often used in eye tracking > and human interface studies. > > Is anyone aware of any projects or users of ACT-R in the Los Angeles area? > > I do work in quantitative finance, natural language processing and am trying > to understand ACT-R in practice better. I read the book and went through the > code examples for the Python implementation but do not understand the > framework and how it is used in practice well enough to do an independent > implementation. > > I can't help you with users in the Los Angeles area, but I can make some suggestions for learning more about ACT-R. For information about the overall architecture I would recommend reading the paper "An Integrated Theory of the Mind" available from the ACT-R website: . I would also recommend going through the ACT-R tutorial which includes example models for a variety of different tasks. The tutorial is included with the official ACT-R software distribution or can also be downloaded separately from the software page of the ACT-R website: . Hope that helps, Dan From kevin.gluck at us.af.mil Thu Apr 24 08:27:01 2014 From: kevin.gluck at us.af.mil (GLUCK, KEVIN A DR-04 USAF AFMC 711 HPW/RHAC) Date: Thu, 24 Apr 2014 12:27:01 +0000 Subject: [ACT-R-users] Post-Doctoral Research in Machine Learning and Cognitive Modeling Message-ID: <358C5929AD0E82468006A37A29D5811B2D8DD038@52ZHTX-D06-03C.area52.afnoapps.usaf.mil> Post-Doctoral Research in Machine Learning and Cognitive Modeling Sensors Directorate and 711thHuman Performance Wing Air Force Research Laboratory Wright-Patterson AFB, Ohio Available immediately and until filled: This research effort focuses on modeling the human capacities and limits for rapid, generalizable pattern recognition and using those results to inform improvements to machine learning algorithms. Topics of interest related to this research include, but are not necessarily limited to: learning in sparse data environments, interpolation and extrapolation to novel stimuli, high-dimensional similarity-based recognition and retrieval processes, and human-machine teaming. Desired qualifications: * PhD in Computer Science, Computer Engineering, Cognitive Science, Cognitive Psychology, Statistics, Informatics, or a relevant related field; * Experience with machine learning theory, applications and programming * Experience with mathematical or computational process modeling; * Experience in or desire to learn malware analysis tools and techniques; * Proficiency in one or more of: C, C++, Java, MATLAB, Python (scikit-learn), R, cognitive architectures; * Enthusiasm and curiosity for working in an inter-disciplinary team research setting helpful; * Previous experience in a research laboratory helpful; Applicant must be a U.S. Citizen. The position is open to all qualified U.S. Citizens without regard to race, color, religion, sex (including pregnancy and gender identity), national origin, political affiliation, sexual orientation, marital status, disability, genetic information, age, membership in an employee organization, parental status, military service, or other non-merit factor. This position is a contractor appointment for up to 1 year full-time, with potential follow-on. Applicants should send a CV/resume and statement of research interests, both in pdf form, to Felicia Harlow (Felicia.Harlow at us.af.mil) and to (cognitive.modeling at us.af.mil). Inquiries are welcome. ************ Kevin Gluck, PhD kevin.gluck at us.af.mil From koedinger at cmu.edu Mon Apr 28 09:49:54 2014 From: koedinger at cmu.edu (Ken Koedinger) Date: Mon, 28 Apr 2014 09:49:54 -0400 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty Message-ID: <535E5C82.10607@cmu.edu> I'm looking for references for the following statement and figure the ACT-R community may have some: "One way to empirically evaluate the quality of a cognitive model is to test whether it can be used to accurately predict differences in task difficulty." I'm particularly interested in references to models that predict error rates (but reaction time prediction is ok too) across a number of related tasks. Models that predict errors at steps in tasks and/or specific strategy or error differences are even better. One such reference is our own tech report below -- see constraint C3 in Table 1. Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF ] Others? Including your own work? Thanks! Ken From bejohn at us.ibm.com Mon Apr 28 12:04:33 2014 From: bejohn at us.ibm.com (Bonnie E John) Date: Mon, 28 Apr 2014 12:04:33 -0400 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty In-Reply-To: <535E5C82.10607@cmu.edu> References: <535E5C82.10607@cmu.edu> Message-ID: Oh, I forgot to mention, Leonghwee's work is with CogTool-Explorer, which predicts the errors of novice users, so it addresses Ken's desire for error rates. Of course, all the "normal" CogTool work predicts differences in task execution time across tasks and UIs, as does all the Keystroke-Level Model and GOMS work before and since CogTool. So if you want details of KLM predictions, I'd start a The Psychology of Human-Computer Interaction by Card, Moran and Newell and work forwards. That book has dozens of examples with match to data. From: Ken Koedinger To: act-r-users at actr-server.hpc1.cs.cmu.edu Date: 04/28/2014 09:51 AM Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty Sent by: "ACT-R-users" I'm looking for references for the following statement and figure the ACT-R community may have some: "One way to empirically evaluate the quality of a cognitive model is to test whether it can be used to accurately predict differences in task difficulty." I'm particularly interested in references to models that predict error rates (but reaction time prediction is ok too) across a number of related tasks. Models that predict errors at steps in tasks and/or specific strategy or error differences are even better. One such reference is our own tech report below -- see constraint C3 in Table 1. Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF ] Others? Including your own work? Thanks! Ken _______________________________________________ ACT-R-users mailing list ACT-R-users at act-r.psy.cmu.edu https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From bejohn at us.ibm.com Mon Apr 28 12:00:02 2014 From: bejohn at us.ibm.com (Bonnie E John) Date: Mon, 28 Apr 2014 12:00:02 -0400 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty In-Reply-To: <535E5C82.10607@cmu.edu> References: <535E5C82.10607@cmu.edu> Message-ID: Leonghwee Teo's thesis and, more directly, the subsequent CHI paper had great predictive power for new users of a website. In 36 tasks, it predicted 93% of the easy tasks (95% of the users would finish successfully within 3 minutes and 93% of the hard tasks (>75% of humans couldn't complete within 3 minutes) with 14% false alarms (incorrectly identifying easy or medium tasks as hard). Looks like I neglected to put it in the ACT-R repository, so I'll send it to you in a separate email. Bonnie From: Ken Koedinger To: act-r-users at actr-server.hpc1.cs.cmu.edu Date: 04/28/2014 09:51 AM Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty Sent by: "ACT-R-users" I'm looking for references for the following statement and figure the ACT-R community may have some: "One way to empirically evaluate the quality of a cognitive model is to test whether it can be used to accurately predict differences in task difficulty." I'm particularly interested in references to models that predict error rates (but reaction time prediction is ok too) across a number of related tasks. Models that predict errors at steps in tasks and/or specific strategy or error differences are even better. One such reference is our own tech report below -- see constraint C3 in Table 1. Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF ] Others? Including your own work? Thanks! Ken _______________________________________________ ACT-R-users mailing list ACT-R-users at act-r.psy.cmu.edu https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From frank.ritter at psu.edu Mon Apr 28 13:50:57 2014 From: frank.ritter at psu.edu (Frank Ritter) Date: Mon, 28 Apr 2014 13:50:57 -0400 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty In-Reply-To: References: <535E5C82.10607@cmu.edu> Message-ID: the Diag model with Bibby and the followup work by Friedrich both predict with models not only which problems will be more difficult, but also the transfer across problems. The work with St. Amant shows interface differences http://acs.ist.psu.edu/papers/ritterBip.pdf Ritter, F. E., & Bibby, P. A. (2008). Modeling how, when, and what learning happens in a diagrammatic reasoning task. Cognitive Science. 32, 862-892. [This paper does not cite a paper by Bovair et al. (1990), and it should. Not because ideas were taken from it, at least explicitly, but because the Bovair work is similar and would be useful to anyone going further in this area {The acquisition and performance of text-editing skill: A cognitive complexity analysis. Human-Computer Interaction, 5, 1-48.}] http://acs.ist.psu.edu/papers/friedrichR09.pdf Friedrich, M. B., & Frank E. Ritter, F. E. (2009). Reimplementing a diagrammatic reasoning model in Herbal. In Proceedings of ICCM - 2009- Ninth International Conference on Cognitive Modeling. 438-439. Manchester, England. http://acs.ist.psu.edu/papers/stamantHR07.pdf St. Amant, R., Horton, T. E., & Ritter, F. E. (2007). Model-based evaluation of expert cell phone menu interaction. ACM Transactions on Computer-Human Interaction, 14(1), Article 1 (May 2007), 24 pages. cheers, Frank At 12:04 -0400 28/4/14, Bonnie E John wrote: >Oh, I forgot to mention, Leonghwee's work is with CogTool-Explorer, >which predicts the errors of novice users, so it addresses Ken's >desire for error rates. >Of course, all the "normal" CogTool work predicts differences in >task execution time across tasks and UIs, as does all the >Keystroke-Level Model and GOMS work before and since CogTool. So if >you want details of KLM predictions, I'd start a The Psychology of >Human-Computer Interaction by Card, Moran and Newell and work >forwards. That book has dozens of examples with match to data. > > Ken Koedinger ---04/28/2014 09:51:49 AM---I'm looking for >references for the following statement and figure the ACT-R >community may have some > >From: Ken Koedinger >To: act-r-users at actr-server.hpc1.cs.cmu.edu >Date: 04/28/2014 09:51 AM >Subject: [ACT-R-users] Ref for empirical tests of cognitive >models by predicting task difficulty >Sent by: "ACT-R-users" > > > > > >I'm looking for references for the following statement and figure the >ACT-R community may have some: > >"One way to empirically evaluate the quality of a cognitive model is to >test whether it can be used to accurately predict differences in task >difficulty." > >I'm particularly interested in references to models that predict error >rates (but reaction time prediction is ok too) across a number of >related tasks. Models that predict errors at steps in tasks and/or >specific strategy or error differences are even better. One such >reference is our own tech report below -- see constraint C3 in Table 1. > >Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical >domain theory of early algebra problem solving.CMU-HCII Tech Report >02-100.[PDF ><http://pact.cs.cmu.edu/koedinger/pubs/Koedinger,%20McLaren%20.pdf>] > >Others? Including your own work? > >Thanks! >Ken > > >_______________________________________________ >ACT-R-users mailing list >ACT-R-users at act-r.psy.cmu.edu >https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users > > >_______________________________________________ >ACT-R-users mailing list >ACT-R-users at act-r.psy.cmu.edu >https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users From glenn.gunzelmann at us.af.mil Tue Apr 29 08:30:21 2014 From: glenn.gunzelmann at us.af.mil (GUNZELMANN, GLENN F DR-03 USAF AFMC 711 HPW/RHAC) Date: Tue, 29 Apr 2014 12:30:21 +0000 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty In-Reply-To: <535E5C82.10607@cmu.edu> References: <535E5C82.10607@cmu.edu> Message-ID: <5CC9CA51959B144AA55D3C7CAF41791B2599BBD6@52ZHTX-D08-01A.area52.afnoapps.usaf.mil> Here's a paper that looks at different types of problems in the Tower of Hanoi that vary in difficulty, with a model that predicts exploratory/final path behavior: http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2003-gunzelmann_an derson.pdf Here's a second, with a model with three different strategies, one of which accounts for performance across seven different instrument-flight maneuvers of varying difficulty in a UAV simulation: http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2010-myers_gluck_g unzelmann_krusmark.pdf Finally, in case it is relevant when task difficulty varies as a function of the individual's state (i.e., well-rested versus sleep-deprived), this one looks at errors (and RT) in a simple addition/subtraction task: http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2012-gunzelmann_mo ore_gluck_dinges.pdf -Glenn -----Original Message----- From: ACT-R-users [mailto:act-r-users-bounces at ACTR-SERVER.HPC1.CS.cmu.edu] On Behalf Of Ken Koedinger Sent: Monday, April 28, 2014 9:50 AM To: act-r-users at ACTR-SERVER.HPC1.CS.cmu.edu Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty I'm looking for references for the following statement and figure the ACT-R community may have some: "One way to empirically evaluate the quality of a cognitive model is to test whether it can be used to accurately predict differences in task difficulty." I'm particularly interested in references to models that predict error rates (but reaction time prediction is ok too) across a number of related tasks. Models that predict errors at steps in tasks and/or specific strategy or error differences are even better. One such reference is our own tech report below -- see constraint C3 in Table 1. Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF ] Others? Including your own work? Thanks! Ken _______________________________________________ ACT-R-users mailing list ACT-R-users at act-r.psy.cmu.edu https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 6803 bytes Desc: not available URL: From troy.d.kelley6.civ at mail.mil Tue Apr 29 09:06:17 2014 From: troy.d.kelley6.civ at mail.mil (Kelley, Troy D CIV (US)) Date: Tue, 29 Apr 2014 13:06:17 +0000 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty (UNCLASSIFIED) In-Reply-To: References: <535E5C82.10607@cmu.edu> Message-ID: Classification: UNCLASSIFIED Caveats: NONE Ken, I did some work in predicting error rates awhile ago, in another life ;-). BTW, I think this is a money maker for cognitive modeling. It is one of the best applied uses of cognitive modeling. Kelley, Troy, D.; Patton, D.; Allender, L. (2001). Error rates in mental manipulation of spatial images. Perceptual and Motor skills, 92, 985-992. Kelley, Troy, D., Patton, Debra, J. Allender, L. (2001). Predicting Situation Awareness Errors using Cognitive Modeling. In: M. J. Smith, G. Salvendy, D. Harris, R.J. Koubek. (Eds.) Proceedings of Human-Computer Interaction International 2001 Conference: (Vol. 1) Usability Evaluation and Interface Design: Cognitive Engineering, Intelligent Agents and Virtual Reality. (pp. 1455 - 1459). Lawrence Erlbaum Associates, Mahwah, New Jersey. Kelley, T. D., Scribner, D. R., (2003). Developing a Predictive Model of Dual Task Performance. ARL Technical Report ARL-MR-0556. Kelley, Troy, D.; Lee, Frank., Wiley, Patrick.; (2000). Developing and ACT-R model of Spatial Manipulation. ARL-Technical report. (ARL-TR-2179). Aberdeen Proving Ground, Aberdeen, Maryland. Troy Kelley Cognitive Robotics Team Leader Human Research and Engineering Directorate U.S. Army Research Laboratory Aberdeen, MD 21005 Voice :410-278-5869 -----Original Message----- From: ACT-R-users [mailto:act-r-users-bounces at ACTR-SERVER.HPC1.CS.cmu.edu] On Behalf Of Bonnie E John Sent: Monday, April 28, 2014 12:00 PM To: Ken Koedinger Cc: act-r-users at ACTR-SERVER.HPC1.CS.cmu.edu; ACT-R-users Subject: Re: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty Leonghwee Teo's thesis and, more directly, the subsequent CHI paper had great predictive power for new users of a website. In 36 tasks, it predicted 93% of the easy tasks (95% of the users would finish successfully within 3 minutes and 93% of the hard tasks (>75% of humans couldn't complete within 3 minutes) with 14% false alarms (incorrectly identifying easy or medium tasks as hard). Looks like I neglected to put it in the ACT-R repository, so I'll send it to you in a separate email. Bonnie Inactive hide details for Ken Koedinger ---04/28/2014 09:51:49 AM---I'm looking for references for the following statement and Ken Koedinger ---04/28/2014 09:51:49 AM---I'm looking for references for the following statement and figure the ACT-R community may have some From: Ken Koedinger To: act-r-users at actr-server.hpc1.cs.cmu.edu Date: 04/28/2014 09:51 AM Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty Sent by: "ACT-R-users" ________________________________ I'm looking for references for the following statement and figure the ACT-R community may have some: "One way to empirically evaluate the quality of a cognitive model is to test whether it can be used to accurately predict differences in task difficulty." I'm particularly interested in references to models that predict error rates (but reaction time prediction is ok too) across a number of related tasks. Models that predict errors at steps in tasks and/or specific strategy or error differences are even better. One such reference is our own tech report below -- see constraint C3 in Table 1. Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF ] Others? Including your own work? Thanks! Ken _______________________________________________ ACT-R-users mailing list ACT-R-users at act-r.psy.cmu.edu https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users Classification: UNCLASSIFIED Caveats: NONE -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5619 bytes Desc: not available URL: From susan.chipman at gmail.com Tue Apr 29 12:46:05 2014 From: susan.chipman at gmail.com (Susan Chipman) Date: Tue, 29 Apr 2014 10:46:05 -0600 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty In-Reply-To: <5CC9CA51959B144AA55D3C7CAF41791B2599BBD6@52ZHTX-D08-01A.area52.afnoapps.usaf.mil> References: <535E5C82.10607@cmu.edu> <5CC9CA51959B144AA55D3C7CAF41791B2599BBD6@52ZHTX-D08-01A.area52.afnoapps.usaf.mil> Message-ID: A name has come to mind that is probably the name of the ETS researcher I mentioned to Ken re efforts to predict the difficulty of math test items -- Isaac Bejar (pronounced Behar). Probably there were other such efforts at ETS for obvious reasons. However, that work did not involve cognitive modeling. Frank Ritter's response reminded me that David Kieras (quite possibly in collaboration with Bovair) did some very impressive work on transfer of training operational procedures for a mythical machine. Very precise prediction of learning times in relation to order of training and overlapping production rules was achieved. I used graphs showing these results in briefings at ONR to impress the physical scientists and engineers that we really were doing SCIENCE. Susan Chipman On Tue, Apr 29, 2014 at 6:30 AM, GUNZELMANN, GLENN F DR-03 USAF AFMC 711 HPW/RHAC wrote: > > Here's a paper that looks at different types of problems in the Tower of > Hanoi that vary in difficulty, with a model that predicts exploratory/final > path behavior: > > http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2003-gunzelmann_an > derson.pdf > > Here's a second, with a model with three different strategies, one of which > accounts for performance across seven different instrument-flight maneuvers > of varying difficulty in a UAV simulation: > > http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2010-myers_gluck_g > unzelmann_krusmark.pdf > > Finally, in case it is relevant when task difficulty varies as a function > of > the individual's state (i.e., well-rested versus sleep-deprived), this one > looks at errors (and RT) in a simple addition/subtraction task: > > http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2012-gunzelmann_mo > ore_gluck_dinges.pdf > > -Glenn > > > -----Original Message----- > From: ACT-R-users [mailto:act-r-users-bounces at ACTR-SERVER.HPC1.CS.cmu.edu] > On Behalf Of Ken Koedinger > Sent: Monday, April 28, 2014 9:50 AM > To: act-r-users at ACTR-SERVER.HPC1.CS.cmu.edu > Subject: [ACT-R-users] Ref for empirical tests of cognitive models by > predicting task difficulty > > > I'm looking for references for the following statement and figure the ACT-R > community may have some: > > "One way to empirically evaluate the quality of a cognitive model is to > test > whether it can be used to accurately predict differences in task > difficulty." > > I'm particularly interested in references to models that predict error > rates > (but reaction time prediction is ok too) across a number of related tasks. > Models that predict errors at steps in tasks and/or specific strategy or > error differences are even better. One such reference is our own tech > report below -- see constraint C3 in Table 1. > > Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain > theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF > ] > > Others? Including your own work? > > Thanks! > Ken > > > _______________________________________________ > ACT-R-users mailing list > ACT-R-users at act-r.psy.cmu.edu > https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users > > _______________________________________________ > ACT-R-users mailing list > ACT-R-users at act-r.psy.cmu.edu > https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Pirolli at parc.com Tue Apr 29 13:21:24 2014 From: Peter.Pirolli at parc.com (Peter.Pirolli at parc.com) Date: Tue, 29 Apr 2014 17:21:24 +0000 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty In-Reply-To: References: <535E5C82.10607@cmu.edu> <5CC9CA51959B144AA55D3C7CAF41791B2599BBD6@52ZHTX-D08-01A.area52.afnoapps.usaf.mil> Message-ID: The Kieras & Bovair work I remember dealt with learning to fire the Starship Enterprise phaser bank. Here are a couple of references. Kieras, D. E., & Bovair, S. (1984). The role of a mental model in learning to operate a device. Cognitive Science, 8, 255-273. Kieras, D. E., & Bovair, S. (1986). The acquisition of procedures from text: A production system analysis of transfer of training. Journal of Memory and Language, 25, 507-524. Those papers may have initiated the thread of "identical elements" production system models of transfer that included the Singley & Anderson book Singley, M. K., & Anderson, J. R. (1989). Transfer of cognitive skill. Cambridge, MA: Harvard University Press. I I I did a paper based specifically on predicting error rates based on such models, and somewhat inspired by the Kieras & Bovair work on mental models Pirolli, P. (1991). Effects of examples and their explanations in a lesson on recursion: A production system analysis. Cognition and Instruction, 8, 207-259. I'm not sure how you want to draw the line between measurement and prediction, but here's a paper on combining objective measurement with production system representations to objectively measure difficulty and ability for individual skills in the Lisp Tutor Pirolli, P., & Wilson, M. (1998). A theory of the measurement of knowledge content, access, and learning. Psychological Review, 105, 58-82. On Apr 29, 2014, at 9:46 AM, Susan Chipman > wrote: A name has come to mind that is probably the name of the ETS researcher I mentioned to Ken re efforts to predict the difficulty of math test items -- Isaac Bejar (pronounced Behar). Probably there were other such efforts at ETS for obvious reasons. However, that work did not involve cognitive modeling. Frank Ritter's response reminded me that David Kieras (quite possibly in collaboration with Bovair) did some very impressive work on transfer of training operational procedures for a mythical machine. Very precise prediction of learning times in relation to order of training and overlapping production rules was achieved. I used graphs showing these results in briefings at ONR to impress the physical scientists and engineers that we really were doing SCIENCE. Susan Chipman On Tue, Apr 29, 2014 at 6:30 AM, GUNZELMANN, GLENN F DR-03 USAF AFMC 711 HPW/RHAC > wrote: Here's a paper that looks at different types of problems in the Tower of Hanoi that vary in difficulty, with a model that predicts exploratory/final path behavior: http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2003-gunzelmann_an derson.pdf Here's a second, with a model with three different strategies, one of which accounts for performance across seven different instrument-flight maneuvers of varying difficulty in a UAV simulation: http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2010-myers_gluck_g unzelmann_krusmark.pdf Finally, in case it is relevant when task difficulty varies as a function of the individual's state (i.e., well-rested versus sleep-deprived), this one looks at errors (and RT) in a simple addition/subtraction task: http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2012-gunzelmann_mo ore_gluck_dinges.pdf -Glenn -----Original Message----- From: ACT-R-users [mailto:act-r-users-bounces at ACTR-SERVER.HPC1.CS.cmu.edu] On Behalf Of Ken Koedinger Sent: Monday, April 28, 2014 9:50 AM To: act-r-users at ACTR-SERVER.HPC1.CS.cmu.edu Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty I'm looking for references for the following statement and figure the ACT-R community may have some: "One way to empirically evaluate the quality of a cognitive model is to test whether it can be used to accurately predict differences in task difficulty." I'm particularly interested in references to models that predict error rates (but reaction time prediction is ok too) across a number of related tasks. Models that predict errors at steps in tasks and/or specific strategy or error differences are even better. One such reference is our own tech report below -- see constraint C3 in Table 1. Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF ] Others? Including your own work? Thanks! Ken _______________________________________________ ACT-R-users mailing list ACT-R-users at act-r.psy.cmu.edu https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users _______________________________________________ ACT-R-users mailing list ACT-R-users at act-r.psy.cmu.edu https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users _______________________________________________ ACT-R-users mailing list ACT-R-users at act-r.psy.cmu.edu https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.ritter at psu.edu Tue Apr 29 13:16:31 2014 From: frank.ritter at psu.edu (Frank Ritter) Date: Tue, 29 Apr 2014 13:16:31 -0400 Subject: [ACT-R-users] Ref for empirical tests of cognitive models by predicting task difficulty In-Reply-To: References: <535E5C82.10607@cmu.edu> <5CC9CA51959B144AA55D3C7CAF41791B2599BBD6@52ZHTX-D08-01A.area52.afnoapps.u saf.mil> Message-ID: it might be one of these: Kieras, D. E., & Bovair, S. (1984). The role of a mental model in learning how to operator a device. Cognitive Science, 8, 255-273. Kieras, D. E., & Bovair, S. (1986). The acquisition of procedures from text: A production system model. Journal of Memory and Language, 25, 507-524. Bovair, S., Kieras, D. E., & Polson, P. G. (1990). The acquisition and performance of text-editing skill: A cognitive complexity analysis. Human-Computer Interaction, 5, 1-48. cheers, Frank At 10:46 -0600 29/4/14, Susan Chipman wrote: > A name has come to mind that is probably the name of the >ETS researcher I mentioned to Ken re efforts to predict the >difficulty of math test items -- Isaac Bejar (pronounced Behar). >Probably there were other such efforts at ETS for obvious reasons. >However, that work did not involve cognitive modeling. > > Frank Ritter's response reminded me that David Kieras >(quite possibly in collaboration with Bovair) did some very >impressive work on transfer of training operational procedures for a >mythical machine. Very precise prediction of learning times in >relation to order of training and overlapping production rules was >achieved. I used graphs showing these results in briefings at ONR >to impress the physical scientists and engineers that we really were >doing SCIENCE. > >Susan Chipman > > > >On Tue, Apr 29, 2014 at 6:30 AM, GUNZELMANN, GLENN F DR-03 USAF AFMC >711 HPW/RHAC ><glenn.gunzelmann at us.af.mil> >wrote: > > >Here's a paper that looks at different types of problems in the Tower of >Hanoi that vary in difficulty, with a model that predicts exploratory/final >path behavior: >derson.pdf>http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2003-gunzelmann_an >derson.pdf > >Here's a second, with a model with three different strategies, one of which >accounts for performance across seven different instrument-flight maneuvers >of varying difficulty in a UAV simulation: >unzelmann_krusmark.pdf>http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2010-myers_gluck_g >unzelmann_krusmark.pdf > >Finally, in case it is relevant when task difficulty varies as a function of >the individual's state (i.e., well-rested versus sleep-deprived), this one >looks at errors (and RT) in a simple addition/subtraction task: >ore_gluck_dinges.pdf>http://palm.mindmodeling.org/~glenng/pubs/journalarticles/2012-gunzelmann_mo >ore_gluck_dinges.pdf > >-Glenn > > > >-----Original Message----- >From: ACT-R-users >[mailto:act-r-users-bounces at ACTR-SERVER.HPC1.CS.cmu.edu] >On Behalf Of Ken Koedinger >Sent: Monday, April 28, 2014 9:50 AM >To: >act-r-users at ACTR-SERVER.HPC1.CS.cmu.edu >Subject: [ACT-R-users] Ref for empirical tests of cognitive models by >predicting task difficulty > >I'm looking for references for the following statement and figure the ACT-R >community may have some: > >"One way to empirically evaluate the quality of a cognitive model is to test >whether it can be used to accurately predict differences in task >difficulty." > >I'm particularly interested in references to models that predict error rates >(but reaction time prediction is ok too) across a number of related tasks. >Models that predict errors at steps in tasks and/or specific strategy or >error differences are even better. One such reference is our own tech >report below -- see constraint C3 in Table 1. > >Koedinger, K.R., & MacLaren, B. A. (2002).Developing a pedagogical domain >theory of early algebra problem solving.CMU-HCII Tech Report 02-100.[PDF ><http://pact.cs.cmu.edu/koedinger/pubs/Koedinger,%20McLaren%20.pdf>] > >Others? Including your own work? > >Thanks! >Ken > > >_______________________________________________ >ACT-R-users mailing list >ACT-R-users at act-r.psy.cmu.edu >https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users > > >_______________________________________________ >ACT-R-users mailing list >ACT-R-users at act-r.psy.cmu.edu >https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users > > > >_______________________________________________ >ACT-R-users mailing list >ACT-R-users at act-r.psy.cmu.edu >https://mailman.srv.cs.cmu.edu/mailman/listinfo/act-r-users