From cl+ at andrew.cmu.edu Fri Oct 6 12:06:22 1995 From: cl+ at andrew.cmu.edu (Christian Lebiere) Date: Fri, 6 Oct 1995 12:06:22 -0400 (EDT) Subject: Poll: 1996 ACT-R Workshop and Summer School Message-ID: We have started planning the 1996 ACT-R Summer School and Workshop. As usual, the summer school will consist of an 8-to-10 day introduction to the theory and practice of ACT-R. Admission will be restricted to a small number of researchers, and stipends will be available. The two-day workshop will feature presentations on research and new developments. In addition, a session devoted to the introduction of new and advanced features of the system will take place between the summer school and workshop. It will last about 4 days and will be open to both summer school and workshop participants. At this stage, we would like feedback on which time from mid-May to early July would be most convenient for likely participants. Please reply directly to me (cl+ at cmu.edu). Thank you, Christian Lebiere From J.Segal at mcs.surrey.ac.uk Thu Oct 12 14:17:20 1995 From: J.Segal at mcs.surrey.ac.uk (Judith Segal) Date: Thu, 12 Oct 1995 14:17:20 -0400 Subject: Empirical studies of programmers Message-ID: Greetings. Some in the ACT-R community will be interested in this (sorry the fill everybody else's mailbox). Wayne **************************************************************************** EMPIRICAL STUDIES OF PROGRAMMERS: SIXTH WORKSHOP. WASHINGTON DC JAN 5TH - 7TH, 1996 ***************************************************************************** The ESP workshops are the premier forum in North America for the presentation of both field and laboratory studies of programmers. The edited volumes contain journal-length and journal-quality papers and represent much of the best work in this area. ************************** PROGRAMME **************************************** Friday 5th January 2.00 - 4.00 Pre-workshop tutorial: 'Zippy's TOE: a cognitive architecture of program design' Rob Rist 4.00 - 6.00 Registration 6.00 - 8.00 Opening Reception and Poster Session Saturday 6th January 7.15 - 8.00 Continental-style buffet breakfast and late registration 8.00 - 9.00 Career Contribution Award to, and Address by, Thomas Green (MRC Applied Psychology Unit, Cambridge, UK) Deconstraining Users: Weakening the strategy implications of programming environments. 9.00 - 10.00 Paper Session: Theories of Programming Robert Rist: System Structure and Design Deborah A. Boehm-Davis, Jean E. Fox & Brian H. Philips: Techniques for exploring program comprehension 10.00 - 10.30 Break 10.30 - 12.00 Paper Session: Transfer and Maintenance Susan Wiedenbeck & Jean Scholtz: Adaptation of programming plans in transfer between programming languages: A development approach Jawed Siddiqi, Rick Osborn, Chris Roast & Babak Khazaei: The pitfalls of changing programming paradigms. John Daly, Andrew Brooks, James Miller, Marc Roper & Murray Wood: Evaluating the effect of inheritance on the maintainability of object-oriented software 12.00 - 13.30 Lunch Invited Presentation: Francoise Detienne: Empirical research on object-oriented design: from individuals to teams 13.30 -15.30 Paper Session: Graphical representations and display-based problem solving and visual languages Judith Good: The 'right' tool for the task: An investigation of external representations, program abstractions, and task requirements. Simon P. Davies: Display-based problem solving strategies in computer programming Helen Hasan, Colin Jones & Edward Gould: Prototyping tools for expert and novice application development. Francesmary Modugno, Albert T. Corbett & Brad A. Myers: Evaluating program representation in a demonstrational visual shell 15.30 - 16.15 Break 16.15 - 17.45 PANEL Dennis R. Goldenson, Robert W. Stoddard, Victor R. Basili, Khaled El Eman & Carmel J. Trammell: The use of designed experiments in software engineering 20.30- Workshop Dinner and awards Sunday 7th January 7.45 - 8.30 Continental-style buffet breakfast 8.30 - 10.00 Paper Session: Teaching Programming Christopher M. Hoadley, Marcia C. Linn, Lydia M. Mann & Michael J. Clancy: When, why and how do novice programmers reuse code? Thomas C. Ormerod & Linden J. Ball: An empirical evaluation of Ted, a techniques editor for PROLOG programming Judith Segal: Learning about the algebraic specification of abstract data types 10.00- 10.30 Break 10.30 - 12.00 PANEL Thea Turner, Bill Curtis, Jim Herbsleb & Mike Atwood: Empirical studies of programming organisations ******************************************************************************** INVITED SPEAKERS. Thomas Green will be given the Career Contribution Award on Saturday. His interests include cognitive psychology and programming language design, interaction as an action language, cognitive dimensions of notations and devices, and models of information artefacts. He has inspired many by his great enthusiasm, and is truly a pillar of the community. Francoise Detienne, of INRIA, Paris, is co-Editor (with Rob Rist) of the recent special issue of HCI on Object-Oriented Programming. Her Invited Presentation will summarize the current state of research in this area. Rob Rist, of Sydney, Australia, has a paper in an upcoming issue of Cognitive Science on, "Zippy," his theory of everything (TOE) related to programming. His pre-workshop tutorial will preview the Cognitive Science paper and discuss features of his computational, cognitive architecture. ******************************************************************************** ACCOMMODATION ESP6 will be held at the Ramada Hotel, Alexandria, Virginia, about 15 minutes by Metro from the Capital and Museum district of Washington. The hotel provides free shuttle buses to and from Washington National Airport, and to and from the nearest Metro stop. Accommodation may be booked for ESP attendees at the specially negotiated price of $69/room/night, single or double. To book accommodation, please mention ESP to get the special price and phone (within USA) 1-800-272-6232, fax (USA) 703 683 7597 or write: Ramada Hotel Old Town, 901, N. Fairfax Street, Alexandria, VA 22314 USA ******************************************************************************** REGISTRATION FORM. Your first name _______________________ Your family name _________________ Your address ______________________________________________________ ______________________________________________________ _______________________________________________________ Will you be attending the pre-workshop tutorial: "Zippy's TOE: A unified theory of program design?" Yes/No [There is no extra charge for attending this tutorial] Do you require wheelchair access? Yes/No Do you have any special dietary requirements? No Yes, vegetarian Yes, Kosher Yes, other (please specify). Will you be arriving earlier than Friday?* Yes/No Will you be acccompanied by a spouse/partner?* or by child/ren?* Spouse/partner Yes/No Child/children Yes/No Do you require extra tickets for the Opening Reception on Friday 5th or the Workshop Dinner on Saturday 6th January at a cost of $45 per ticket? If yes, please state how many extra tickets are required for the Reception: Please state how many extra tickets are required for the Dinner: Please enclose a check/money order made payable to The ESP Foundation for $140 for students, $175 others. Add $45 for each extra ticket for the Opening Reception and/or Workshop Dinner. PLEASE RETURN TO: Dennis R. Goldenson, Software Engineering Institute, 4500 Fifth Avenue, 3rd floor, Pittsburgh, PA 15213-2691 USA The registration fee includes: the Opening Reception, continental breakfasts Saturday & Sunday, Saturday lunch, breaks, Workshop Dinner and paperback Proceedings. Extra and back copies of the Proceedings will be available for purchase. * There will be no formal program for early arrivals, or for accompanying spouses or children, but informal activities may be arranged. PLEASE NOTE: in order to book accommodation, please contact the hotel directly as above. ***************************************************************************** If you have queries, please contact a member of the organising committee: Wayne Gray (gray at gmu.ed) Co-chair Deborah Boehm-Davis (dbdavis at gmu.ed) Co-chair Dennis Goldenson (dg at sei.cmu.edu) Finance & Registration Judith Segal (j.segal at surrey.ac.uk) Publicity. ***************************************************************************** From niels at tcw2.ppsw.rug.nl Thu Oct 19 15:27:06 1995 From: niels at tcw2.ppsw.rug.nl (Niels A. Taatgen) Date: Thu, 19 Oct 95 15:27:06 MET Subject: Modelling CG in ACT-R and Soar (fwd) Message-ID: The following message appeared on the Soar mailing list, and is probably of interest to ACT-R people as well. Soarers (and maybe ACTers too): INTRODUCTION (Richard Young) In the summer of 1993, John Rieman wrote a model in ACT-R that simulates someone doing part of "the Cricket Graph task". We imagine someone who is an experienced Macintosh user, but has never used a graphing program before. They are asked to produce a graph from a given data file using the Cricket Graph (CG) program, without help. (John has videotapes of real subjects attempting exactly this task.) The ACT-R model simulates subjects using their knowledge of how to start the Word program to figure out, by analogy, how to start CG, and then making menu selections and choices from a dialogue box in order to produce the first cut at the graph. During the last 12 months John has been working with me and Andrew Howes to build a Soar model covering much the same ground. Whereas the emphasis with ACT-R was mainly on getting it to do the task, with the Soar model the focus was on getting it to exhibit the empirically observed phenomenon of "iteratively deepening attention" -- the observation that users repeatedly visit and re-visit menus, spending longer considering the options on the later passes. The Soar model is consequently at a finer grainsize than the ACT-R model, with an explicit representation of shifts of visual attention. On the other hand it doesn't get as far through the task: it makes the menu selections, but it doesn't deal with the dialogue box. Below are John's thoughts and a little data about working with the two architectures, followed by a few further comments from me. ============================================================================== Modelling CG in ACT-R and Soar John Rieman, 16 October 95 THE TWO MODELS The tables below compare the ACT-R model of the Cricket Graph task that I did for my dissertation with the Soar IDXL model I've completed this year. The purpose of the comparison is to evaluate effort versus results in the two environments. Please note that this is a VERY ROUGH comparison, which I slapped together in the last few hours before I left Cambridge. There could be some errors in the numbers, but they are pretty close to correct. The first table shows that the two models have similar core capabilities, with different extensions. The ACT-R model can handle the line-graph dialog box, but the Soar model can learn from instruction. The Soar model's visual scanning features put it at a lower grain size. Bottom line is that the models are about the same "size" in terms of task coverage. In the second table I look at the amount of code needed to implement the models. The most meaningful measure is "words," since that abstracts away the variety in variable-name length. On that measure, the complete Soar IDXL package, including simulation, is four times the size of the ACT-R model. If we take the simulation code out of both models (including the lisp productions that directly set WM in ACT-R), and also remove all comment lines, then the ratio drops to roughly two-to-one. It's hard to do a time-to-develop comparison, but here are some boundary conditions. I started working with ACT-R in May '93 and completed most of the model by early August. I spent a couple of weeks tying up loose ends in December. That's four months maximum to learn ACT-R and do the model, with other activities going on at the same time. The Soar work took a year, also interspersed with other activities. In both systems, I did some exploratory modelling of other domains that wasn't included in the final code. On the Soar project, I spent more time exploring other people's work, and also more time learning (and sometimes fighting!) the ever-changing Soar environment -- sde, tcl, nnpscm, Soar 6.x/y/z, etc. RE-USE A big potential advantage of working with Soar is that a lot of work has already been done in that architecture. During my year in Cambridge I learned much more about Soar and previous work than I ever knew about ACT-R. But not many of those ideas actually show up in the Soar CG model. It has a lot of ideas from Richard Young and Andrew Howes (recognition/recall learning, Cambridge i/o, unstacked subgoals). Our reuse of chunks is similar to Scott Huffman's, but I think we reached the point independently. The overall structure of scan/match/comprehend is similar to my prior work in ACT-R -- I assume there are similarly interactive models in Soar (Tac-Air?) but I didn't have time to look into them. And an important issue that worried me in the ACT-R framework, concept learning/extension/modification, still isn't addressed. We looked at the SCA work, but it was too hard to incorporate it into a standard Soar framework. Overall, it was difficult to build on prior work in the Soar community. The ideas are interesting, but many of them are very specific to Soar (i.e., how to do data chunking, or how to correct bad chunks), which makes them hard to report in papers for a non-Soar audience. And actually picking up someone else's system and reusing it (or even reimplementing it) is usually too much effort. Many times I'd decide to investigate an issue, I'd spend a week or two reading old Soar papers and writing test code, and then decide I didn't have time to redo someone's dissertation. So I'd hack together some simpler alternative, and say, "well, we COULD do it the other way, if we had more time." Of course, that has nasty impacts on learning, and it leaves analysis of interactions between the earlier work and our own to armchair analysis. But life is short. I also found it problematic that much of the earlier work is available only in technical reports and dissertations. The high-level message of these "insider" publications is often difficult for an outsider/novice to understand (a published paper would be forced to clarify some of that). And the details, which should be useful for reimplementation, are almost always specific to earlier versions of Soar. Balanced against those problems, of course, is the on-line accessibility of the Soar archives, and the willingness of the Soar community to help. Still, I wish I'd had a "Data Structures and Algorithms" text for Soar, pulling a lot of these ideas together into a more didactic form. A library of reusable routines -- NL, SCA, etc. -- would be another great resource. CONCLUSIONS Obviously there are a lot of factors to consider in deciding on an environment/architecture for cognitive modelling. On one dimension, the economics of my experience gives the nod to ACT-R: roughly half as much code in less than half the time, for roughly the same functionality. A more important measure would be something like "insights per lines of code." I can't say exactly what that was in Soar or ACT-R, but for my work in HCI, I think we could get more out of a still higher level environment. I wouldn't mind doing "our" part of the model in an architecture at the level of either Soar or ACT-R, if I could plug that into a system that already had NL, and a visual system, and a motor system, and concept learning, and a tie-in to a simulator, etc. But we have to do/redo all of those subsystems, and that's just too much effort for the payback. Just my opinion ... . ------------------------------------------------------------------------------ Feature Comparison: ACT-R and Soar CG Models ACT-R Soar Comments Task capabilities ----------------- Starts CG + + Selects line from the Graph menu + + Completes two lists in dialog box + Clicks OK in dialog box + + Cognitive capabilities ---------------------- Uses multipart task description + + Shifts task focus as work progresses + needed for 2-list d-box Scans environment + Label follows (identity match) + + Label follows synonyms + Envisions result + Maintains task-status record + + Learns recognition chunks + Learns by instruction + Learns by analogy + + built-in ACT; code in Soar Learns from error + ACT-R disables bad prods Iteratively deepening attention + Simulation ---------- Environment connects directly to WM + Environment connects to limited I/O + ------------------------------------------------------------------------------ Code in ACT-R and Soar CG Models (words) ACT-R Soar Code (with comments) 7215 24198 Code (w/o comments)* 3764 15028 Simulation code 1222 10663 Code w/o simulation 2542 4365 * "w/o comments" means that I grepped out all lines with a ";". I seldom put comments on the same line as code, so that's 99% correct. ============================================================================== CLOSING COMMENTS (Richard Young) Most of the gaps in the table of capabilities are not particularly significant. They simply reflect the different priorities of the two modelling efforts and what there wasn't time to complete. An interesting exception may be the "learns recognition chunks" entry. ACT models usually don't engage in the acquisition of numerous "small" recognition and comprehension chunks in the way that current Soar models do. The interesting speculation is that, where the Soar CG model uses productions to approximate rational exploratory behaviour, balancing the costs and benefits of different courses of action, in an ACT-R model we might expect to see iteratively deepening attention arise from the operation of the basic machinery for rational analysis, rather than being implemented by productions as in Soar. I agree about 98% with what John says. I think the remark about being able to plug the part of a model we are interested in "into a system that already had NL, and a visual system, and a motor system, and concept learning" and so on, is a little fanciful. It supposes that other people working on other parts of the overall cognitive system are in a more advanced and stable position than we ourselves are. It also assumes that the way that different cognitive facilities fit together is indeed by being "plugged in". I suspect that the way we will finally get the different parts of the cognitive model to work together is from the inside, as it were, rather than the outside: by iteratively re-working the partial models of NL, vision, concept learning, exploratory learning, and so on, until their constraints, representations, and basic processes are compatible and inter-operative. In other words, exactly the process of reimplementing and fitting together other people's ideas that John complains about having to do. And I don't think this is at all particular to Soar. Just my opinion ... . ============================================================================== -- --------------------------------------------------------------------------- Niels Taatgen Technische Cognitiewetenschap / Cognitive Science and Engineering Grote Kruisstraat 2/1 9712 TS Groningen, The Netherlands Email: n.a.taatgen at bcn.rug.nl WWW: http://tcw2.ppsw.rug.nl/~niels ---------------------------------------------------------------------------