Modelling CG in ACT-R and Soar (fwd)
Niels A. Taatgen
niels at tcw2.ppsw.rug.nl
Thu Oct 19 15:27:06 EDT 1995
The following message appeared on the Soar mailing list, and is
probably of interest to ACT-R people as well.
Soarers (and maybe ACTers too):
INTRODUCTION
(Richard Young)
In the summer of 1993, John Rieman wrote a model in ACT-R that simulates
someone doing part of "the Cricket Graph task". We imagine someone who is an
experienced Macintosh user, but has never used a graphing program before.
They are asked to produce a graph from a given data file using the Cricket
Graph (CG) program, without help. (John has videotapes of real subjects
attempting exactly this task.) The ACT-R model simulates subjects using
their knowledge of how to start the Word program to figure out, by analogy,
how to start CG, and then making menu selections and choices from a dialogue
box in order to produce the first cut at the graph.
During the last 12 months John has been working with me and Andrew Howes to
build a Soar model covering much the same ground. Whereas the emphasis with
ACT-R was mainly on getting it to do the task, with the Soar model the
focus was on getting it to exhibit the empirically observed phenomenon of
"iteratively deepening attention" -- the observation that users repeatedly
visit and re-visit menus, spending longer considering the options on the
later passes. The Soar model is consequently at a finer grainsize than the
ACT-R model, with an explicit representation of shifts of visual attention.
On the other hand it doesn't get as far through the task: it makes the menu
selections, but it doesn't deal with the dialogue box.
Below are John's thoughts and a little data about working with the two
architectures, followed by a few further comments from me.
==============================================================================
Modelling CG in ACT-R and Soar
John Rieman, 16 October 95
THE TWO MODELS
The tables below compare the ACT-R model of the Cricket Graph task that I
did for my dissertation with the Soar IDXL model I've completed this year.
The purpose of the comparison is to evaluate effort versus results in the two
environments.
Please note that this is a VERY ROUGH comparison, which I slapped together in
the last few hours before I left Cambridge. There could be some errors in
the numbers, but they are pretty close to correct.
The first table shows that the two models have similar core capabilities,
with different extensions. The ACT-R model can handle the line-graph dialog
box, but the Soar model can learn from instruction. The Soar model's visual
scanning features put it at a lower grain size. Bottom line is that the
models are about the same "size" in terms of task coverage.
In the second table I look at the amount of code needed to implement the
models. The most meaningful measure is "words," since that abstracts away
the variety in variable-name length. On that measure, the complete Soar IDXL
package, including simulation, is four times the size of the ACT-R model. If
we take the simulation code out of both models (including the lisp
productions that directly set WM in ACT-R), and also remove all comment
lines, then the ratio drops to roughly two-to-one.
It's hard to do a time-to-develop comparison, but here are some boundary
conditions. I started working with ACT-R in May '93 and completed most of
the model by early August. I spent a couple of weeks tying up loose ends in
December. That's four months maximum to learn ACT-R and do the model, with
other activities going on at the same time. The Soar work took a year, also
interspersed with other activities. In both systems, I did some exploratory
modelling of other domains that wasn't included in the final code. On the
Soar project, I spent more time exploring other people's work, and also more
time learning (and sometimes fighting!) the ever-changing Soar environment --
sde, tcl, nnpscm, Soar 6.x/y/z, etc.
RE-USE
A big potential advantage of working with Soar is that a lot of work has
already been done in that architecture. During my year in Cambridge I
learned much more about Soar and previous work than I ever knew about ACT-R.
But not many of those ideas actually show up in the Soar CG model. It has a
lot of ideas from Richard Young and Andrew Howes (recognition/recall
learning, Cambridge i/o, unstacked subgoals). Our reuse of chunks is similar
to Scott Huffman's, but I think we reached the point independently. The
overall structure of scan/match/comprehend is similar to my prior work in
ACT-R -- I assume there are similarly interactive models in Soar (Tac-Air?)
but I didn't have time to look into them. And an important issue that
worried me in the ACT-R framework, concept learning/extension/modification,
still isn't addressed. We looked at the SCA work, but it was too hard to
incorporate it into a standard Soar framework.
Overall, it was difficult to build on prior work in the Soar community. The
ideas are interesting, but many of them are very specific to Soar (i.e., how
to do data chunking, or how to correct bad chunks), which makes them hard to
report in papers for a non-Soar audience. And actually picking up someone
else's system and reusing it (or even reimplementing it) is usually too much
effort. Many times I'd decide to investigate an issue, I'd spend a week or
two reading old Soar papers and writing test code, and then decide I didn't
have time to redo someone's dissertation. So I'd hack together some simpler
alternative, and say, "well, we COULD do it the other way, if we had more
time." Of course, that has nasty impacts on learning, and it leaves analysis
of interactions between the earlier work and our own to armchair analysis.
But life is short.
I also found it problematic that much of the earlier work is available
only in technical reports and dissertations. The high-level message of
these "insider" publications is often difficult for an outsider/novice to
understand (a published paper would be forced to clarify some of that).
And the details, which should be useful for reimplementation, are almost
always specific to earlier versions of Soar. Balanced against those problems,
of course, is the on-line accessibility of the Soar archives, and the
willingness of the Soar community to help. Still, I wish I'd had a
"Data Structures and Algorithms" text for Soar, pulling a lot of these
ideas together into a more didactic form. A library of reusable routines --
NL, SCA, etc. -- would be another great resource.
CONCLUSIONS
Obviously there are a lot of factors to consider in deciding on an
environment/architecture for cognitive modelling. On one dimension, the
economics of my experience gives the nod to ACT-R: roughly half as much code
in less than half the time, for roughly the same functionality.
A more important measure would be something like "insights per lines of
code." I can't say exactly what that was in Soar or ACT-R, but for my work
in HCI, I think we could get more out of a still higher level environment. I
wouldn't mind doing "our" part of the model in an architecture at the level
of either Soar or ACT-R, if I could plug that into a system that already had
NL, and a visual system, and a motor system, and concept learning, and a
tie-in to a simulator, etc. But we have to do/redo all of those subsystems,
and that's just too much effort for the payback.
Just my opinion ... .
------------------------------------------------------------------------------
Feature Comparison: ACT-R and Soar CG Models
ACT-R Soar Comments
Task capabilities
-----------------
Starts CG + +
Selects line from the Graph menu + +
Completes two lists in dialog box +
Clicks OK in dialog box + +
Cognitive capabilities
----------------------
Uses multipart task description + +
Shifts task focus as work progresses + needed for 2-list d-box
Scans environment +
Label follows (identity match) + +
Label follows synonyms +
Envisions result +
Maintains task-status record + +
Learns recognition chunks +
Learns by instruction +
Learns by analogy + + built-in ACT; code in Soar
Learns from error + ACT-R disables bad prods
Iteratively deepening attention +
Simulation
----------
Environment connects directly to WM +
Environment connects to limited I/O +
------------------------------------------------------------------------------
Code in ACT-R and Soar CG Models (words)
ACT-R Soar
Code (with comments) 7215 24198
Code (w/o comments)* 3764 15028
Simulation code 1222 10663
Code w/o simulation 2542 4365
* "w/o comments" means that I grepped out all lines with a ";". I seldom
put comments on the same line as code, so that's 99% correct.
==============================================================================
CLOSING COMMENTS
(Richard Young)
Most of the gaps in the table of capabilities are not particularly
significant. They simply reflect the different priorities of the two
modelling efforts and what there wasn't time to complete. An interesting
exception may be the "learns recognition chunks" entry. ACT models usually
don't engage in the acquisition of numerous "small" recognition and
comprehension chunks in the way that current Soar models do. The interesting
speculation is that, where the Soar CG model uses productions to
approximate rational exploratory behaviour, balancing the costs and benefits
of different courses of action, in an ACT-R model we might expect to see
iteratively deepening attention arise from the operation of the basic
machinery for rational analysis, rather than being implemented by productions
as in Soar.
I agree about 98% with what John says. I think the remark about being able
to plug the part of a model we are interested in "into a system that already
had NL, and a visual system, and a motor system, and concept learning" and so
on, is a little fanciful. It supposes that other people working on other
parts of the overall cognitive system are in a more advanced and stable
position than we ourselves are. It also assumes that the way that different
cognitive facilities fit together is indeed by being "plugged in". I suspect
that the way we will finally get the different parts of the cognitive model
to work together is from the inside, as it were, rather than the outside: by
iteratively re-working the partial models of NL, vision, concept learning,
exploratory learning, and so on, until their constraints, representations,
and basic processes are compatible and inter-operative. In other words,
exactly the process of reimplementing and fitting together other people's
ideas that John complains about having to do. And I don't think this is at
all particular to Soar.
Just my opinion ... .
==============================================================================
--
---------------------------------------------------------------------------
Niels Taatgen
Technische Cognitiewetenschap / Cognitive Science and Engineering
Grote Kruisstraat 2/1
9712 TS Groningen, The Netherlands
Email: n.a.taatgen at bcn.rug.nl
WWW: http://tcw2.ppsw.rug.nl/~niels
---------------------------------------------------------------------------
More information about the ACT-R-users
mailing list