2-parameter search in ACT-R (e.g. Slow Kendler)

[Richard M Young]

I have some observations about the process of "2D parameter search", and a
fairly dramatic example of the landscape being searched.  I'll try and keep
this note fairly short.  Its purpose is just to give a summary, and to ask
if it rings any bells for anyone and see if anyone has any comments or
further examples, or can point me to existing discussion.

This investigation came about in the following way.  For my undergraduate
class in Cognitive Modelling (final-years honours students in Cognitive
Science), for coursework I ask them, in part, to adapt the model for the
Fast Kendler Ss given in Chapter 4 of the 1998 book, to be a model of the
Slow Kendler Ss -- that's basically the exercise in Section 8 of the ACT-R
on-line tutorial.  The exercise includes a search  to optimise the values
of two parameters in order to minimise the deviation (RMSD) of the model
predictions from the empirical data.  The parameters are (1) the :egs noise
setting, and (2) the number of :eventual-successes on newly acquired rules,
which for brevity I'll refer to as the ":rule" parameter.

I noticed that students were finding apparently optimal fits in very
different regions of the parameter space.  For example, some were finding
minimal values of around 1.0-1.5 near the point (:rule=20, :egs=0.60),
while others were apparently finding values of 0.5-1.0 near (:rule=90,
:egs=0.15).

To understand what was going on, I undertook a careful search, and sketched
some contour maps to show the structure of the space.  The result surprised
me.  There is no single optimum (i.e. "lowest point").  Instead, the low
parts of the parameter space form a long, narrow valley with a level floor.
For much of its length, the valley is very narrow -- indeed, it's more
like a gorge, with very steep sides.  For higher values of the :rule
parameter -- say, above 50 -- the valley is virtually straight and almost
parallel to the :rule axis, just narrowing further and converging slightly
with the axis at higher values.  Below a :rule of 50 (down to 1), the
valley curves round to be more parallel to the :egs axis.

This valley has a couple of features which I find remarkable (and would
like to understand better).  One is how level its floor is.  The
lowest-level contour I could consistently trace is for RMSD value of 1.8,
and that holds for almost the entire length.  The contour is clearly still
there at :rule=200, though it perhaps disappears by around :rule=500, with
the valley floor rising to around 2.0.  At the "near" end, for values of
:rule of 5 or less, there are hints of a 1.5-level contour -- so I suppose,
strictly speaking, this is the "best fitting" part of the space -- and the
valley floor does rise slightly, to around 2.5, at its very end for
:rule=1.  But the dominant finding is how level it is.  It would be like
finding a gorge you could walk along the bottom of for tens of miles
without any change of height.

The second feature is how narrow the valley is, especially for higher :rule
values.  For example, at :rule=200, the 1.8 contour is only 0.005 of an
:egs unit wide.

This landscape has implications, both practical and theoretical, for 2D
parameter search.  For instance:

(a) The usual informal method for 2-parameter search -- certainly what I
teach in class, and I think what is done in the ACT summer school -- works
fine if the landscape takes of the form of a nicely-shaped basin, but it is
not adequate for searching a structure such as the one I've described.

(b) With much of the valley so narrow, it can be difficult to find without
a "map" or some other guide.

(c) The lack of a reasonable minimum fit falsifies the main assumption
behind parameter-search -- namely, that the data determine a set of
parameter values.  In this case, they don't.  Furthermore, it could be
argued that it also undermines the case for the model as a plausible
account of the cognitive processes in the task, since there is no set of
parameter values which can be justified as being "correct".

Comments are welcome.  I'll try to write up a report on this investigation
and make it available, thought it's hard to see how it could be published
since it would be of interest to hardly anyone outside the ACT community.

-- Richard