2-parameter search in ACT-R (e.g. Slow Kendler)

[Wheeler Ruml]

> [...]  if it rings any bells for anyone and see if anyone has any
> comments or further examples, or can point me to existing
> discussion. [...]  Instead, the low parts of the parameter space
> form a long, narrow valley with a level floor.

This reminds me of the explorations I did of Gary Dell et al's model
of lexical access during speech production (Psych Review, 2000, 107(3)
pp609--634, see especially 620-623, figures 9-11).  I found a long,
narrow valley of parameter settings that gave similar fits.  In this
case, this was a problem because some of the good fits occurred at
settings that should have produced poor behavior.

> (a) The usual informal method for 2-parameter search [...] is not
> adequate for searching a structure such as the one I've described.

Like Roman, I found it important to use an automated technique when
optimizing the fit - it would be quite tedious and difficult to detect
the gradient by hand.  I'm not familiar with Roman's method, but the
one I used is described briefly in the paper (and I've also used it
successfully in subsequent work).  I suspect it will scale better to
higher-dimensional searches than an annealing-based approach, since it
tries to adapt somewhat to the valley structure.  It is also careful
to minimize the number of model simulations needed to estimate the fit
at a particular setting.  (I can try to make Common Lisp code
available if that would be useful for anyone.)

I would love to hear from folks about automated methods they use to
fit simulation models to data or studies of such methods - someday I'd
like to do a comprehensive study if one hasn't already been done!

> (c) The lack of a reasonable minimum fit falsifies the main
> assumption behind parameter-search -- namely, that the data
> determine a set of parameter values.  In this case, they don't.

Couldn't you just say that the data determine a set of settings?
Since there is sampling error in the human data (and perhaps the model
too), at best the data are just defining a distribution over the
settings anyway.

> Furthermore, it could be argued that it also undermines the case for
> the model as a plausible account of the cognitive processes in the
> task, since there is no set of parameter values which can be
> justified as being "correct".

Hmmm - I'm not following this.  Why do the parameter values have to be
unique, as long as there exists some that match the data?  If the
model can match anything, that's a different problem (a la Roberts &
Pashler), and if it produces the same human-like behavior at many
settings, that just means that there are fewer real degrees of freedom
in the model than there are parameters, which seems harmless.  And in
practice, as Niels points out, there are surely other constraints that
could be used to help constrain the choice.

Best wishes,

Wheeler
-- 
Wheeler Ruml, http://www.eecs.harvard.edu/~ruml/, 617/495-2081 voice
ruml at eecs.harvard.edu, Maxwell Dworkin Lab 219    617/496-1066 fax