[ACT-R-users] model validation

[Rick Cooper]

Dear Fehmida,

I'm curious that no one has replied to your query - I've been hoping
that someone would, because you ask a good and difficult question. I
have comments rather than a definitive answer.

There are many cases where researchers report correlational statistics
for ACT-R (and other) models. Often the fits are remarkable - so good
that I view them with skepticism, and wonder how much jiggery-pockery
is going on underneath to get those fits. Sometimes there is parameter
fitting involved. Othertimes I suspect it is carefully crafted
productions. In any case, there are plenty of papers critiquing the
"good fits", as given by correlation or r^2 stats. (A good question to
ask is "Is the model's fit better than the fit we would obtain by
retesting subjects? If it is, then I feel justified in being
suspicious.)

I personally like replicating the experimental results - do the same
experiment with the model and establish whether the various main
effects and interactions are present (but we aware that the model is
likely to have less error variance than real subjects, unless you
really have tried to account seriously for individual differences).

But, in my opinion, the most convincing results come from doing
cross-validation. Split the data in two, fit the data to one half, and
then report the r^2 for the second half.

I hope this is of help - if nothing else it might stir someone else to
contribute.

Regards,

Rick Cooper


2008/9/14 Fehmida Hussain <hussain.fehmida at gmail.com>:
> Sorry for reposting, I was desperately looking for some advice on this:
>
> hi all,
>
> I have a query regarding model validation.
> I have implemented a few ACT-R 6 models of attentional networks simulating
> experimental studies. I have human data available to validate my model. the
> human study itself uses statistics like ANOVA to determine significane of
> the conditions and interactions between variables.
>
> Is is sufficient (from the point of view of model validation ) for me to
> just use Correl and SD to validate the model results against the human data
> or is it better to repeat the statistical test to show the same
> significane? What will be considered better from the point of view of
> defending my thesis?
>
> thanks
> Fehmida
>
> _______________________________________________
> ACT-R-users mailing list
> ACT-R-users at act-r.psy.cmu.edu
> http://act-r.psy.cmu.edu/mailman/listinfo/act-r-users
>
>



-- 
Rick Cooper, PhD
http://www.bbk.ac.uk/psyc/staff/academic/rcooper
Reader in Cognitive Science,
School of Psychology,
Birkbeck, University of London