[ACT-R-users] experiments to validate the behavior of a model

[Rhiannon L Weaver]

Hi Florian,

Are you talking about how many model simulations to run, or how much human 
data to collect to compare to simulations?  If it is a question of model 
simulations, it depends on how you want to characterize the behavior.  In 
essence, each time the model runs, you can record any number of 
characteristics of that run in order to get a profile of the behavior. Say 
you record J items for each model run.  So eg, total time, number of 
turns, number of corrections, when they see the cloud, whether or not they 
avoid it, etc...

If you are interested in averaging this behavior across multiple runs, and 
characterizing the model by this average, then there is some standard 
multivariate statistical analysis that you can apply to do a "sample size" 
calculation-- in this sense, it would be the number of model runs you need 
to do in order to estimate the mean value of your J items, and how they 
interact (eg, the full covariance matrix), within a certain level of 
statistical confidence.  See for example, Johnson and Wichern, "Applied 
Multivariate Statistical Analysis" and the chapters on estimating 
multivariate means and profile analysis assuming Gaussian distributions. 
This is the book I'm familiar with, but it can be rather dense.  A 
less-theoretical approach might be offered in Afifi, Clark, and May, 
"Computer-aided multivariate analysis", though I'm just going by reviews 
and a perusal of the table of contents.

In any case I recommend looking at the full distribution of the data 
points across model runs in addition to the mean values, if only to see 
whether you can assume that they are distributed as Multivariate Normal or 
not.

The methods are similar for compiling human data, but because you have 
less control over the parameters, you might need to think of sample size 
within subjects.  Ie, if you suspect, for example, that you have people of 
different baseline skill levels, you might want to get several repetitions 
of the task per person to take that into account, if you are interested in 
a within-subjects characterization of behavior.

Best,
Rhiannon

On Tue, 16 Sep 2008, act-r-users-request at act-r.psy.cmu.edu wrote:

> Send ACT-R-users mailing list submissions to
> 	act-r-users at act-r.psy.cmu.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://act-r.psy.cmu.edu/mailman/listinfo/act-r-users
> or, via email, send a message with subject or body 'help' to
> 	act-r-users-request at act-r.psy.cmu.edu
>
> You can reach the person managing the list at
> 	act-r-users-owner at act-r.psy.cmu.edu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ACT-R-users digest..."
>
>
> Today's Topics:
>
>   1. experiments to validate behaviour of a cognitive model
>      (Florian Frische)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 16 Sep 2008 13:36:24 +0200
> From: Florian Frische <Florian.Frische at offis.de>
> Subject: [ACT-R-users] experiments to validate behaviour of a
> 	cognitive model
> To: act-r-users at act-r.psy.cmu.edu
> Message-ID:
> 	<OFDCE8D3F3.6C3ABEE3-ONC12574C6.003F7D42-C12574C6.003FE6BA at offis.uni-oldenburg.de>
>
> Content-Type: text/plain; charset="US-ASCII"
>
> Hi all,
>
> I would like to ask you about the amount of experimental data in order to
> validate a model's behaviour.
> I think there are several parameters that have an effect on the amount of
> data (suspects, trials) that is needed.
> The following 2 examples should help to describe what this question is
> about:
>
> Example 1: Let's say we have a 2-armed bandit that always returns -1 on
> the one arm and 1 on the other arm. We Would like
> to validate the behaviour of our model towards the behaviour of gamblers
> (The strategy that they use to maximize the overall outcome)
>
> Example 2: We have a complex flight task where a pilot has to avoid a
> thunderstorm. We would like to validate the behaviour
> of our pilot model in this task towards real pilots behaviour.
>
> It is obvious that example 1 is less complex than example 2 and I think
> that we need much more experimental data to get reliable results
> for the second example. I suppose there is a relationship beetween task
> complexity (e.g. independent variables) and the number of suspects/number
> of trials.
> But how can I assess how many suspects/trials I need (How many
> participants and how many repeats)?
>
> Thanks a lot for participating in this discussion,
>
> Florian Frische
>
> OFFIS
> FuE Bereich Verkehr | R&D Division Transportation
> Escherweg 2 - 26121 Oldenburg - Germany
> Phone.: +49 441 9722-523
> E-Mail: florian.frische at offis.de
> URL: http://www.offis.de
>
>
>
> ------------------------------
>
> _______________________________________________
> ACT-R-users mailing list
> ACT-R-users at act-r.psy.cmu.edu
> http://act-r.psy.cmu.edu/mailman/listinfo/act-r-users
>
>
> End of ACT-R-users Digest, Vol 38, Issue 11
> *******************************************
>