[ACT-R-users] Models of listening to continuous speech?

[Martin Greaves]

Hi Bonnie

Further to Richard's comments, one of the key issues we were interested 
in was rehearsal of auditory items.  We attempted to model the word 
length effect in ACT-R using the audicon to represent a phonological 
loop-like structure.  Whereas the Huss & Byrne model used declarative 
memory for item storage and decay, and encoded list position within the 
item chunk, we wanted to see whether a phonological loop could be 
achieved using the intrinsic mechanisms of the audicon to account for 
order and decay.  However, we found that maintaining any form of 
rehearsal in the audicon to be difficult to implement.

One of the key problems is determining when the end of a list of items 
has been achieved, since ACT-R places 'rehearsed' items back in the 
audicon as unattended items.  Hence, once an item has been rehearsed, it 
is added to the end of the existing list of items with nothing to 
discriminate it from items in the current (unrehearsed) list of items.  
Consequently, the audicon does not allow discrimination of lists.  In 
order to get around this problem and determine when the end of the list 
had been reached we added an 'end' item.  As Richard said, this was 
argued as being analogous to a drawing of breath or a pause prior to any 
further rehearsal of items.

A further problem with the model was that it failed to reproduce serial 
order effects.  Although we obtained a good fit for word length, 
particularly for one-syllable words, we found that serial position 
curves were 'm' shaped rather than displaying extended primacy and 
single-item recency.  This was because items simply dropped out of 
memory if they were not rehearsed within the decay period, with no 
possibility of recovery due to the all-or-nothing decay in the audicon.  
Without intending to be in any way critical, this was not really 
surprising given the operation of the audicon.  However, rehearsal would 
not appear to be an issue in your case, so recall from the audicon 
should simply represent an extended recency effect.

In the running memory span models we took a slightly different approach 
to the model.  We assumed that storage might utilise a range of 
available short-term memory structures, with recall from whichever 
structures still contained items at recall.  This allowed continued use 
of the audicon but brings declarative  memory into play, since a 
spin-off from sub-vocal rehearsal is an item chunk in DM.  The audicon 
was considered it to operate simply as an extended buffer containing the 
last 2-3 items presented.  Thus, we recalled whatever was available from 
the audicon at the point of recall, with order implicit in recall, prior 
to recall of earlier items from declarative memory.  Using this approach 
we were able to account for a number of possible rehearsal strategies 
and what we termed rehearsal 'micro-strategies' (cf. Gray & Boehm-Davis, 
2000) reproducing a range of patterns of recall observed empirically.  
Were were also able to account for patterns of delayed recall when items 
in the audicon had long since decayed through recall from declarative 
memory.

Hope this is of help,

Martin






Richard M Young wrote:
> Hello Bonnie,
>
> Some years ago, following on from David Huss's work with Mike Byrne, 
> Martin Greaves and I played around with simple models using the 
> audicon to deal with lists of isolated words in a short-term memory 
> experiment.  At least in our hands, the audicon was assumed to hold 
> "words" in some unspecified phonemic/phonological/articulatory code, 
> which therefore required an access to DM/LTM in order to retrieve the 
> word as a lexical item.
>
> As I remember it -- which is not well -- we found it a bit clunky to 
> use the audicon to deal with ephemeral material spread out in time. 
> With no disrespect to those who had worked on it, I think it's fair to 
> say that not much thought had been given to the underlying design of a 
> transducer dealing with speech sounds.  For example, I *think* we had 
> to add an END signal to indicate the silence following the last item 
> in a list.
>
> Martin later build partial models of the running memory span task 
> using a similar approach, and included them in his PhD thesis.  He may 
> be able to add to what I'm saying here.
>
> Word recognition in speech is a well-studied area in cognitive 
> psychology, and there are some good (ad hoc) models around, and I 
> believe a reasonable amount of consensus.  It had crossed my mind from 
> time to time that it would be an interesting area to bring into 
> contact with Act-R.  There are of course Act-R models of the lexical 
> retrieval stage itself.
>
> Good luck!
>
> ~ Richard
>
> At 11:18 -0400 11/8/09, Bonnie John wrote:
>> Folks,
>>
>> Anyone have an models using ACT-R's Audition Module to listen to and
>> comprehend continuous speech?
>> If not continuous speech, how about short phrases or anything other than
>> tones?
>> Any experience using the Audition Module at all?
>>
>> I'm asking because I'd like to make models of ACT-R using the JAWS
>> screen reader to navigate a web site and compare it to ACT-R models of
>> visually navigating a web site. I'd like to read papers or speak with
>> someone who has had experience with the Audition Module to get a
>> heads-up on peaks and pitfalls.
>>
>> I did browse the Publications page, but could only find papers that use
>> vision (but I could have missed the listening ones - apologies if I 
>> did).
>>
>> Thanks,
>> Bonnie