What have neural networks achieved?

[Jay McClelland]

Max Coltheart <max at currawong.bhs.mq.edu.au> writes:
> Randy O'Reilly said:
> 
>    Interestingly, one of the "sucesses" of neural networks in this case
>    was their dramatic failure in the form of the catestrophic
>    interference phenomenon.  This failure tells us something about the
>    limitations of the cortical memory system, and thus, why we might need
>    a hippocampus.
>    
> Think about the structure of this argument for a moment. It runs thus:
> 
> 1. Neural networks suffer from catastrophic interference.
> 2. Therefore the cortical memory system suffers from catastrophic
> interference.
> 3. That's why we might need a hippocampus.
> 
> Is everyone happy with the idea that (1) implies (2)?

Randy may not have provided quite a full enough description of the
observations we made in the McClelland, McNaughton and O'Reilly
article concerning what we called 'Complementary Learning Systems' in
hippocampus and neocortex.  The argument is quite a bit richer than
Max's comment suggests, but I will endeavor to summarize it
(for full justification and demonstration simulations, see
the paper).  The arguement, based on the successes as well as the
failures of connectionist models of learning and memory, was this:

The discovery of the structure present in large ensembles of events
and experiences, such as e.g., the structure present in the relations
between spelling and sound, requires what we called 'interleaved
learning' --- learning in which the connection weights are adapted
gradually so that the overall structure present in the ensemble can
guide the learning process.  It also requires the use of a
componential coding scheme, which is essential for good generalization
(this theme also appears in the Plaut et al paper, mentioned in my
previous post in this discussion).  We claimed the neocortex was
specialized for structure-sensitive learning, and we observed that
neural networks that exhibit this form of learning WOULD exhibit
catastrophic interference IF forced to learn quickly, either by
turning up the learning rate or by massive repetition of a very small
and thus necessarily non-representative sample of training cases.
Basically, what's simply happening is that the network is learning the
non-representative structure present in the sample, at the expense of
whatever it might previously have learned.

Max and others might be interested to know that cortical memory
systems have been shown to suffer from catestrophic-interference like
effects.  Massive repetition of a couple of tactile stimuli spanning
several fingers can destroy the topographic map in somatosensory
cortex (this is research from Merzenich's group).  Generally, however,
the cortex avoids catestrophic interference by using a relatively
small learning rate, so that, in the normal course of events, the
weights will reflect a sufficient sample of the environment.  

To allow rapid learning of the contents of a particular experience,
the arguement goes, a second learning system, complementary to the
first, is needed; such a system has a higher learning rate and recodes
inputs using what we call 'sparse, random conjunctive coding' to
minimize interference (while simultaneously reducing the adequacy of
generalization).  These characteristics are just the ones that appear
to characterize the hippocampal system: it is the part of the brain
known to be crucial for the rapid learning of the contents of specific
experiences; it is massively plastic; and neuronal recording studies
indicate that it does indeed use sparse, random conjunctive coding.

Citations for the relevant articles follow.  

 -- Jay McClelland

--------------------------------------

@Article{McClellandMcNaughtonOReilly95,
  author =	"McClelland, J. L. and McNaughton, B. L. and O'Reilly, R. C." ,
  year	=	"1995" ,
  title	=	"Why there are complementary learning systems in the 
		 hippocampus and neocortex: {Insights} from the successes and 
		 failures of connectionist models of learning and
		  memory" ,
 journal=	"Psychological Review",
  volume =	"102",
  pages =	"419-457"
}

@Article{OReillyMcClelland94,
  author = 	 "O'Reilly, R. C. and McClelland, J. L.",
  title = 	 "Hippocampal conjunctive encoding, storage, and
		  recall:  {Avoiding} a tradeoff",
  journal =	 "Hippocampus",
  year = 	 1994,
  volume =	 4,
  pages =	 "661-682"
}

@Article{McClellandGoddard96,
author	=	"McClelland, J. L. and Goddard, N. H." ,
year	=	"1996" ,
title	=	"Considerations arising from a complementary learning
		  systems perspective on hippocampus and neocortex" ,
journal	=	"Hippocampus" ,
volume	=	"6" ,
pages	=	"654--665"
}

@Article{PlautlETAL96,
  author = 	 "Plaut, D. C. and McClelland, J. L. and Seidenberg,
		  M. S. and Patterson, K. E.",
  title = 	 "Understanding Normal and Impaired Word Reading: 
                  {Computational} Principles in Quasi-Regular Domains",
  journal =      "Psychological Review",
  volume =       "103",
  pages =        "56-115",
  year = 	 "1996"
}

Sorry, I'm not sure the correct citation for the Merzenich finding
mentioned.  It may be:

@Article{WangMerzenichSameshimaJenkins95,
  author = 	 {Wang, X. and Merzenich, M. M. and Sameshima, K. and
		  Jenkins, W. M.},
  title = 	 {Remodelling of hand representation in adult cortex
		  determined by timing of tactile stimulation},
  journal = 	 {Nature},
  year = 	 {1995},
  volume = 	 {378},
  pages = 	 {71-75}
}