semi-distributed representations

[Bob French]

One simultaneous advantage and disadvantage of fully 
distributed representations is that one representation 
will affect many others.  This phenomenon of interference
is what allows networks to generalize but it is also what leads
to the problem of catastrophic forgetting.

It is reasonable to suppose that the amount of interference 
in backpropagation networks is directly proportional to the amount
of overlap of representations in the hidden layer (the "overlap"
of two representations can be defined as the dot product of their
activation vectors).  The greater the overlap (i.e., the more
distributed the representations), the more the network will be
affected by catastrophic forgetting, but the better it will be at
generalizing.  The less the overlap (i.e., the more local
the representations), the less the network will be affected by
catastrophic forgetting, but the worse it will be at
generalizing.  

If we want nets that do not need to be retrained completely
when new data is presented to them but still retain their
ability to generalize, we must therefore use representations
that are neither too local, nor too distributed, what I have
called "semi-distributed" representations.  

I have a paper to appear in CogSci Proceedings 1991 that proposes 
this relationship between the amount of overlap of representations 
in the hidden layer and catastrophic forgetting and generalization.
The paper outlines one simple method that allows a BP network to 
evolve its own semi-distributed representations as it learns.

               - Bob French
               Center for Research on Concepts and Cognition
               Indiana University