TR: reduce catastrophic interference w. context biasing

[Bob French]

The following paper is now available from the Ohio State neuroprose
archive.  It will be presented at the Cognitive Science Society
Conference in Atlanta in August.  It is six pages long.  The work
presented in this paper will be part of a larger paper on catastrophic
interference to appear later this fall.  Any comments will be welcome.


        Dynamically constraining connectionist networks
      to produce distributed, orthogonal representations 
              to reduce catastrophic interference 

                        Robert M. French 
                    Department of Psychology
                    University of Wisconsin
                       Madison, WI  53713

email:  french at head.neurology.wisc.edu
   or:  french at willamette.edu


It is now well known that when a connectionist network is trained on
one set of patterns and then attempts to add new patterns to its
repertoire, catastrophic interference may result.  The use of sparse,
orthogonal hidden-layer representations has been shown to reduce
catastrophic interference.  The author demonstrates that the use of
sparse representations not only adversely affects a network's ability
to generalize but may, in certain cases, also result in worse
performance on catastrophic interference.  This paper argues for the
necessity of maintaining hidden-layer representations that are both as
highly distributed and as highly orthogonal as possible.  The author
presents a fast recurrent learning algorithm, called context-biasing,
that dynamically solves the problem of constraining hidden-layer
representations to simultaneously produce good orthogonality and
distributedness.  On the data tested for this study, context-biasing
is shown to reduce catastrophic interference by more than 50% compared
to standard backpropagation.  In particular, this technique succeeds
in reducing catastrophic interference on data where sparse, orthogonal
distributions failed to produce any improvement.


Retrieve this paper by anonymous ftp from;
archive.cis.ohio-state.edu (128.146.8.52).
  in the pub/neuroprose  directory

The name of the paper in this archive is: 
   french.context-biasing.ps.Z


For those without ftp access, write to me at:

Robert M. French
Dept. of Psychology
University of Wisconsin
Madison, Wisconsin  53706

and I'll send you hard copy.