Technical report announcement

[David.Servan-Schreiber@A.GP.CS.CMU.EDU]

The following technical report is available upon request:

   ENCODING SEQUENTIAL STRUCTURE IN SIMPLE RECURRENT NETWORKS
  David Servan-Schreiber, Axel Cleeremans & James L. McClelland
                          CMU-CS-88-183

We explore a network architecture introduced by Elman (1988)  for
predicting  successive  elements  of a sequence. The network uses
the pattern of activation over a set of hidden units  from  time-
step  t-1,  together with element t, to predict element t+1. When
the network is trained with strings  from  a  particular  finite-
state  grammar,  it  can  learn  to  be  a  perfect  finite-state
recognizer for the grammar. When the net has a minimal number  of
hidden  units, patterns on the hidden units come to correspond to
the nodes of the grammar; however,  this  correspondence  is  not
necessary  for  the  network  to  act  as  a perfect finite-state
recognizer. We explore the conditions under which the network can
carry  information  about distant sequential contingencies across
intervening elements to distant  elements.  Such  information  is
maintained   with  relative  ease  if  it  is  relevant  at  each
intermediate step; it tends to be lost when intervening  elements
do  not  depend on it. At first glance this may suggest that such
networks  are  not  relevant  to  natural  language,   in   which
dependencies  may  span indefinite distances. However, embeddings
in natural language are not  completely  independent  of  earlier
information.  The  final  simulation  shows  that  long  distance
sequential contingencies can be encoded by the  network  even  if
only  subtle statistical properties of embedded strings depend on
the early information.

Send surface mail to :

    Department of Computer Science
    Carnegie Mellon University
    Pittsburgh, PA. 15213-3890
    U.S.A

or electronic mail to Ms. Terina Jett:

    Jett at CS.CMU.EDU     (ARPA net)

Ask for technical report CMU-CS-88-183.