CRL TR-9101: The importance of starting small

[Jeff Elman]

The following technical report is available.  Hardcopies may be
obtained by sending your name and postal address to crl at crl.ucsd.edu.

A compressed postscript version can be retrieved through ftp
(anonymous/ident) from crl.ucsd.edu (128.54.165.43) in the file
pub/neuralnets/tr9101.Z. 


                 CRL Technical Report 9101

                 "Incremental learning, or
             The importance of starting small"

                      Jeffrey L. Elman

              Center for Research in Language
      Departments of Cognitive Science and Linguistics
            University of California, San Diego
                     elman at crl.ucsd.edu

                          ABSTRACT

     Most work in learnability theory assumes that both  the
environment  (the  data  to  be  learned)  and  the learning
mechanism are static.  In the  case  of  children,  however,
this  is an unrealistic assumption.  First-language learning
occurs, for example, at precisely that point  in  time  when
children undergo significant developmental changes.

     In this paper I describe the results of simulations  in
which   network models are unable to learn a complex grammar
when both the network and the input remain unchanging.  How-
ever,  when either the input is presented incrementally, or-
-more realistically--the network begins with limited  memory
that  gradually  increases, the network is able to learn the
grammar.  Seen in this light, the  early  limitations  in  a
learner  may  play  both  a positive and critical role,  and
make it possible to master a body of knowledge  which  could
not be learned in the mature system.