Reprint: Constructive Learning of Recurrent Neural Networks

[Lee Giles]

The following reprint is available via the NEC Research
Institute ftp archive external.nj.nec.com. Instructions for
retrieval from the archive follow the summary.

----------------------------------------------------------------------------------


       "Constructive Learning of Recurrent Neural Networks"

   D. Chen, C.L. Giles, G.Z. Sun, H.H. Chen, Y.C. Lee, M.W. Goudreau

              University of Maryland, College Park
              NEC Research Institute, Princeton, NJ

                     ABSTRACT

Recurrent neural networks are a natural model for learning and predicting 
temporal signals. In addition, simple recurrent networks have been shown 
to be both theoretically and experimentally capable of learning 
finite state automata {cleeremans89,giles92a,minsky67,pollack91,
siegelmann92}. However, it is difficult to determine what is the minimal 
neural network structure for a particular automaton. Using a large recurrent 
network, which would be versatile in theory, in practice proves to be very 
difficult to train. Constructive or destructive recurrent methods might offer 
a solution to this problem. We prove that one current method, Recurrent Cascade 
Correlation, has fundamental limitations in representation and thus in its 
learning capabilities. We give a preliminary approach on how to get around 
these limitations by devising a ``simple" constructive training method 
that adds neurons during training while still preserving the powerful fully 
recurrent structure. Through simulations we show that such a method can learn 
many types of regular grammars that the Recurrent Cascade Correlation method is 
unable to learn. 

-------------------------------------------------------------------------------------

                          FTP INSTRUCTIONS

                unix> ftp external.nj.nec.com
                Name: anonymous
                Password: (your_userid at your_site)
                ftp> cd pub/giles/papers
                ftp> binary
                ftp> get icnn_93_contructive.ps.Z
                ftp> quit
                unix> uncompress icnn_93_contructive.ps.Z


----------------------------------------------------------------------------------------

--                                 
C. Lee Giles / NEC Research Institute / 4 Independence Way
Princeton, NJ 08540 / 609-951-2642 / Fax 2482
==