TR announcement: Long Short Term Memory

[Josef Hochreiter]

FTP-host:  flop.informatik.tu-muenchen.de (131.159.8.35)
FTP-filename: /pub/fki/fki-207-95.ps.gz    
or
FTP-host: fava.idsia.ch (192.132.252.1)
FTP-filename: /pub/juergen/fki-207-95.ps.gz     
or something like
netscape http://www.idsia.ch/~juergen


                     LONG SHORT TERM MEMORY
        
            Technical Report FKI-207-95 (8 pages, 50 K) 


        Sepp Hochreiter                  Juergen Schmidhuber

      Fakultaet fuer Informatik                IDSIA
   Technische Universitaet Muenchen        Corso Elvezia 36
      80290 Muenchen, Germany          6900 Lugano, Switzerland


  ``Recurrent backprop'' for learning to store information over 
  extended time periods takes too long. The main reason is 
  insufficient, decaying error back flow.  We describe a novel, 
  efficient ``Long Short Term Memory'' (LSTM) that overcomes 
  this and related problems. Unlike previous approaches, LSTM 
  can learn to bridge arbitrary time lags by enforcing constant 
  error flow. Using gradient descent, LSTM explicitly learns when 
  to store information and when to access it. In experimental 
  comparisons with ``Real-Time Recurrent Learning'', ``Recurrent 
  Cascade-Correlation'', ``Elman nets'', and ``Neural Sequence 
  Chunking'', LSTM leads to many more successful runs, and learns 
  much faster. Unlike its competitors, LSTM can solve tasks 
  involving minimal time lags of more than 1000 time steps, even 
  in noisy environments.


If you don't have gzip/gunzip,  we can mail you an uncompressed 
postscript  version  (as a  last resort). Comments welcome.


Sepp Hochreiter
Juergen Schmidhuber