sensory coding with LOCOCODE

[Josef Hochreiter]

                            LOCOCODE

    Sepp Hochreiter,  TUM              Juergen Schmidhuber, IDSIA       

    TR FKI-222-97 (19 pages, 23 figures, 450 KB, 4.2 MB gunzipped) 

Low-complexity coding and decoding  (LOCOCODE)  is a novel approach to
sensory coding  and unsupervised learning.  Unlike previous methods it
explicitly takes into account the  information-theoretic complexity of 
the code generator:  lococodes (1) convey information  about the input 
data and  (2) can be computed and decoded by  low-complexity mappings. 
We implement  LOCOCODE  by training autoassociators with  Flat Minimum 
Search, a recent, general method for discovering low-complexity neural 
nets. Experiments show:  unlike codes obtained with standard autoenco-
ders,  lococodes are based on  feature detectors,  never unstructured, 
usually sparse,  sometimes factorial or local (depending on the data). 
Although  LOCOCODE's  objective function does not  contain an explicit 
term enforcing  sparse or  factorial codes,  it extracts optimal codes 
for difficult versions of the "bars" benchmark problem.  Unlike, e.g., 
independent component analysis (ICA) it does not need to know the num-
ber of  independent data sources.  It produces  familiar, biologically 
plausible  feature detectors  when applied to real world images.  As a 
preprocessor  for a  vowel recognition  benchmark problem  it sets the 
stage for excellent classification performance.

         ftp://ftp.idsia.ch/pub/juergen/lococode.ps.gz
         ftp://flop.informatik.tu-muenchen.de/pub/fki/fki-222-97.ps.gz

         http://www7.informatik.tu-muenchen.de/~hochreit/pub.html
         http://www.idsia.ch/~juergen/onlinepub.html

(invited talk at "Theoretical Aspects of Neural Computation" (TANC97), 
Hong Kong, May 97 - short spin-off papers to be published by Springer)

Comments welcome.                                       Sepp & Juergen