paper on ICA and LOCOCODE

[Josef Hochreiter]

                 Feature extraction through LOCOCODE                

   Sepp Hochreiter,  TUM                Juergen Schmidhuber, IDSIA       

   Neural Computation, in press   (28 pages, 0.7MB, 5MB gunzipped) 

LOw-COmplexity COding and DEcoding  (LOCOCODE)  is a novel approach to 
sensory coding and unsupervised learning.   Unlike previous methods it
explicitly takes into account the  information-theoretic complexity of 
the code generator:      it computes lococodes that convey information 
about the input data and can be computed and decoded by low-complexity 
mappings.  We implement LOCOCODE by training autoassociators with Flat
Minimum Search (Neural Computation 9(1):1-42, 1997),  a general method
for discovering low-complexity neural nets.     It turns out that this
approach can unmix an  unknown number of  independent  data sources by 
extracting a minimal number of  low-complexity features  necessary for
representing the data.   Experiments show:  unlike codes obtained with
standard autoencoders, lococodes are based on feature detectors, never
unstructured, usually sparse, sometimes factorial or local  (depending 
on statistical properties of the data).       Although LOCOCODE is not 
explicitly designed to enforce sparse or factorial codes,  it extracts 
optimal codes for  difficult  versions of the  bars benchmark problem, 
whereas ICA and PCA do not.        It produces familiar,  biologically 
plausible  feature  detectors  when applied to real world images,  and 
codes with fewer bits per pixel than ICA and PCA.   Unlike ICA it does 
not need to know the number of independent sources.  As a preprocessor 
for a  vowel  recognition  benchmark  problem it  sets  the  stage for 
excellent classification performance.            Our results reveil an 
interesting,  previously  ignored  connection  between  two  important 
fields: regularizer research, and ICA-related research.       They may 
represent  a  first  step  towards  unification  of regularization and 
unsupervised learning.

   ftp://ftp.idsia.ch/pub/juergen/lococode.ps.gz
   ftp://flop.informatik.tu-muenchen.de/pub/articles-etc/
   hochreiter.lococode.ps.gz

   http://www7.informatik.tu-muenchen.de/~hochreit/pub.html
   http://www.idsia.ch/~juergen/onlinepub.html

   Conference spin-offs:
1. Low-complexity coding and decoding. In K. M. Wong, I. King, D. 
   Yeung, eds., Proc. TANC'97, 297-306, Springer, 1997.
2. Unsupervised coding with LOCOCODE. In W. Gerstner, A. Germond, M. 
   Hasler, J.-D.  Nicoud, eds., ICANN'97, 655-660, Springer, 1997.
3. LOCOCODE versus PCA and ICA. In L. Niklasson, M. Boden, T. Ziemke,
   eds., ICANN'98, 669-674, Springer, 1998.
4. Source separation as a by-product of regularization.  
   To be presented at NIPS'98, 1998.

Sepp & Juergen