New TR on unsupervised learning

[Juergen Schmidhuber]

    LEARNING FACTORIAL CODES BY PREDICTABILITY MINIMIZATION 
                        ..
                       Jurgen Schmidhuber 
                 Department of Computer Science 
                     University of Colorado 
	
       (Compact version of Technical Report CU-CS-565-91)


                            ABSTRACT

I present  a novel  general  principle for  unsupervised  learning of 
distributed non-redundant internal  representations of input patterns  
or input sequences.  With a given set of representational units, each 
unit  tries to react  to the environment  such that it  minimizes its 
predictability  by an  adaptive  predictor  that sees  all the  other 
units. This encourages each unit to filter `abstract concepts' out of 
the environmental  input such that  these concepts are  statistically 
independent of those  upon which the  other units focus.   I  discuss 
various  simple  yet  potentially  powerful  implementations  of  the 
principle which aim at finding binary factorial codes (Barlow, 1989}, 
i.e. codes  where  the probability of  the occurrence of a particular 
input is simply the product of the probabilities of the corresponding 
code symbols.  Unlike  previous  methods  the novel  principle  has a 
potential  for removing  not only  linear but also  non-linear output 
redundancy.  Methods for finding  factorial codes automatically embed 
Occam's razor  for finding  codes  using  a minimal  number of  units.  
Illustrative experiments show  that algorithms based on the principle 
of predictability  minimization are  practically feasible.  The final 
part of this paper  describes an entirely  local algorithm that has a 
potential for learning  unique representations of extended sequences.  


---------------------------------------------------------------------

To obtain a copy, do:

             unix>         ftp archive.cis.ohio-state.edu
             Name:         anonymous
             Password:     neuron
             ftp>          binary
             ftp>          cd pub/neuroprose
             ftp>          get schmidhuber.factorial.ps.Z
             ftp>          bye
             unix>         uncompress schmidhuber.factorial.ps.Z
             unix>         lpr  schmidhuber.factorial.ps

---------------------------------------------------------------------

There is no hardcopy mailing list.  I will read my mail only 
occasionally during the next three weeks or so. 

 ..
Jurgen