TR on Factor Analysis Using Delta-Rule Wake-Sleep Learning

[Radford Neal]

                       Technical Report Available

         FACTOR ANALYSIS USING DELTA-RULE WAKE-SLEEP LEARNING

                           Radford M. Neal
           Dept. of Statistics and Dept. of Computer Science
                        University of Toronto

                             Peter Dayan
              Department of Brain and Cognitive Sciences
                Massachusetts Institute of Technology

                             24 July 1996

  We describe a linear network that models correlations between
  real-valued visible variables using one or more real-valued hidden
  variables - a *factor analysis* model.  This model can be seen as a
  linear version of the "Helmholtz machine", and its parameters can be
  learned using the "wake-sleep" method, in which learning of the
  primary "generative" model is assisted by a "recognition" model, whose
  role is to fill in the values of hidden variables based on the values
  of visible variables.  The generative and recognition models are
  jointly learned in "wake" and "sleep" phases, using just the delta
  rule.  This learning procedure is comparable in simplicity to Oja's
  version of Hebbian learning, which produces a somewhat different
  representation of correlations in terms of principal components.  
  We argue that the simplicity of wake-sleep learning makes factor 
  analysis a plausible alternative to Hebbian learning as a model of
  activity-dependent cortical plasticity.

This technical report is available in compressed Postscript by ftp to
the following URL:

          ftp://ftp.cs.toronto.edu/pub/radford/ws-fa.ps.Z

----------------------------------------------------------------------------
Radford M. Neal                                       radford at cs.utoronto.ca
Dept. of Statistics and Dept. of Computer Science radford at utstat.utoronto.ca
University of Toronto                     http://www.cs.utoronto.ca/~radford
----------------------------------------------------------------------------