The Helmholtz Machine and the Wake-Sleep Algorithm

[Peter Dayan]

FTP-host: ftp.cs.toronto.edu
FTP-filename: pub/dayan/wake-sleep.ps.Z
FTP-filename: pub/dayan/helmholtz.ps.Z

Two papers about stochastic and deterministic Helmholtz machines are
available in compressed postscript by anonymous ftp from
ftp.cs.toronto.edu - abstracts are given below.

We regret that hardcopies are not available. 

----------------------------------------------------------------------

pub/dayan/wake-sleep.ps.Z

      The Wake-Sleep Algorithm for Unsupervised Neural Networks
				   
  Geoffrey E Hinton, Peter Dayan, Brendan J Frey and Radford M Neal

Abstract:

We describe an unsupervised learning algorithm for a multilayer
network of stochastic neurons.  Bottom-up "recognition" connections
convert the input into representations in successive hidden layers and
top-down "generative" connections reconstruct the representation in
one layer from the representation in the layer above.  In the "wake"
phase, neurons are driven by recognition connections, and generative
connections are adapted to increase the probability that they would
reconstruct the correct activity vector in the layer below.  In the
"sleep" phase, neurons are driven by generative connections and
recognition connections are adapted to increase the probability that
they would produce the correct activity vector in the layer above.


Submitted for publication

----------------------------------------------------------------------

pub/dayan/helmholtz.ps.Z

			The Helmholtz Machine

  Peter Dayan, Geoffrey E Hinton, Radford M Neal and Richard S Zemel

Abstract:

Discovering the structure inherent in a set of patterns is a
fundamental aim of statistical inference or learning.  One fruitful
approach is to build a parameterised stochastic generative model,
independent draws from which are likely to produce the patterns.  For
all but the simplest generative models, each pattern can be generated
in exponentially many ways. It is thus intractable to adjust the
parameters to maximize the probability of the observed patterns, We
describe a way of finessing this combinatorial explosion by maximising
an easily computed lower bound on the probability of the observations.
Our method can be viewed as a form of hierarchical self-supervised
learning that may relate to the function of bottom-up and top-down
cortical processing pathways.


Neural Computation, in press.

----------------------------------------------------------------------