Connectionists: Two new papers on generative models

[Geoffrey Hinton]

Preprints of the following two papers are available at 
http://www.cs.toronto.edu/~hinton/

"A FAST LEARNING ALGORITHM FOR DEEP BELIEF NETS"
Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh

                      ABSTRACT

We show how to use ``complementary priors'' to eliminate the
explaining away effects that make inference difficult in
densely-connected belief nets that have many hidden layers.  Using
complementary priors, we derive a fast, greedy algorithm that can
learn deep, directed belief networks one layer at a time, provided the
top two layers form an undirected associative memory. The fast, greedy
algorithm is used to initialize a slower learning procedure that
fine-tunes the weights using a contrastive version of the wake-sleep
algorithm. After fine-tuning, a network with three hidden layers forms
a very good generative model of the joint distribution of handwritten
digit images and their labels. This generative model gives 
better digit classification than the best discriminative learning
algorithms. The low-dimensional manifolds on which the digits lie are
modelled by long ravines in the free-energy landscape of the top-level
associative memory and it is easy to explore these ravines by using
the directed connections to display what the associative memory has in
mind.
(submitted to Neural Computation)

_______________________________________________________________


"INFERRING MOTOR PROGRAMS FROM IMAGES OF HANDWRITTEN DIGITS"
          Geoffrey Hinton and Vinod Nair

                       ABSTRACT

We describe a generative model for handwritten digits that uses
two pairs of opposing springs whose stiffnesses are controlled by
a motor program. We show how neural networks can be trained to
infer the motor programs required to accurately reconstruct the
MNIST digits. The inferred motor programs can be used directly
for digit classification, but they can also be used in other
ways. By adding noise to the motor program inferred from an MNIST
image we can generate a large set of very different images of the
same class, thus enlarging the training set available to other
methods. We can also use the motor programs as additional, highly
informative outputs which reduce overfitting when training a
feed-forward classifier.
(submitted to NIPS)