Request for Boltzmann Machine information

[Geoffrey Hinton]

Lorien Pratt's message says that I implied that the Boltzmann machine deserves
no further study.  This is not what I believe.  It is certianly much slower
than back-propagation at many tasks and appears to scale more poorly than BP
when the depth of the network is increased.  So it would be silly to use it
instead of back-propagation for a typical current application.


However, and there may still be methods of improving the boltzmann machine
learning algorithm a lot.  For example, Anderson and Peterson have reported
that a mean-field version of the procedure learns much faster.  The mean-field
version is basically a learning procedure for Hopfield-Tank nets (which are
the mean-field version of Boltzmann machines).  It allows learning in Hopfield
-Tank nets that have hidden units.  

Also, the BM learning procedure is easier than BP to put directly into
hardware.  Alspector at bellcore has recently fabricated and tested a chip that
goes about 100,000 times faster than a minicomputer simulation.

Finally, the BM procedure can learn to model the higher-order statistics of
the desired state vectors of the output units.  BP cannot do this. 

In summary, the existing, standard Boltzmann machine learning procedure
is much slower than BP at tasks for which BP is applicable.  However,
the mean-field version may be closer in efficiency to BP, and other
developments are possible.


REFERENCES

The best general description of Boltzmann Machines is:

G. E. Hinton and T. J. Sejnowski
Learning and Relearning in Boltzmann machines
In D.~E. Rumelhart, J.~L. McClelland, and the~PDP~Research~Group,
Parallel Distributed Processing: {Explorations} in the
Microstructure of Cognition. {Volume I Foundations}}, 
MIT Press, Cambridge,  MA, 1986.

Some recent developments are described in section 7 of:

G. E. Hinton
"Connectionist Learning Procedures" 
Technical Report CMU-CS-87-115 (version 2)
Available from  computer science dept, CMU, Pittsburgh PA 15213.

The hardware implementation is described in:

J. Alspector and R.~B. Allen.
A neuromorphic VLSI learning system.
In P. Loseleben, editor, Advanced Research in VLSI:
Proceedings of the 1987 Stanford Conference.
MIT Press, Cambridge, Mass.,  1987.

The mean field lerning procedure is described in:

C. Peterson and J.~R. Anderson.
A Mean Field Theory Learning Algorithm for Neural Networks.
MCC Technical Report E1-259-87, Microelectronics and Computer
Technology Corporation, 1987.


Geoff