paper available: Mean Field Theory for Sigmoid Belief Networks

[Lawrence Saul]

FTP-host: psyche.mit.edu
FTP-file: pub/lksaul/belief.ps.Z

The following paper is now available by anonymous ftp.

==================================================================

Mean Field Theory for Sigmoid Belief Networks (12 pages)

Lawrence K. Saul, Tommi Jaakkola, and Michael I. Jordan
Center for Biological and Computational Learning
Massachusetts Institute of Technology
79 Amherst Street, E10-243
Cambridge, MA 02139
 
Abstract:

Bayesian networks (a.k.a. belief networks) are stochastic
feedforward networks of discrete or real-valued units.  
In this paper we show how to calculate a rigorous lower 
bound on the likelihood of observed activities in sigmoid 
belief networks.  We view these networks in the framework 
of statistical mechanics and derive a mean field theory 
for the average activities of the units.  The advantage of 
this framework is that the mean field free energy gives a
rigorous lower bound on the log-likelihood of any partial
instantiation of the network's activity.  The feedforward
directionality of belief networks gives rise to terms that 
do not appear in the mean field theory for symmetric networks 
of binary units.  Nevertheless, the mean field equations have 
a simple closed form and can be solved by iteration to yield 
a lower bound on the likelihood.  Empirical results suggest 
that this bound may be tight enough to serve as a basis for 
inference and learning.

==================================================================