Announcement of newpaper, Information Geometry and EM algorithm

[Shun-ichi Amari]

The following paper is now available via anonymous ftp from the
neuroprose archive.  It is a technical report, METR 94-4, University of Tokyo
and will appear in "Neural Networks".
It consisits of two files,
am19.ps  for the main body (85 pages)
figs-ps  for the figures.


If you have any problems, contact to 
mura at sat.t.u-tokyo.ac.jp


---------
FTP-host: archive.cis.ohio-state.edu
FTP-file: pub/neuroprose/amari.geometryofem.tar.Z

This includes two files,  am19.ps and figs.ps

Use the unix command uncompress  +  tar
to uncompress and divide into two files.

---------

"Information Geometry of the EM and em Algorithms for Neural Networks"

by Shun-ichi Amari

In order to realize an input-output relation given 
by noise-contaminated examples, it is effective to 
use a stochastic model of neural networks.  A model 
network includes hidden units whose activation values 
are not specified nor observed. It is useful to estimate 
the hidden variables from the observed or specified 
input-output data based on the stochastic model. 
Two algorithms, the EM- and em-algorithms, have 
so far been proposed for this purpose. The EM-algorithm 
is an iterative statistical technique of using the 
conditional expectation, and the em-algorithm is a 
geometrical one given by information geometry.  The 
$em$-algorithm minimizes iteratively the Kullback-Leibler 
divergence in the manifold of neural networks. 
These two algorithms are equivalent in most cases. 
The present paper gives a unified information geometrical
framework  for studying stochastic models of neural networks,
by forcussing on the EM and em algorithms, and proves 
a condition  which guarantees their equivalence. 
Examples include 1) Boltzmann machines with hidden units, 
2) mixtures of experts, 3) stochastic multilayer perceptron, 
4) normal mixture model, 5) hidden Markov model, among others.