TR: Fast Learning of On-line EM Algorithm

[Masa-aki SATO]

The following paper is available on my web site:

http://www.hip.atr.co.jp/~masaaki/

We would greatly appreciate comments and suggestion.

 TITLE:   "Fast Learning of On-line EM Algorithm"
                           Masa-aki Sato
ATR Human Information Processing Research Laboratories

------------------------------------------------------------------
                       Abstract

  In this article, an on-line EM algorithm is derived for general
Exponential Family models with Hidden variables (EFH models).
It is proven that the on-line EM algorithm is equivalent to 
a stochastic gradient method with the inverse of the Fisher 
information matrix as a coefficient matrix.
As a result, the stochastic approximation theory guarantees 
the convergence to a local maximum of the likelihood function.

  The performance of the on-line EM algorithm is examined
by using the mixture of Gaussian model,
which is a special type of the EFH model. 
The simulation results show that the on-line EM algorithm
is much faster than the batch EM algorithm and the on-line
gradient ascent algorithm.

  The fast learning speed is achieved by the systematic
design of the learning rate schedule.
Moreover, it is shown that the on-line EM algorithm can escape
from a local maximum of the likelihood function in the early 
training phase, even when the batch EM algorithm is trapped 
to a local maximum solution. 

  It is pointed out that the on-line EM algorithm has a similar form as 
the natural gradient method proposed by Amari (1998),  which 
gives the optimal asymptotic convergence. 
The inverse of the Fisher information matrix in the on-line EM 
algorithm may contribute to fast learning performance. 
In our on-line EM algorithm, however, it is not necessary to calculate 
the inverse of the Fisher information matrix.
In the future, it would be interesting to study the relation of 
our algorithm to the natural gradient method.

--------------------------------------
Masa-aki Sato
ATR Human Information Processing Research Laboratories
2-2, Hikaridai, Seika-cho, Soraku-gun
Kyoto 619-0288 Japan

phone : 0774-95-1039
fax   : 0774-95-1008
E-mail: masaaki at hip.atr.co.jp