Paper on on-line EM algorithm for NRBF

[Shin Ishii]

The following paper is available on my web site:

http://mimi.aist-nara.ac.jp/~ishii/publication.html

We would greatly appreciate comments and suggestion.

-------------------------------------------------------------------
On-line EM algorithm for the normalized Gaussian network

	Masa-aki Sato and Shin Ishii

	To appear in Neural Computation

A Normalized Gaussian Network (NGnet) (Moody and Darken 1989) is
a network of local linear regression units. The model softly
partitions the input space by normalized Gaussian functions and
each local unit linearly approximates the output within the
partition. In this article, we propose a new on-line EM algorithm
for the NGnet, which is derived from the batch EM algorithm
(Xu, Jordan and Hinton 1995) by introducing a discount factor.

We show that the on-line EM algorithm is equivalent to the batch
EM algorithm if a specific scheduling of the discount factor is
employed. In addition, we show that the on-line EM algorithm can
be considered as a stochastic approximation method to find the
maximum likelihood estimator.
A new regularization method is proposed in order to deal with a
singular input distribution. In order to manage dynamic
environments, where the input-output distribution of data changes
over time, unit manipulation mechanisms such as unit production,
unit deletion, and unit division are also introduced based on the
probabilistic interpretation.

Experimental results show that our approach is suitable for 
function approximation problems in dynamic environments.
We also apply our on-line EM algorithm to robot dynamics problems
and compare our algorithm with the Mixtures-of-Experts family.
-------------------------------------------------------------------

Shin Ishii
Nara Institute of Science and Technology
ATR Human Information Processing Research Laboratories