Combining Gaussian Mixture Density Estimates

[Dirk Ormoneit]

In a recent message, Anders Krogh mentioned the possibility to
use averaging based on slightly different (e.g. resampled)
training sets as a method of regularization.
In our latest work, we found that this regularizing effect of 
network averaging may be advantageously exploited for Gaussian 
mixture density estimation. Regularization is particularly 
important in this case, because the overfitting problem is even 
more severe than, for example, in the regression case. In our 
experiments we found that Leo Breiman's *bagging* (averaging of 
estimators which were derived from resampled training sets) yields 
a performance which is comparable and sometimes even superior to 
a  Bayesian regularization approach. As pointed out by Breiman, a 
basic precondition for obtaining an improvement with *bagging* is 
that the individual estimators are relatively unstable. This is 
particularly the case for Gaussian mixture estimates.
The title of our paper is

        Improved Gaussian Mixture Density Estimates Using Bayesian
                Penalty Terms and Network Averaging

                by Dirk Ormoneit and Volker Tresp


                                ABSTRACT

We compare two regularization methods which can be used to improve
the generalization capabilities of Gaussian mixture density estimates.
The first method consists of defining a Bayesian prior distribution on
the parameter space. We derive EM (Expectation Maximization) update
rules  which  maximize the a posterior parameter probability in
contrast to the usual EM rules for Gaussian mixtures which  maximize
the likelihood function. In the second approach we apply ensemble
averaging to density estimation. This includes Breiman's "bagging",
which has recently been found to produce impressive results for
classification networks. To our knowledge this is the first time that
ensemble averaging is applied to improve density estimation.


A version of this paper is submitted to NIPS'95. A technical report is 
available under the name 'fki-205-95.ps.gz' on the FTP-site 

	flop.informatik.tu-muenchen.de

To get it, execute the following steps:

% ftp flop.informatik.tu-muenchen.de
Name (flop.informatik.tu-muenchen.de:ormoneit): anonymous
Password:  (your email adress)
ftp> cd pub/fki
ftp> binary
ftp> get fki-205-95.ps.gz
ftp> bye
% gunzip fki-205-95.ps.gz

Dirk