Multiple Models, Committee of nets etc...

[Michael P. Perrone]

Joydeep Ghosh writes:
> in our experiments, the difference between simple averaging
> and the best among other arbitration mechanisms does not
> seem statistically significant, thus supporting Waibel and
> Hampshire's observations.  The combination of
> networks trained on different feature vectors, on the other
> hand leads to 15-25% reduction in errors on a very difficult data set.

The result I discussed in a previous posting (that there is a 1/n relation
between the MSE of the averaged estimator and the avg. population MSE)
helps explain this result in the following terms:

   Averaging is more effective when the estimates are more distinct.

Thus in the example that Joydeep gives, the fact that different
features where used to generate different estimates suggests that those
estimates will be distinct (unless the features carry the same information).
also we have the advantage that using fewer features, we can use smaller
nets which helps avoid problems like over-fitting and the curse of dimensionality.

-Michael
--------------------------------------------------------------------------------
Michael P. Perrone                                      Email: mpp at cns.brown.edu
Institute for Brain and Neural Systems                  Tel:   401-863-3920
Brown University                                        Fax:   401-863-3934
Providence, RI 02912