Harris Drucker comments

[Michael P. Perrone]

Harris Drucker writes:
> In our work in OCR
> using multilayer networks (single layer networks are not powerful enough)
> boosting has ALWAYS improved performance. 

This is a direct result of averaging [1].


> Can someone find a better single network (using the original database) that is
> better than a boosted committee.  Maybe.  But good networks are hard to find and
> if you can find it, you can probably boost it.

This is the important take-home message for all of these averaging techniques:
If you can generate a good estimator, you can ALWAYS improve it using averaging.
Of course, you will eventually reach the point of diminishing returns on your
resource investment (e.g. averaging several different sets of averaged estimators
and then averaging the average of the averages ad infinitum).


> A final note:  rather than straight voting, we have found that simply summing
> the respective outputs of the three neural networks gives MUCH better results
> (as quoted above).  

This result is due to the fact that averaging the outputs is guaranteed to improve
performance for MSE whereas averaging the Winner Take All output (i.e. voting)
corresponds to a different optimization measure and there is no guarantee that 
averaging in one topology will improve the performance in the other [2].


[1] Michael P. Perrone and Leon N Cooper,
    When Networks Disagree: Ensemble Method for Neural Networks, 
    In _Neural Networks for Speech and Image Processing_,
    R. J. Mammone (ed.), Chapman-Hall, London: 1993).

[2] Michael P. Perrone and Leon N Cooper,
    Learning from what's been learned: Supervised learning in multi-neural network systems,
    Proceedings of the World Conference on Neural Networks 1993, INNS.


Michael
--------------------------------------------------------------------------------
Michael P. Perrone                                      Email: mpp at cns.brown.edu
Institute for Brain and Neural Systems                  Tel:   401-863-3920
Brown University                                        Fax:   401-863-3934
Providence, RI 02912