Combining Generalizers

[Shimon Edelman]

Kagan Tumer <kagan at pine.ece.utexas.edu> wrote:

> Lately, there has been a great deal of interest in combining
> estimates, and especially combining neural network outputs.
> Combining has a LONG history, with seminal ideas contained
> in Selfridge's Pandemonium (1958) and Nilsson's book on Learning 
> ...
> Another important question is, how much benefit (%age, limits, reliability,..)
> can combining methods yield. At least two recent PhD theses
> (Perrone, Hashem) mathematically address this issue for REGRESSION problems,
> 
> We have approached this problem for CLASSIFICATION problems
> by studying the effect of combining on the 
> decision boundaries. The results pinpoint the mechanism by
> which classification results are improved, and provide limits,
> including a new way of estimating Bayes' rate.
> ...

To me, Selfridge's Pandemonium looks more like a concept that launched
a thousand variations on the Winner-Take-All motif than a precursor of
combining estimates (of course, non-maximum suppression can also be
counted as a mode of combining estimates, albeit not a very productive
one :-). Nevertheless, it can serve as a basis for building a powerful
representation of the subspace of the input space relevant to the
task, if the Master Demon outputs the vector of response strengths of
each of its subordinates, rather than the identity of the one that
happens to shout the loudest(*). I think this is a good example of the
importance of REPRESENTATION: the ensemble of responses is a much
richer representation than the identity of the strongest response, and
is more likely to constitute a better basis for CLASSIFICATION. In the
past couple of years, I have been trying to clarify the computational
basis of the effectiveness of representation by a bank of
(individually poorly tuned) classifiers. Most of the results of this
project to date are available via my Web page.

(*) Shimon Edelman, "Representation, Similarity, and the Chorus of
Prototypes", Minds and Machines 5:45-68 (1995), 
ftp://eris.wisdom.weizmann.ac.il/pub/mam.ps.Z
See also the recent works by Jonathan Baxter, available from
Neuroprose (sorry, I don't have an URL handy). 

-Shimon

Shimon Edelman, Applied Math & CS, Weizmann Institute
http://www.wisdom.weizmann.ac.il/~edelman/shimon.html
Cyber    Rights    Now:    Accept    No    Compromise