committees, agent teams, redundancy, Monte Carlo, ...

[Wray Buntine]

Some recent postings on connectionists have spurred me to draw
a few comparisons with earlier work.  I'm always amazed at when a
new discovery is made, about 10 people will rediscover variations
of the same thing and often times be too busy (and admirably) exploring 
the frontiers of research to relate their work to the current pool
of research.

The following stuff may seem different when you get into the fine
detail, but they look mighty similar from a distance.

	committees (Thodberg '93, MacKay '93, both in connectionists)
	agent teams (Beyer & Smieja in connectionists as beyer.teams.ps),
	stacked generalization (Wolpert in Neural networks 5(2) '92, and
		         	Breiman in a TR from UC Berkeley Stats, '93)
	model averaging (Buntine and Weigend, Complex Systems, 5(1), '91,
			 and on learning decision trees see Buntine,
			 Statistics and Computing, v2, '92 who got the
			 idea from S. Wok and C. Carter in UAI-87)
	Monte Carlo methods (surely this stuff is related!! see recent 
			     papers by Radford Neal, e.g. NIPS5)
	error correcting codes (Dietterich and Bakiri, AAAI-91)
	redundant knowledge  (M. Gams, 4th EWSL '89)
	
	+ probably lots more
	

e.g	MacKay's version of committee's which got the energy prediction
	prise smells like Breiman's version of Wolpert's stacked
	generalization

e.g.    Thodberg's committee's is a clever & pragmatic implementation
	of the model averaging approach suggested in Buntine and
	Weigend which itself is a cheap description of a standard
	Bayesian trick

e.g.    in the statistical community i'm told "model uncertainty"
	is all the rage in some circles, i.e.  you don't return
	           a single network but several for comparison
	and early work goes back many decades

I find this fascinating because a few years ago we were all 
rediscovering smoothing (weight decay, weight elimination, regularisation,
MDL, early stopping, cost complexity tradeoffs, etc., etc. etc.) 
in its various shapes and forms.

Now we all seem to be rediscovering the use of multiple models,
i.e.  the next step in sophisticated learning algorithms.
NB.   i use the term "rediscovering" because I wouldn't dare
	attribute the discovery to any one individual !!!

Nice work !!  What's next ?

------------
Wray Buntine
NASA Ames Research Center                 phone:  (415) 604 3389
Mail Stop 269-2                           fax:    (415) 604 3594
Moffett Field, CA, 94035-1000 		  email:  wray at kronos.arc.nasa.gov