committees

[Wray Buntine]

I'm not convinced that the notion of an "unbiased estimator" is useful
here.  

It comes from classical statistics and is really a means of justifying
the choice of an estimator for lack of better ideas.  An estimator is
"unbiased" if the average of the estimator based on all the other
samples which we might have seen (but didn't) is equal to the "truth".

Notice that unbiased estimators and the use of Occam's razor conflict.
We all routinely throw away an "unbiased" neural network,
	i.e.  the best fitting network, 
in favor of a smoother, simpler network,
	i.e.  by early stopping, weight decay, ....,
which is very clearly "biased".

So I think its a great thing to be biased.  

One reason for averaging is because we have several quite different
biased networks that we think are reasonable, so like any good
gambler, we hedge our bets.  Of course, averaging is also standard
Bayesian practice, i.e. an obvious result of the mathematics.

----------
Wray Buntine
NASA Ames Research Center                 phone:  (415) 604 3389
Mail Stop 269-2                           fax:    (415) 604 3594
Moffett Field, CA, 94035-1000 		  email:  wray at kronos.arc.nasa.gov