committees are bad

[John B. Hampshire II]

Belief in committees is paradoxically based on the notion that
each member of the committee is a biased estimator of
the Bayes-optimal classifier --- I stress that I am restricting
my comments to pattern classification; I'm not commenting on
function approximation (e.g., regression).  Regression (i.e.,
estimating probabilities) and classification are not the same
thing.  The idea behind committees is that the average of a bunch
of biased estimators will constitute an unbiased estimator.
This is a provably *bad* idea, absent a proof that the
biases all cancel (I'll bet there is no such proof in
any of the committee work).  Nevertheless, committees are
obviously popular because the classifiers we typically
generate in the connectionist community are provably
biased --- even with regularization, pruning, and all the
other complexity reduction tricks.

Put in more organic terms, committees of humans often
comprise large numbers of unremarkably average, biased
individuals:  their purpose is to achieve
what one remarkable, unbiased individual could do alone.  By
virtue of their number, they generally involve
huge development and maintenance overhead.
This is a waste of resources.  Compensating for one biased
committee member with another one that has a different
bias generally gives us a committee with lots of bias
rather than one with no bias.  The United States Congress
is a perfect illustrative example:  consider each member as
a biased estimator of the ideal politician, and consider
how effective the average of their efforts is...

Barak certainly makes a valid point re. the initial
parameterization issue, although it is also not
that important if your model is minimum (or near minimum)
complexity --- this gets into the issue of estimation variance,
versus that of estimation bias.

I'll take a single *provably* unbiased classifier over
a committee of biased ones any day.

Vapnik is right: excessive complexity is anathema.
So are Geman, Bienenstock, and Doursat: connectionists
face a "bias/variance dilemma".  Fortunately, there
is a relatively simple way to generate unbiased,
low-complexity, minimum-variance classifiers.

For those who care, I am prepared to defend this post
with supporting proofs.  However, I won't do it over
connectionists in deference to those who don't care.


-John