Multiple Models, Committee of nets etc...

[Mahesan Niranjan]

  >From: Wray Buntine <wray at ptolemy.arc.nasa.gov>
  >Subject:  RE:  committees, agent teams, redundancy, Monte Carlo, ...
  >Date: Wed, 21 Jul 1993 01:57:31 GMT
  >
  [...]
  >e.g     MacKay's version of committee's which got the energy prediction
  >        prise smells like Breiman's version of Wolpert's stacked
  >        generalization
  [...]
  >Now we all seem to be rediscovering the use of multiple models,
  >i.e.  the next step in sophisticated learning algorithms.
  [...]

An interesting and somewhat easy to understand application of multiple
models is in the area of target-tracking (e.g. Bar-Shalom & Fortmann,
'Tracking and Data Association', Academic Press 1988, ISBN 0-12-079760).
They show how to run several models in parallel, recursively estimating
them with Kalman filtering and use the innovation probabilities for
model selection. Apart from terminology (like you dont see the term
"evidence" there), a lot of the ideas are in that framework too; but the
assumptions etc are much clearer (at least to me), and the language
not so strong. We have used this method to track parametric models of
highly nonstationary signals (e.g. Formants in speech).

The committee of networks doing the energy prediction (committee members
chosen by ranking models by performance on cross-validation set, and the
average performance of these being better than the best member) is a
somewhat surprising result to me. Surprising because, the average predictions
are taken without weighting by the model probabilities (which are difficult
to compute). In practice, even for linear models in Gaussian noise, I find
probabilities tend to differ by large numbers, for models that look
very similar. Hence if these are difficult to evaluate and are assumed
equal, I would have expected the average performance to be worse than the
best member.

In real life too, committees tend to be less efficient than the good
individual members (when you give the members equal say), but thats a
different story :-)

niranjan