Papers on ensemble learning

[Anders Krogh]

Dear connectionists,

the following paper is now available from our web page
(http://www.ph.ed.ac.uk/~pkso/papers/EnsemblePREVII.ps.gz) :

STATISTICAL MECHANICS OF ENSEMBLE LEARNING

Anders Krogh and Peter Sollich
(Physical Review E, 55:811-825, 1997)

Abstract

Within the context of learning a rule from examples, we study the
general characteristics of learning with ensembles.  The generalization
performance achieved by a simple model ensemble of linear students is
calculated exactly in the thermodynamic limit of a large number of input
components and shows a surprisingly rich behavior.  Our main findings
are the following.  For learning in large ensembles, it is advantageous
to use underregularized students, which actually overfit the training
data.  Globally optimal generalization performance can be obtained by
choosing the training set sizes of the students optimally.  For smaller
ensembles, optimization of the ensemble weights can yield significant
improvements in ensemble generalization performance, in particular if
the individual students are subject to noise in the training process.
Choosing students with a wide range of regularization parameters makes
this improvement robust against changes in the unknown level of
corruption of the training data.


An abbreviated version of this paper appeared in NIPS 8 and is also
available (http://www.ph.ed.ac.uk/~pkso/papers/EnsembleNIPSVI.ps.gz).
Both papers are also available from http://www.cbs.dtu.dk/krogh/refs.html.

Further papers of potential interest to readers of connectionists can be
found on our home pages:

* Peter Sollich (http://www.ph.ed.ac.uk/~pkso/publications): learning
  from queries, online learning, finite size effects in neural networks,
  ensemble learning

* Anders Krogh (http://www.cbs.dtu.dk/krogh/): Most current work on
using hidden Markov models in `computational biology.'


All comments and suggestions are welcome -

Anders Krogh and Peter Sollich

--------------------------------------------------------------------------
 Peter Sollich                           Department of Physics
                                         University of Edinburgh
 e-mail: P.Sollich at ed.ac.uk              Kings Buildings
 phone: +44 - (0)131 - 650 5293          Mayfield Road
 fax:   +44 - (0)131 - 650 5212          Edinburgh EH9 3JZ, U.K.
--------------------------------------------------------------------------

Anders Krogh

Center for Biological Sequence Analysis (CBS)
Technical University of Denmark
Building 206
DK-2800 Lyngby
DENMARK

Phone:  +45 4525 2470
Fax:    +45 4593 4808
E-mail: krogh at cbs.dtu.dk
_____________________________________________