bayesian methods for back-propagation, update

[Wray Buntine]

To appear in December 1991 issue of {\it Complex Systems}.
An early draft appeared in the Neuroprose archive July 1991,
and was available as NASA Ames AI Research Branch
TR FIA-91-22.  The new version is considerably improved and
contains new, updated and corrected material.

A limited number of reprints are available. by writing to
Wray Buntine.  (Andreas is currently on holidays!)   Please
only request one if your library doesn't get Complex Systems
journal.  

PS.   distributing the first draft via connectionists allowed us
      to get all sorts of helpful feedback on the early version!

-------------

		Bayesian Back-Propagation

  Wray L. Buntine 			Andreas S. Weigend
  wray at ptolemy.arc.nasa.gov		andreas at psych.stanford.edu
  RIACS \& NASA Ames Research Center	Xerox Palo Alto Research Center 
  Mail Stop 269-2			3333 Coyote Hill Rd.
  Moffet Field, CA 94035, USA		Palo Alto, CA, 94304, USA

Connectionist feed-forward networks, trained with back-propagation,
can be used both for non-linear regression and for (discrete
one-of-$C$) classification.  This paper presents approximate Bayesian
methods to statistical components of back-propagation: choosing a cost
function and penalty term (interpreted as a form of prior
probability), pruning insignificant weights, estimating the
uncertainty of weights, predicting for new patterns
(``out-of-sample''), estimating the uncertainty in the choice of this
prediction (``error bars''), estimating the generalization error,
comparing different network structures, and handling missing values in
the training patterns. These methods extend some heuristic techniques
suggested in the literature, and in most cases require a small
additional factor in computation during back-propagation, or
computation once back-propagation has finished.