TR: Ace-of-Bayes with ARD

[Hans Henrik Thodberg]

The following 33 pages long manuscript is now available by ftp:
FTP-host: archive.cis.ohio-state.edu
FTP-filename: /pub/neuroprose/thodberg.bayes-ard.ps.Z
or URL (WWW):
ftp://archive.cis.ohio-state.edu/pub/neuroprose/thodberg.bayes-ard.ps.Z
Hardcopies are not avaliable.
----------------------------------------------------------------------------
               A Review of Bayesian Neural Networks 
        with an Application to Near Infrared Spectroscopy.

                      Hans Henrik Thodberg   
               The Danish Meat Research Institute    

                            Abstract

MacKay's Bayesian framework for backpropagation is a practical and powerful 
means to improve the generalisation ability of neural networks. It is based
on a Gaussian approximation to the posterior weight distribution. The 
framework is extended, reviewed and demonstrated in a pedagogical way. The 
notation is simplified using the ordinary weight decay parameter, and a 
detailed and explicit procedure for adjusting several weight decay 
parameters is given. 

Bayesian backprop is applied in the prediction of fat content in minced meat
from near infrared spectra. It outperforms ``early stopping'' as well as 
quadratic regression. The evidence of a committee of differently trained 
networks is computed, and the corresponding improved generalisation is 
verified. The error bars on the predictions of the fat content are computed. 
There are three contributors: The random noise, the uncertainty in the 
weights, and the deviation among the committee members. The Bayesian 
framework is compared to Moody's GPE. Finally, MacKay and Neal's Automatic 
Relevance Determination, in which the weight decay parameters depend on the 
input number, is applied to the data with improved results.

----------------------------------------------------------------------------
The manuscript is a revised version of thodberg.ace-of-bayes.ps.Z which is 
also in neuroprose. The main changes are the following: Pruning has 
been taken out (it is treated in a separate paper), the treatment of 
committees is extended, and there is a new section demonstrating the powerful 
Automatic Relevance Determination. The data used in the paper are now 
available by ftp. The paper is submitted to IEEE Trans. on Neural Networks.
Comments are welcome!
----------------------------------------------------------------------------
Hans Henrik Thodberg                Email(NEW!!): thodberg at nn.dmri.dk
Danish Meat Research Institute      Phone: (+45) 42 36 12 00
Maglegaardsvej 2, Postboks 57       Fax:   (+45) 42 36 48 36
DK-4000 Roskilde, Denmark
----------------------------------------------------------------------------