TR available: "Priors for infinite networks"

[Radford Neal]

FTP-host: ftp.cs.toronto.edu
FTP-filename: /pub/radford/pin.ps.Z


 The following technical report is now available via ftp, as described below.


                       PRIORS FOR INFINITE NETWORKS

                              Radford M. Neal
                     Department of Computer Science 
                          University of Toronto 

                               1 March 1994

    Bayesian inference begins with a prior distribution for model parameters 
    that is meant to capture prior beliefs about the relationship being 
    modeled.  For multilayer perceptron networks, where the parameters are 
    the connection weights, the prior lacks any direct meaning --- what 
    matters is the prior over functions computed by the network that is 
    implied by this prior over weights.  In this paper, I show that priors 
    over weights can be defined in such a way that the corresponding priors 
    over functions reach reasonable limits as the number of hidden units in 
    the network goes to infinity.  When using such priors, there is thus no 
    need to limit the size of the network in order to avoid ``overfitting''.  
    The infinite network limit also provides insight into the properties of 
    different priors.  A Gaussian prior for hidden-to-output weights results 
    in a Gaussian process prior for functions, which can be smooth, Brownian, 
    or fractional Brownian, depending on the hidden unit activation function 
    and the prior for input-to-hidden weights.  Quite different effects can 
    be obtained using priors based on non-Gaussian stable distributions.  In 
    networks with more than one hidden layer, a combination of Gaussian and
    non-Gaussian priors appears most interesting.


The paper may be obtained in PostScript form as follows:

    unix> ftp ftp.cs.toronto.edu  (or 128.100.3.6, or 128.100.1.105)
          (log in as user 'anonymous', your e-mail address as password)

    ftp> cd pub/radford
    ftp> binary
    ftp> get pin.ps.Z
    ftp> quit

    unix> uncompress pin.ps.Z
    unix> lpr pin.ps (or however you print PostScript)

The report is 22 pages in length.  Due to figures, the uncompressed
PostScript is about 2 megabytes in size.  The files pin[123].ps.Z in
the same directory contain the same paper in smaller chunks; these may
prove useful if your printer cannot digest the paper all at once.
Some of the figures take a while to print; the largest such is the
sole content of pin2.ps

    Radford Neal

    radford at cs.toronto.edu