Paper Announcement (Neuroprose)

[Steven J. Nowlan]

    ** Paper available via Neuroprose ***************************************
    ** Please do not forward to other mailing lists or boards.  Thank you. **

The following paper has been placed in the Neuroprose
archives at Ohio State. The file is nowlan.soft-share.ps.Z
Ftp instructions follow the abstract.


      -----------------------------------------------------

                    Simplifying Neural Networks
                      by Soft Weight-Sharing

                       Steven J. Nowlan
                Computational Neuroscience Laboratory
                      The Salk Institute
                        P.O. Box 5800
                     San Diego, CA 92186-5800

                      Geoffrey E. Hinton
                 Department of Computer Science
                     University of Toronto
                    Toronto, Canada M5S 1A4


                            ABSTRACT:

One way of simplifying neural networks so they generalize better is to add an
extra term to the error function that will penalize complexity.  Simple
versions of this approach include penalizing the sum of the squares of the
weights or penalizing the number of non-zero weights.  We propose a more
complicated penalty term in which the distribution of weight values is
modelled as a mixture of multiple gaussians.  A set of weights is simple if
the weights have high probability densities under the mixture model.  This can
be achieved by clustering the weights into subsets with the weights in each
cluster having very similar values.  Since we do not know the appropriate
means or variances of the clusters in advance, we allow the parameters of the
mixture model to adapt at the same time as the network learns.  Simulations on
two different problems demonstrate that this complexity term is more effective
than previous complexity terms.

      -----------------------------------------------------
                        FTP INSTRUCTIONS

Either use "Getps nowlan.soft-share.ps.Z", or do the following:

     unix> ftp archive.cis.ohio-state.edu (or 128.146.8.52)
     Name: anonymous
     Password: neuron
     ftp> cd pub/neuroprose
     ftp> binary
     ftp> get nowlan.soft-share.ps.Z
     ftp> quit
     unix> uncompress nowlan.soft-share.ps.Z
     unix> lpr -s nowlan.soft-share.ps (or however you print postscript)


Steven J. Nowlan
Computational Neuroscience Laboratory
The Salk Institute
P.O. Box 85800
San Diego, CA
92186-5800

Work Phone: 619-453-4100 X463
e-mail:  nowlan at helmholtz.sdsc.edu