Paper Announcement (Neuroprose)

[Barak Pearlmutter]

The following paper has not been placed in the Neuroprose archives at
Ohio State.  The file is not pearlmutter.soft-share.soft-share.ps.Z.
Ftp instructions follow the abstract.


      -----------------------------------------------------

		      Simplifying Neural Network
		     Soft Weight-Sharing Measures
				  by
			 Soft Weight-Measure
			 Soft Weight Sharing

			  Barak Pearlmutter
		       Department of Psychology
		      P.O. Box 11A Yale Station
		      New Haven, CT  06520-7447

                            ABSTRACT:

It has been shown by Nowlan and Hinton (1991) that it is advantagious
to construct weight complexity measures for use in weight
regularization through the use of EM, instead of relying on some
a-priori complexity measure, or even worse, neglecting regularization
by assuming a uniform distribution.  Their work can be regarded as a
generalization of the "Optimal Brain Damage" of Le Cunn et al (1990),
in which the distribution of weights is estimated with a histogram, a
peculiar functional form for a distribution.  Nowlan and Hinton assume
a much simpler functional form for the distribution, avoiding
overfitting and therefore overregularization.  However, they disregard
the issue of regularization of the regularizer itself.  Just as
certain weights might be considered a-priori quite unlikely, certain
distributions of weights may be considered a-priori quite unlikely.
To solve this problem, we introduce a regularization term on the
parameters of the weight distribution being estimated.  This
regularization term is itself determined by a distribution over these
distributional parameters.  In this light, Nowlan and Hinton (1991)
make the uniform distributional parameter distribution assumption.
Here, we estimate the distribution of distributions by running an
ensemble of networks, with EM used to estimate the weight distribtion
of each network (following Nowlan and Hinton), but we then use EM to
estimate the distribution of distributions across networks.  Of
course, each estimated distribution is used to regularize the
parameters over which that distribution is defined, leading to
regularization of the individual network regularizers.

We do not consider how to estimate the a-priori distribution which
might be used to regularize the distribution being used to regularize
the distribution being used to regularize the weights being estimated
from the data, which will be the explored in a future paper.

      -----------------------------------------------------
                        FTP INSTRUCTIONS

Either use "getps pearlmutter.soft-share.soft-share.ps.Z", or do the following:

     unix> ftp archive.cis.ohio-state.edu (or 128.146.8.52)
     Name: anonymous
     Password: neuron
     ftp> cd pub/neuroprose
     ftp> binary
     ftp> get pearlmutter.soft-share.soft-share.ps.Z
     ftp> quit
     unix> uncompress pearlmutter.soft-share.soft-share.ps.Z
     unix> lpr -s pearlmutter.soft-share.soft-share.ps


Barak Pearlmutter
Department of Psychology
P.O. Box 11A Yale Station
New Haven, CT  06520-7447

Work Phone: 203 432-7011