CRG-TR-89-4 available

[Carol Plathan]

The following technical report by Yann le Cun, CRG-TR-89-4/June 1989, is now
available.  Please send me your (physical) mailing address to receive this
report:

		 GENERALIZATION AND NETWORK DESIGN STRATEGIES
				       
				 Yann le Cun*
                        Department of Computer Science
		             University of Toronto
                     TECHNICAL REPORT CRG-89-4 / June l989

				   ABSTRACT
				       
An interesting property of connectionist systems is their ability to learn
from examples. Although most recent work in the field concentrates on reducing
learning times, the most important feature of a learning machine is its
generalization performance.  It is usually accepted that good generalization
performance on real-world problems cannot be achieved unless some a priori
knowledge about the task is built into the system.  Back-propagation networks
provide a way of specifying such knowledge by imposing constraints both on the
architecture of the network and on its weights. In general, such constraints
can be considered as particular transformations of the parameter space.

Building a constrained network for image recognition appears to be a feasible
task. We describe a small handwritten digit recognition problem and show that,
even though the problem is linearly separable, single layer networks exhibit
poor generalization performance.  Multilayer constrained networks perform very
well on this task when organized in a hierarchical structure with shift
invariant feature detectors.  These results confirm the idea that minimizing
the number of free parameters in the network enhances generalization.

The paper also contains a short description of a second order version of
back-propagation that uses a diagonal approximation to the Hessian matrix.
------------- 

*Present address: Room 4G-332, AT&T Bell Laboratories, Crawfords
Corner Rd, Holmdel, NJ 07733
			
Note:    A shortened version of the Technical Report will appear in:
R. Pfeifer,  Z. Schreter, F. Fogelman, and L. Steels (editors),
"Connectionism in Perspective", Zurich, Switzerland, 1989. Elsevier.