TR announcement: On-Line Learning in Soft Committee Machines

[D Saad]

FTP-host: archive.cis.ohio-state.edu
FTP-file: pub/neuroprose/saad.online.ps.Z

The file saad.online.ps.Z is now available for
copying from the Neuroprose repository:


On-Line Learning in Soft Committee Machines (33 pages)  


David Saad -  Department of Physics, University of Edinburgh, 
              Edinburgh EH9 3JZ, UK.

Sara A. Solla - CONNECT, The Niels Bohr Institute, Blegdamsdvej 17, 
                Copenhagen 2100, Denmark.

The paper has been submitted for publication in Phys.Rev.E, a letter
describing the main results is to appear in Phys.Rev.Lett.


Abstract:
--------
The problem of on-line learning in two-layer neural networks is studied
within the framework of statistical mechanics. A fully connected committee
machine with $K$ hidden units is trained by gradient descent to perform a 
task defined by a teacher committee machine with M hidden units acting on
randomly drawn inputs. The approach, based on a direct averaging over the
activation of the hidden units, results in a set of first order differential
equations which describe the dynamical evolution of the overlaps among the
various hidden units and allow for a computation of the generalization error.
The equations of motion are obtained analytically for general K and M, and
provide a new and powerful tool used here to study a variety of realizable,
over-realizable, and unrealizable learning scenarios, and to analyze the 
role of the learning rate in controlling the evolution and convergence of
the learning process.