NIPS workshop: The Dynamics Of On-Line Learning

[saadd]

                 THE DYNAMICS OF ON-LINE LEARNING

     NIPS workshop, Friday and Saturday, December 2-3, 1995
               7:30AM to 9:30AM -- 4:30PM to 6:30PM

Organizers: Sara A. Solla (CONNECT, The Niels Bohr Institute)
            and David Saad (Aston University)

On-line learning refers to a scenario in which the couplings of the
learning machine are updated after the presentation of each example.
The current hypothesis is used to predict an output for the
current input; the corresponding error signal is used for
weight modification, and the modified hypothesis is used for
output prediction at the subsequent time step. This type of algorithm
addresses general questions of learning dynamics,
and has attracted the attention of both the computational learning theory
and the statistical physics communities. Recent progress has provided
tools that allow for the investigation of learning scenarios that
incorporate many of the aspects of the learning of complex tasks: multilayer
architectures, noisy data, regularization through weight decay, the use
of momentum, tracking changing environments,
presentation order when cycling repeatedly through a finite training set...
An open and somewhat controversial question to be discussed in the workshop
is the role of the learning rate in controlling the evolution and convergence
of the learning process.


The purpose of the workshop is to review the theoretical tools
available for the analysis of on-line learning, to evaluate the current state
of research in the field, and to predict possible contributions to
the understanding and description of real world learning scenarios.
We also seek to identify future research directions using these methods,
their limitations and expected difficulties.


The topics to be addressed in this workshop can be grouped as follows:

1) The investigation of on-line learning from the point of view of stochastic
approximation theory. This approach is based on formulating a master
equation to describe the dynamical evolution of a probability density
which describes the ensemble of trained networks in the space of weights
of the student network. (Todd Leen, Bert Kappen, Jenny Orr)

2) The investigation of on-line learning from the point of view of
statistical mechanics. This approach is based on the derivation
of dynamical equations for the overlaps among the weight
vectors associated with the various hidden units in both student
and teacher networks. The dynamical evolution of the overlaps
provides a detailed characterization of the learning process and
determines the generalization error. (Sara Solla, David Saad, Peter Riegler,
David Barber, Ansgar West, Naama Barkai, Jason Freeman, Adam Prugel-Bennett)

3) The identification of optimal strategies for on-line learning.
Most of the work has concentrated on the learning of classification
tasks. A recent Bayesian formulation of the problem provides a
unified derivation of optimal on-line equations. (Shun-ichi Amari,
Nestor Caticha, Manfred Opper)

The names in parenthesis identify the speakers. A list of additional
participants includes Michael Kearns, Yann Le Cun, Yoshiyuki Kabashima, 
Mauro Copelli, Noboru Murata, Klaus Mueller.



    PROGRAM FOR THE NIPS WORKSHOP ON "THE DYNAMICS OF ON-LINE LEARNING" 


Friday
------

Morning:      7:30AM to 9:30AM
Introduction: Todd Leen
Speakers:     Jenny Orr 
              Bert Kappen 


Afternoon:    4:30PM to 6:30PM
Speakers:     Shun-ichi Amari   
              Nestor Caticha 
              Manfred Opper
            
Saturday
--------

Morning:      7:30AM to 9:30AM
Introduction: Sara Solla/David Saad
Speakers:     David Barber  
              Ansgar West
              Peter Riegler 

Afternoon:    4:30PM to 6:30PM
Speakers:     Adam Prugel-Bennett 
              Jason Freeman 
              Peter Riegler 
              Naama Barkai 





More details, abstracts and references can be found on:

http://neural-server.aston.ac.uk/nips95/workshop.html