reprint available

[john moody]

Fellow Connectionists:

The following reprint has been placed on Jordan Pollack's neuroprose archive:

  LEARNING RATE SCHEDULES FOR FASTER STOCHASTIC GRADIENT SEARCH

         Christian Darken*, Joseph Chang+, and John Moody*
       Yale Departments of Computer Science* and Statistics+

                              ABSTRACT 

Stochastic gradient descent is a general algorithm that includes LMS, 
on-line backpropagation, and adaptive k-means clustering as special cases.  
The standard choices of the learning rate $\eta$ (both adaptive and fixed 
functions of time) often perform quite poorly.  In contrast, our recently 
proposed class of ``search then converge'' ($STC$) learning rate schedules 
(Darken and Moody, 1990b, 1991) display the theoretically optimal asymptotic 
convergence rate and a superior ability to escape from poor local minima.
However, the user is responsible for setting a key parameter.  We propose 
here a new methodology for creating the first automatically adapting learning 
rates that achieve the optimal rate of convergence.


To retrieve it via anonymous ftp, do the following:

% ftp cheops.cis.ohio-state.edu
Connected to cheops.cis.ohio-state.edu.
220 cheops.cis.ohio-state.edu FTP server ready.
Name: anonymous
331 Guest login ok, send ident as password.
Password: your email addressneuron
230 Guest login ok, access restrictions apply.
ftp> binary
200 Type set to I.
ftp> cd pub/neuroprose
250 CWD command successful.
ftp> get darken.learning_rates.ps.Z
200 PORT command successful.
150 Opening ASCII mode data connection for darken.learning_rates.ps.Z (238939 by
tes).
226 Transfer complete.
local: darken.learning_rates.ps.Z remote: darken.learning_rates.ps.Z
239730 bytes received in 11 seconds (22 Kbytes/s)
ftp> quit
221 Goodbye.
% uncompress darken.learning_rates.ps
% lpr -P printer_name darken.learning_rates.ps

Enjoy,

John Moody
-------