Paper available

[Luis B. Almeida]

The following paper is available for download:


         Parameter Adaptation in Stochastic Optimization

Luis B. Almeida, Thibault Langlois, Jose D. Amaral and Alexander Plakhov


                            ABSTRACT

Optimization is an important operation in many domains of science and
technology. Local optimization techniques typically employ some form of
iterative procedure, based on derivatives of the function to be
optimized (objective function). These techniques normally involve
parameters that must be set by the user, often by trial and error.
Those parameters can have a strong influence on the convergence
speed of the optimization. In several cases, a significant speed
advantage could be gained if one could vary these parameters during the
optimization, to reflect the local characteristics of the function being
optimized. Some parameter adaptation methods have been proposed for this
purpose, for deterministic optimization situations. For stochastic (also
called on-line) optimization situations, there appears to be no simple
and effective parameter adaptation method.

This paper proposes a new method for parameter adaptation in stochastic
optimization. The method is applicable to a wide range of objective
functions, as well as to a large set of local optimization techniques.
We present the derivation of the method, details of its application
to gradient descent and to some of its variants, and examples of its use
in the gradient optimization of several functions, as well as in the
training of a multilayer perceptron by on-line backpropagation.


The paper has 24 pages, and is available in compressed postscript form
(162 kB) at

    ftp://146.193.2.131/pub/lba/papers/adsteps.ps.gz

and in uncompressed postscript form (956 kB) at

    ftp://146.193.2.131/pub/lba/papers/adsteps.ps


Comments are welcome.

Luis B. Almeida
                                  Phone:  +351-1-3100246,+351-1-3544607
INESC                             Fax:    +351-1-3145843
R. Alves Redol, 9                 E-mail: lba at inesc.pt       
1000 Lisboa, Portugal             http://ilusion.inesc.pt/~lba/lba.html 
------------------------------------------------------------------------
   *** Indonesia is killing innocent people in East Timor ***

          see   http://amadeus.inesc.pt/~jota/Timor/