Why does the error rise in a SRN?

[Ray Watrous]

Simon -

An increase in error can occur with fixed step size algorithms
since although the step is in the negative gradient direction,
the size can be such that the new point is actually uphill.
The algorithm simply steps across a ravine, as it were, and ends
higher up on the other side.

This is a well-known property of such algorithms, but seems to be
encountered in practice more frequently with recurrent networks.
This is due to the fact that small changes in some regions of weight
space can have large effects on the error because of the nonlinear
feedback in the recurrent network.

There are many effective ways of controlling this behavior; you
may want to consult the line search algorithms in a standard text
on nonlinear optimization (such as D. Luenberger Linear and Nonlinear
Programming, 1984).


Raymond Watrous
Siemens Corporate Research
755 College Road East
Princeton, NJ 08540

(609) 734-6596