Scaled Conjugate Gradient

[Patrick Grother]

The factor of 30 speed up of conjugate gradient over backprop that I quoted in
my piece of November 6 is due to an excellent scaled conjugate gradient
algorithm from Martin Moeller. Some conjgrad algorithms have been
criticised on the basis that a costly line search is performed per epoch.
Moeller's method sidesteps this expense by means of an automatic
Levenberg-Marquardt step size scaling at each iteration. This effectively
regulates the indefiniteness of the Hessian matrix.

Patrick Grother
Advanced Systems Division
NIST

November 7