papers available on stereo and learning

[Jonathan Baxter]

Ning Qian wrote:

>    Date: Wed, 17 Mar 1999 11:07:38 +0000
>    From: Geoffrey Hinton <hinton at gatsby.ucl.ac.uk>
>
>    > I show that in the limit of continuous time, the momentum parameter
>    > is analogous to the mass of Newtonian particles that move through a
>    > viscous medium in a conservative force field.
>
>    At the risk of sounding like Jim Bower, I would like to opint out that
>    mechanical model was the original motivation for the momentum method.
>
>    Geoff Hinton
>
> The original motivation was fully discussed and acknowledged (by
> citing Rumelhart, Hinton and Williams's chapter in the PDP book) in
> the Introduction.  The paper went far beyond that by first showing
>         m
> p =  ------
>      m + mu
>
> (where p is the momentum parameter, m is the mass, mu is the friction
> coefficient), and then providing a stability and convergence analysis
> for both the continuous and discrete cases.
>
> Best regards,
> Ning

The momentum term in steepest descent methods was introduced and analysed by
B Poljak in 1964:

B.T Poljak, 1964.  "Some Methods of Speeding up the Convergance of Iteration
Methods". Z.  VyCisl. Mat. i Mat. Fiz, Vol. 4, pp 1-17.

He called it the "Heavy Ball" method.

I don't have the original paper but a good secondary source is "Neurodynamic
Programming"  by Bertsekas and Tsitsiklis,, Athena Scientific, 1996, pp
104--105. The convergence results are in there.

Cheers,

Jonathan Baxter