Why batch learning is slower

[Thomas H. Hildebrandt]

   Date: Thu, 19 Mar 92 10:59:36 EST
   From: nin at cns.brown.edu (Nathan Intrator)


	      "Why Batch Learning is Slower Than Per-Sample Learning"
			      Thomas H. Hildebrandt

	 From the abstract:
	 "...For either algorithm, convergence is guaranteed as long as no
	 step exceeds the minimum ideal step size by more than a factor of 2.
	 By limiting the discussion to a fixed, safe step size, we can compare
	 the maximum step that can be taken by each algorithm in the worst case."
	 -------
   There is no "FIXED safe step size" for the stochastic version, namely there is
   no convergence proof for a fixed learning rate of the stochastic version.
   The paper cited by Chung-Ming Kuan and Kurt Hornik does not imply that either.
   It is therefore difficult to draw conclusions from this paper.

    - Nathan

I have not done it, but it appears straightforward to show convergence
for the linear network model with a fixed step size.  The actual step
taken is the product of the step size with the derivative of the
error.  If each step taken reduces the error in an unbiased way, then
the process will converge.  In this, I am not really treating a
stochastic version, since in the true sense, this would make the
training set an infinite sequence of random vectors.  For both
algorithms I assumed that there is a finite set of training vectors
which can be examined repeatedly.  I think this is a fairly
standard assumption.

It IS difficult to draw firm conclusions from this paper regarding the
behavior of the two versions of BP on multilayer nonlinear networks,
since the analysis is restricted to a single-layer linear network.  It
was intended to provide some intuition as to the unexpectedly poor
performance of parallel implementations of batch BP, and to suggest an
approach for the analysis of the multilayer nonlinear case.

				Thomas H. Hildebrandt
				Visiting Research Scientist
				CSEE Department
				Lehigh University