Batch Backprop versus Incremental (fwd)

[CRReeves]

On 27th January, Brown Cribbs wrote:

> 
> Dear Connectionists,
> 	I attended the 5th International Conference on Genetic Algorithms
> this summer, and in one of the sessions on combinations of genetic 
> algorithms (GAs) and Neural Nets(ANNs) a gentleman from the U.K.
> suggested that Batch mode learning could possibly be unstable in
> the long term for backpropagation.  I did not know the gentleman
> and when I asked for a reference he could not provide one.
> 
> Does anyone have any kind of proof stating that one method is better 
> than another?  Or that possibly batch backprop is unstable in <<Some>>
> sense?
> 

I thought a contribution from the UK was necessary, particularly in view of 
Scott Fahlman's later, somewhat provocative, remarks!

I've seen a preprint of a paper by Steve Ellacott (University of Brighton) 
in which he considers just this problem. This may have been the paper 
referred to in the original question. He considers the question of 
{\em numerical\/} stability of batch and case-by-case training, and shows 
that in this sense there are conditions under which the delta rule is 
unstable for batch updates. He then proceeds to look at the generalized 
delta rule, with similar results at least in the neighbourhood of a 
local minimum. Of course, the choice of learning rate will affect the 
conclusions in a particular case.

I understand these strange ideas have been expanded into a chapter (called 
"The Numerical Analysis Approach") of a book just published (end '93):

	Mathematical Approaches to Neural Networks
		(J.G.Taylor - Ed.)
		Elsevier Science Publishers
		ISBN 0-444-81692-5

-- 
 ___________________________________________
| Colin Reeves				    |
| Division of Statistics and OR		    |
| School of Mathematical and Information    |
| Sciences             			    |
| Coventry University			    |
| Priory St				    |
| Coventry CV1 5FB			    |
| tel :+44 (0)203 838979		    |
| fax :+44 (0)203 838585		    |
| email: CRReeves at uk.ac.cov.cck		    |
| (alternative email: srx014 at cck.cov.ac.uk) |
|___________________________________________|