The New Training Alg for Feedforward Networks

[Scott_Fahlman@SEF-PMAX.SLISP.CS.CMU.EDU]

    Dr. Subhash C. Kak writes, in his message:
    
      The computing power of this algorithm may be gauged
      from the example that the exclusive-Or problem that requires several
      thousand iterative steps using the backpropagation algorithm was
      solved in 8 steps.
    
    I cannot agree with the assertions made about the speed of
    backpropagation in the XOR problem. Just to be sure, I have just run a
    few tests, using plain backpropagation, in the batch mode, without any
    acceleration technique (not even momentum), and using an architecture
    with 2 input units, 2 hidden units and 1 output unit (more details
    available on request). The runs that didn't stop at local minima, all
    converged between 12 and 30 epochs. About 1/3 of the runs fell in
    local minima.
    
I was also going to comment on Kak's statement that backprop takes "several
thousand" iterative steps to converge.  It's not clear to me  about epochs
or pattern presentations, but in any case that number is too high.  One of
the very first papers on backprop (Rumelhart, Hinton, and Williams, in the
PDP books, refers to a study by Chauvin that solved XOR in an average of
245 epochs.  So Kak's figure is a bit high.

On the other hand, Luis Almeida's figure of 12-30 epochs for 2-2-1 XOR with
vanilla backprop are much, much better than other reported results for
backprop.  If he is actually getting those times, something very
interesting -- perhaps even earth-shaking -- is going on.  I can get times
like that with Quickprop, but I've never seen claims under 100 epochs for
2-2-1 backprop, with or without momentum.  More details on this experiment
would be of interest to many of us, I think.

-- Scott
===========================================================================
Scott E. Fahlman			Internet:  sef+ at cs.cmu.edu
Senior Research Scientist		Phone:     412 268-2575
School of Computer Science              Fax:       412 681-5739
Carnegie Mellon University		Latitude:  40:26:33 N
5000 Forbes Avenue			Longitude: 79:56:48 W
Pittsburgh, PA 15213
===========================================================================