Does backprop need the derivative ??

[john kolen]

Back prop does not need THE derivative.  I have some empirical results
which show that most of the internal mathematical operators of back prop
can be replaced by qualitatively similar operators.  I'm not talking about
reducing bit width, as most of the literature does.  I was interested in
what happens when you replace multiplication with maximum, the sigmoid with
a generic bump, etc.  What was suprising was that all the tweeks basically 
worked.  Back prop is "functionally" stable in the sense that the learning
functional ability remains regardless of minor shifts in internal
organization.  The reason that the reduced accuracy results are the way
that they are  can be traced to the loss of continuity rather than the 
loss of bits.

John Kolen