Does backprop need the derivative ??

[James L. McClelland]

Re the discussion concerning replacing the derivative of
the activations of units with a constant:

Some work has been done using the activation rather than the
derivative of the activation by Nestor Schmajuk.  He is interested
in biologically plausible models and tends to keep hidden units in the
bottom half of the sigmoid.  In that case they can be approximated
by exponentials and so the derivative can be approximated by the
activation.

Approx ref: Schmajuk and DiCarlo, Psychological Review, 1992

 - Jay McClelland