Test & Derivatives in Backprop

[Marwan Jabri]

> From: john kolen <kolen-j at cis.ohio-state.edu>
> 
> [I hope that this makes it to connectionists, the last couple of postings
>  haven't made it back.  So I have summarized these replies in one message
>  for general consumption.]
> 
> Regarding the latest talk about derivatives in backprop, I had looked into
> replacing the different mathematical operations with other, more
> implementation-amenable operations.  This included replacing the
> derivative of the squashing function with d(x)=min(x,1-x).  The results of
> these tests show that backprop is pretty stable as long as the qualitative
> shape of the operations are maintained.  If you replace the derivative with
> a constant or linear (wrt activation) function it doesn't work at all for
> the learning tasks I considered.  As long as the derivative replacement is
> minimal in the extreme activations and maximal at 0.5 (wrt the traditional
> sigmoid), the operation will not suffer dramatically.  
> 
> After reading Fahlman's observation about loosing bits to noise I had the
> following response.  Bits come from binary decisions.  Analog systems
> don't do that in normal processing, normally some continuous value affects
> another continuous value.  No where do they perform A/D conversion and then
> operate on the bits.  If there is no measurement device, then talking about
> bits doesn't make sense.
> 
> John Kolen
> 

Are we talking about analog implementations? I hope so because I am.
If not, then forget this message.

The derivative issue boils down to whether you can implement cheaply,
whatever is the approximation. The implication on the training speed
depends on  how good your gradient approximations are.

The bit-width issue boils down to how you will implement your storage
(weights).  Whether you use analog EEPROM, RAM converted with DACs 
or whatever, you have to deal with bit effects. Except
if you have a new analog high precision storage device that can be implemented
cheaply, in which case I will be eager to learn about.

If you have the analog dream device, then your next problem in analog
implementation is the signal/noise ratio. Except if your analog circuits
are noisyless.

	Marwan

-------------------------------------------------------------------
Marwan Jabri			       Email: marwan at sedal.su.oz.au
Senior Lecturer				      Tel: (+61-2) 692-2240
SEDAL, Electrical Engineering,		      Fax:         660-1228
Sydney University, NSW 2006, Australia     Mobile: (+61-18) 259-086