Does backprop need the derivative ??

[E. Fiesler]

Marwan Jabri wrote:

> Date: Sat, 6 Feb 1993 23:49:53 +1100
> From: Marwan Jabri <marwan at sedal.su.oz.au>
> Subject: Re: Does backprop need the derivative ??
> 
> As the intention of the inquirer is the analog implementation of
> backprop, I see two problems: 1- the question whether the derivative can
> be replaced by a constant, and more importantly 2- whether the precision
> of the analog implementation will be high enough for backprop to work.
> 
> Regarding (1), ...
> 
> Regarding (2), there has been several reports indicating that
> backpropagation simply does not work when the number of bits is reduced
> towards 6-8 bits! 

This is often reported for standard backpropagation. However, a simple
extension of backpropagation can make it work for any precision;
up to 1-2 bits. I'll append the reference(s) below.

				E. Fiesler
				Directeur de Recherche
				IDIAP
				Case postale 609
				CH-1920 Martigny
				Switzerland


@InProceedings{Fiesler-90,
        Author       = "E. Fiesler and A. Choudry and H. J. Caulfield",
        Title        = "A Weight Discretization Paradigm for Optical
                        Neural Networks",
        BookTitle    = "Proceedings of the International Congress on
                        Optical Science and Engineering",
        Volume       = "SPIE-1281",
        Pages        = "164--173",
        Publisher    = "The International Society for Optical
                        Engineering Proceedings",
        Address      = "Bellingham, Washington, U.S.A.",
        Year         = "1990",
        ISBN         = "0-8194-0328-8",
        Language     = "English" }

@Article{Fiesler-93,
        Author       = "E. Fiesler and A. Choudry and H. J. Caulfield",
        Title        = "A Universal Weight Discretization Method for
                        Multi-Layer Neural Networks",
        Journal      = "IEEE Transactions on Systems, Man, and Cybernetics
                        (IEEE-SMC)",
        Publisher    = "The Institute of Electrical and Electronics Engineers
                        (IEEE), Inc.",
        Address      = "New York, New York",
        Year         = "1993",
        ISSN         = "0018-9472",
        Language     = "English",
        Note         = "Accepted for publication." }