TR on Alternative Discrete-time Operators in Neural Networks

[Lee Giles]

The following TR is now available from the University of Maryland,
NEC Research Institute and the Laboratory of Artificial Brain Systems
archives.

************************************************************************************


                   Alternative Discrete-Time Operators and 
                    Their Application to Nonlinear Models


Andrew D. Back [1], Ah Chung Tsoi [2], Bill G. Horne [3], C. Lee Giles [4,5]


[1] Laboratory for Artificial Brain Systems, Frontier Research Program RIKEN,
The Institute of Physical and Chemical Research, 2-1 Hirosawa, Wako--shi,
Saitama 351-01, Japan

[2] Faculty of Informatics, University of Wollongong, Northfields Avenue, 
Wollongong, Australia

[3] AADM Consulting, 9 Pace Farm Rd., Califon, NJ  07830

[4} NEC Research Institute, 4 Independence Way, Princeton, NJ 08540

[5] Inst. for Advanced Computer Studies, U. of Maryland, College Park, MD. 20742


      U. of Maryland Technical Report CS-TR-3738 and UMIACS-TR-97-03                                


                             ABSTRACT


   The shift operator, defined as q x(t) = x(t+1), is the basis for 
   almost all discrete-time models. It has been shown however, that 
   linear models based on the shift operator suffer problems when used 
   to model lightly-damped-low-frequency (LDLF) systems, with poles near 
   $(1,0)$ on the unit circle in the complex plane. This problem occurs 
   under fast sampling conditions. As the sampling rate increases, 
   coefficient sensitivity and round-off noise become a problem as the 
   difference between successive sampled inputs becomes smaller and 
   smaller. The resulting coefficients of the model approach the 
   coefficients obtained in a binomial expansion, regardless of the 
   underlying continuous-time system. This implies that for a given 
   finite wordlength, severe inaccuracies may result. Wordlengths for the 
   coefficients may also need to be made longer to accommodate models which 
   have low frequency characteristics, corresponding to poles in the 
   neighbourhood of (1,0). These problems also arise in neural network 
   models which comprise of linear parts and nonlinear neural activation 
   functions. Various alternative discrete-time operators can be introduced 
   which offer numerical computational advantages over the conventional shift 
   operator. The alternative discrete-time operators have been proposed 
   independently of each other in the fields of digital filtering, 
   adaptive control and neural networks. These include the delta, rho, 
   gamma and bilinear operators. In this paper we first review these 
   operators and examine some of their properties. An analysis of the TDNN
   and FIR MLP network structures is given which shows their susceptibility
   to parameter sensitivity problems. Subsequently, it is shown that
   models may be formulated using alternative discrete-time operators
   which have low sensitivity properties. Consideration is 
   given to the problem of finding parameters for stable alternative 
   discrete-time operators. A learning algorithm which adapts the 
   alternative discrete-time operators parameters on-line is presented 
   for MLP neural network models based on alternative discrete-time 
   operators. It is shown that neural network models which use these 
   alternative discrete-time perform better than those using the shift 
   operator alone.
  

Keywords: Shift operator, alternative discrete-time 
  operator, gamma operator,  rho operator, low sensitivity, time delay 
  neural network, high speed sampling, finite wordlength, LDLF, MLP, 
  TDNN.

___________________________________________________________________________________

http://www.neci.nj.nec.com/homepages/giles.html
http://www.cs.umd.edu/TRs/TR-no-abs.html 
http://zoo.riken.go.jp/abs1/back/Welcome.html


--                                 
C. Lee Giles / Computer Sciences / NEC Research Institute / 
4 Independence Way / Princeton, NJ 08540, USA / 609-951-2642 / Fax 2482
www.neci.nj.nec.com/homepages/giles.html
==