recurrent higher order neural networks

[Lee Giles]

Regarding higher order recurrent nets:

John Kolen mentions:
*****************************************
Higher order recurrent networks are recurrent networks with higher order
connections, (i[1]*i[2]*w[1,2] instead of i[1]*w[1]).  An example of a
high order recurent network is Pollack's sequential cascaded networks
which appear, I believe, in the latest issue of Machine Learning.  This
network can be described as two three-dimensional matrices, W and V, and
the following equations.

        O[t] = Sigmoid( (W . S[t]) . I[t])
        S[t+1]=Sigmoid( (V . S[t]) . I[t])

where I[t] is the input vector, O[t] is the output vector, and S[t] is the
state vector, each at time t.  ( . is inner product)

**********************************************

For other references on higher-order recurrent nets, see the following:
(This list is not meant to be inclusive, but to give some
flavor of the diversity of work in this area.)

Y.C. Lee, et.al,1986, Physica D.
H.H. Chen, et.al, 1986, AIP conference proceedings on Neural Networks
	for Computing
F. Pineda, 1988, AIP conference proceedings for NIPS
Psaltis, et.al, 1988, Neural Networks.
Giles, et al. 1990, NIPS2; and 1991 IJCNN proceedings
Mozer and Bachrach, Machine Learning 1991
Hush, et.al., 1991 Proceedings for Neural Networks for
	Signal Processing.
Watrous and Kuhn, 1992 Neural Computation

In particular the papers by Giles, et.al use a 2nd order RTRL
to learn grammars from grammatical strings. (Similar
work has been done by Watrous and Kuhn.) What may be
of interest is that using a heuristic extraction method,
one can extract the grammar that the recurrent network 
learns (or is learning). 

It's worth noting that higher-order nets usually include
sub-orders as special cases, i.e. 2nd includes 1st.
In addition, sigma-pi units are just a subset of 
higher-order models and in many
cases do not have the computational power of higher-order
models. 
                                  
                                  C. Lee Giles
                                  NEC Research Institute
                                  4 Independence Way
                                  Princeton, NJ 08540
                                  USA

Internet:   giles at research.nj.nec.com
    UUCP:   princeton!nec!giles
   PHONE:   (609) 951-2642
     FAX:   (609) 951-2482