preprint: Predicting the Future (Weigend, Huberman, Rumelhart)

[Andreas Weigend]

                     ______________________________

                         PREDICTING THE FUTURE -
                        A CONNECTIONIST APPROACH
                     ______________________________

                        Andreas S. Weigend   [1]
                        Bernardo A. Huberman [2]
                        David E. Rumelhart   [3]
                     ______________________________

        We investigate the effectiveness of connectionist networks
for predicting the behavior of non-linear dynamical systems. We use
feed-forward networks of the type used by Lapedes and Farber to ob-
tain forecasts in the context of noisy real world data from sunspots
and computational ecosystems. The networks generate accurate future
predictions from knowledge of the past and consistently outperform
traditional statistical non-linear approaches to these problems.

The problem of having too many weights compared to the number of data
points (overfitting) is addressed by adding a term to the cost function
that penalizes large weights. We show that this weight-elimination
procedure successfully shrinks the net down.  We compare different
classes of activation functions and explain why the convergence of
sigmoids is significantly better than the convergence of of radial
basis functions for higher dimensional input.  We suggest the use
of the concept of mutual information to interpret the weights.  We
introduce two measures of non-linearity and compare the sunspot and
ecosystem data to a series generated by a linear autoregressive model.
The solution for the sunspot data is found to be moderately non-linear,
the solution for the prediction of the ecosystem highly non-linear.

        Submitted to "International Journal of Neural Systems"

        If you would really like a copy of the preprint, send
        your physical address to:  hershey at psych.stanford.edu
        (preprint number: Stanford-PDP-90-01, PARC-SSL-90-20)

    [1] Physics  Department,  Stanford University,  Stanford, CA 94305
    [2] Dynamics of Computation Group, Xerox PARC, Palo Alto, CA 94304
    [3] Psychology Department, Stanford University, Stanford, CA 94305
                     ______________________________