Summary of responses: usefulness of chaotic dynamics

[Albert G. Boulanger]

While the following reference is not directly tied to NNets, it is
tied to the the broader program of making computational use of chaos:

"Chaotic Optimization and the Construction of Fractals: Solution of an
Inverse Problem", Giorgio Mantica & Alan Sloan, Complex Systems
3(1989) 37-62.

The agenda here is to make use of the ergodic properties of a
dynamical system driven to be chaotic. (Briefly, an ergodic system is
one that will pass through every possible dynamical state compatible
with its energy.) This corresponds to a high temperature setting in
simulated annealing. (A key to the way annealing works is that it,
too, is ergodic.)  Then they drive the parameter down, and it becomes
controlled by repulsive (objective function: worse) and attractive
(objective function: better) Coulomb charges. These charges are placed
at the successive sites visited. (They also window the number of
charges. The repulsive charge makes this system have tabu-search like
properties.) Thus, they endow the system with memory at the lower
setting of the dynamical parameter. They claim the memory allows more
efficient convergence of the algorithm than annealing.

Here are some short references on chaos and ergodic theory:

"Modern Ergodic Theory"
Joel Lebowitz, & Oliver Penrose
Physics Today, Feb, 1973, 23-29

"Chaos, Entropy, and, the Arrow of Time"
Peter Coveney
New Scientist, 29 Sept, 1990, 49-52  (nontechnical)

"The Second Law of Thermodynamics: Entropy, Irreversibility, and
Dynamics", Peter Coveney, Nature, Vol 333, 2 June 1988, 409-415.
(technical)

"Strange Attractors, Chaotic Behavior, and Information Flow"
Robert Shaw, Z. Naturforsch, 86a(1981), 80-112

"Ergodic Theory, Randomness, and 'Chaos'"
D.S. Ornstein
Science, Vol 243, 13 Jan 1989, 182-

The papers by Coveney and the one by Shaw get into another possible
use of chaos involving many-body "nonlinear systems, far from
equilibrium". Because of the sensitivity of chaotic systems to
external couplings, one can get such systems to act as information (or
noise) amplifiers.  Shaw puts it as getting information to flow from
the microscale to the macroscale. Such self-organizing many-body
systems can be used in a generate-and-test architecture as the pattern
generators. In neural networks, competitive dynamics can give rise to
such behavior. Eric Mjolsness worked on a fingerprint "hallucinator"
that worked like this (as I remember). Optical NNets using 4-wave
mixing with photorefractive crystals have this kind of dynamics too.

Seeking structure in chaos,
Albert Boulanger
aboulanger at bbn.com