No subject

[solla%nordita.dk@vma.CC.CMU.EDU]

Subject: Weight spaces

The concept of `weight space' has been shown to be a useful
tool to explore the ensemble of all possible network configurations,
or wirings, compatible with a fixed, given architecture. [1]

Such spaces are indeed complex, both because of their high
dimensionality and the roughness of the surface defined by the
error function.

It has been shown that different choices of the distance between
the targets and the actual outputs can lead to error surfaces that
are both generally smoother and steeper in the vicinity of the
minima, resulting in an accelerated form of the back-propagation
algorithm. [2]

Full explorations of such weight spaces, or configuration spaces,
defines a probability distribution over the space of functions.
Such distribution is a complete characterization of the functional
capabilities of the chosen architecture. [3]

The entropy of such prior distribution is a useful tool to
characterize the functional diversity of the chosen ensemble.
Monitoring the evolution of the probability distribution over
the space of functions and its associated entropy during learning
provides a quantitative measure of the emergence of generalization
ability. [3,4]

[1] J.S. Denker, D.B. Schwartz, B.S. Wittner, S.A.Solla, R.E. Howard,
L.D. Jackel, and J.J. Hopfield, `Automatic learning, rule extraction,
and generalization', Complex Systems, Vol 1. P. 877-922 (1987).

[2] S.A. Solla. E. Levin, and M. Fleisher, `Accelerated learning in
layered neural networks', Complex Systems, Vol 2, p. 625-639 (1988).

[3] S.A. Solla, `Learning and generalization in layered neural
networks: the contiguity problem', in `Neural networks from models
to applications, ed. by L. Personnaz and G. Dreyfus, IDSET, Paris,
p. 168-177 (1989).

[4] D.B. Schwartz, V.K. Samalam, S.A. Solla, and J.S. Denker,
`Exhaustive learning', Neural Computation, MIT, in press.