"Universal Approximators"

[George Cybenko]

Recently John Kolen asked why "universal approximator" results are
interesting.

Universal approximation results are important because they say
that the technique being used (sigmoidal or radial basis function
networks) has no a priori limitations if complexity (size) of the
network is not a constraint.  Such results are network analogues
of Church's Thesis.  Results about universal approximation properties
for a variety of network types can be found in :

G. Cybenko, "Approximation by Superpositions of a Sigmoidal Function",
Mathematics of Control, Signals and Systems (Springer),  August 1989,
Vol. 2, pp. 303-314.

Since such properties are shared by polynomials, Fourier series, 
splines, etc. etc.,  an important question to ask is "What makes 
network approaches better"?  Parallelism and the existence of
training algorithms does not count as an answer because polynomials
and Fourier series have similar parallelism and training algorithms.

I have a recent report that scratches the surface of this question
and proposes a notion of complexity of classification problems that
attempts to capture the way in which sigmoidal networks might be better on
a class of problems.

The report is titled "Designing Neural Networks" and is currently
issued as CSRD Report #934.  Send me mail or comments if you are
interested in such questions.

George Cybenko
Center for Supercomputing Research and Development
University of Illinois at Urbana-Champaign