"Universal Approximators"

[Eric Hartman]

We agree with George Cybenko's response to John Kolnen
and have some additional comments:

The interest in sigmoidal and gaussian functions stems at 
least in part from their biological relevance; they are 
(much) more relevant than polynomials.

Showing that neural networks serve as universal approximators
is much like having an existence proof for a differential equation:
you know the answer exists, but the theorem does not tell you
how to find it.  For that reason it is an important question
in principle, but not necessarily in practice.

Note that the answer could just as easily have been negative:
there are several classes of functions that do not serve as
universal approximators. (Take, e.g., functions from C_k trying to
approximate functions from C_k+1.) If the answer was negative 
for neural networks, then we would have to think hard about 
why neural networks work so well.

Jim Keeler and Eric Hartman