"Universal Approximators"

[John Merrill]

JK/EH> Jim Keeler and Eric Hartman

JK/EH> The interest in sigmoidal and gaussian functions stems at 
JK/EH> least in part from their biological relevance; they are 
JK/EH> (much) more relevant than polynomials.

That's honestly a debatable point.  It may indeed be that sigmoids are
more "biologically natural" than polynomials, but their use in a
discrete-time system makes the difference hard to establish.  The fact
is that "real" neurons perform computations which are far more
complicated than any kind of "take a dot product and squash" system;
indeed, all the neurobiological evidence indicates that they do no
such thing.

JK/EH> Showing that neural networks serve as universal approximators
JK/EH> is much like having an existence proof for a differential equation:
JK/EH> you know the answer exists, but the theorem does not tell you
JK/EH> how to find it.  For that reason it is an important question
JK/EH> in principle, but not necessarily in practice.

By that standard, once any one universal approximator theorem had been
established, no other would possess even the faintest semblance of
interest.  Since Borel approximation (in the sense of either l_2 or
l_\infty) is easy to establish with sigmoidal networks alone, it seems
to me that the results concerning (eg.) radial basis function would be
hard to swallow.

In fact, the radial basis function theorem gives somewhat better
bounds on the *number* of intermediate nodes necessary, and, as a
consequence, indicates that if you're only interested in
approximation, you want to use RBF's.

--- John