"Universal Approximators"

[Yann le Cun]

John Merrill says:

> In fact, the radial basis function theorem gives somewhat better
> bounds on the *number* of intermediate nodes necessary, and, as a
> consequence, indicates that if you're only interested in
> approximation, you want to use RBF's.

But you can build RBF's (or rather, multidimensional "bumps") with two layers
of sigmoid units. And you just need O(n) units for n bumps.
A third (linear) layer can sum up the bumps.

So i guess the bound on the number of intermediate nodes is the same
in both cases.

You can prove the universality of 2 hidden layer structures that way
(and the proof is a lot simpler than with 1 hidden layer).

  Yann Le Cun