Seminar: Barron on Scaling

[Terry Sejnowski]

Computer Science Seminar

"Approximation Properties of Artificial Neural Networks"
             
Andrew R. Barron
University of Illinois

Monday, January 7, 4 PM
7421 Applied Physics and Mathematics Building
University of California, San Diego

Bounds on the approximation error of a class of feed-forward artificial 
neural network models are presented.  A previous result obtained
by George Cybenko and by Kurt Hornik, Max Stinchcombe, and Hal White shows
that linear combinations of sigmoidal functions are dense in the space of
continuous functions on compact subsets in d dimensions.  In this talk
we examine how the approximation error is related to the number of nodes
in the network.  We impose the regularity condition that the gradient of
the function of d variables has an integrable Fourier transform.  
In particular, bounds are obtained for the integrated squared error of 
approximation, where the integration is taken on any given ball and with 
respect to any probability measure.  It is shown that there is a linear 
combination of n sigmoidal functions such that the integrated squared error 
is bounded by c/n, where the constant c is depends on the radius of the 
ball and the integral of the norm of the Fourier transform of the gradient 
of the function.  A sigmoidal function is the composition of a given bounded 
increasing function of one variable with a linear function of d variables.
Such sigmoidal functions comprise a standard artificial neuron model and 
the linear combination of such functions is a one-layer artificial neural
network.  The surprising aspect of this result is that an approximation rate 
is achieved which is independent of the dimension d, using a number of
parameters O(nd) which grows only linearly in d.  This is in contrast to
traditional series expansions which require exponentially many parameters
O(n^d) to achieve approximation rates of order O(1/n), under somewhat
different hypotheses on the class of functions.  We conclude that the
"curse of dimensionality" does not apply to the class of functions we
examine.