paper available: statistical distribution of NN results

[Lee Giles]

The following manuscript has been accepted in IEEE Transactions on
Neural Networks and is available at the WWW site listed below:

www.neci.nj.nec.com/homepages/giles/papers/IEEE.TNN.statistical.dist.of.trials.ps.Z

We apologize in advance for any multiple postings that may be received.

***********************************************************************


         On the Distribution of Performance from Multiple 
                     Neural Network Trials

Steve Lawrence(1), Andrew D. Back(2), Ah Chung Tsoi(3), C. Lee Giles(1,4)

(1) NEC Research Institute, 4 Independence Way, Princeton, NJ 08540, USA.
(2) Brain Information Processing Group, Frontier Research Program, RIKEN, 
    The Institute of Physical and Chemical Research, 2-1  Hirosawa, 
    Wako-shi, Saitama 351-01, Japan.
(3) Faculty of Informatics, University of Wollongong, Northfields
    Avenue, Wollongong NSW 2522, Australia. 
(4) Institute for Advanced Computer Studies, University of Maryland, 
    College Park, MD 20742, USA.


			ABSTRACT

The performance of neural network simulations is often reported in
terms of the mean and standard deviation of a number of simulations
performed with different starting conditions. However, in many cases,
the distribution of the individual results does not approximate a
Gaussian distribution, may not be symmetric, and may be multimodal. We
present the distribution of results for practical problems and show
that assuming Gaussian distributions can significantly affect the
interpretation of results, especially those of comparison studies.
For a controlled task which we consider, we find that the distribution
of performance is skewed towards better performance for smoother
target functions and skewed towards worse performance for more complex
target functions.  We propose new guidelines for reporting performance
which provide more information about the actual distribution.


Keywords: neural networks, gradient training, backpropagation, error
analysis, convergence, gaussian distribution, probability
distributions, statistical methods, box whiskers, kolmogorov-smirnov
test, mackey-glass, phoneme classification.

__                                
C. Lee Giles / Computer Science / NEC Research Institute / 
4 Independence Way / Princeton, NJ 08540, USA / 609-951-2642 / Fax 2482
www.neci.nj.nec.com/homepages/giles.html
==