Adding noise to training data

[HOLMSTROM@csc.fi]

A note to John Hampshire's comment on this topic:

Adding noise to the training vectors has been suggested and also
used with some success by several authors.  In a forthcoming
article (Lasse Holmstrom and Petri Koistinen,
"Using Additive Noise in Back-Propagation Training",
IEEE Transactions on Neural Networks, January 1992) this
method is discussed from the point of view of mathematical statistics.
It is not claimed that better generalization is always achieved but
mathematical insight is given to the choice of the characteristics of
the additive noise density if using additive noise is attempted.
A critical question is the level (variance) of the additive noise.
One method to estimate a suitable noise level directly from data is to
use a cross-validation method known from statistics.  
In a standard benchmark experiment (Kohonen-
Barna-Chrisley, Neurocomputing 2) significant improvement 
in classification performance was achieved.  The training method
is also shown to be asymptotically consistent provided the noise
level is chosen appropriately.

Lasse Holmstrom