A Harder Learning Problem

[chrisley.pa@Xerox.COM]

This is in response to the recent comments by Alexis Wieland, Charles Bachmann,
and the tech report by Scott Fahlman which was announced on this mailing list.

I agree that a more careful selection of benchmarking tasks is required.
Specifically, there has been little effort spent on comparing networks on the
kinds of tasks that many are advocating as one of the fortes of the neural
network approach:  pattern recognition in natural signals (eg, speech).  The key
characteristic of patterns in natural signals is that they are statistical:  a
sample is often a member of more than one class.  Thus, one does not talk of
zero error, but minimal error.

The reason why explicitly statistical tasks should be used in benchmarking is
that the pattern recognition properties of models vary noticeably when moving
from the deterministic to the statistical case.  For an example of statistical
benchmarking of Backprop, Boltzmann machines, and Learning Vector Quantization,
see Kohonen, Barna and Chrisley, '88, in the proceedings of this year's ICNN.
Also see Huang and Lippmann, '87a and b (ICNN and NIPS).

For example, a typical two category task might have category A as a Gaussian
distribution cetered around the origin with a variance of 2, while category B
might be a Gaussian that is offset in the first dimension by some amount, and
with a variance of 1.  This requires non-linear decision boundaries for optimal
(Bayesian) performance, and the optimal performance may be calculated
analytically (good performance = low misclassification rate).  This is one of
the tasks discussed in our paper, above.

BTW, we found that LVQ was better than BP, especially in high dimensional and
difficult tasks, while the BM had almost optimal performance, although it
required inordinate amounts of computing time.

Ron Chrisley
Xerox PARC SSL
3333 Coyote Hill Road
Palo Alto, CA 94304