batch & on-line training

[Kamil A. Grajski]

The consensus opinion seems to be that on-line learning is preferred
for situations consisting of a classification problem with a large
(possibly redundant) dataset.  What appears to have been a common
experience is that batch-mode training generates impressive MCUP
statistics, but convergence is slower enough that the net gain is 0.
It is difficult to make a scientific judgement still, mostly because
the evidence appears to be largely anecdotal, e.g., "I really tried
hard to make one (batch, or on-line) work, and it beat the other."

It has been observed that several algorithms for accelerating
convergence are designed for (semi-)batch mode.  Were these to be
seriously evaluated, would the net gain 0 still occur?  On the other
hand, with more work could on-line methods widen their apparent
superiority?

I don't think that we're splitting hairs by addressing this issue.
One trend in the implementations side of NNs is to have the highest
MCUPS performance.  In several instances, this is achieved using
mappings/architectures which rest on batch-mode training.  I think
that one might design a neurocomputer differently depending on which
training mode is to be used, e.g., the communication vs computation
curves are different.  So, at the moment, in certain instances, we've
actually put the cart before the horse.  We have fast batch implemen-
tations.  Do we make batch-mode training better, or can we make on-line
so fast and so optimally design a machine that the issue is moot?
(I'm ignoring the (possibly substantial) conflicting requirements
between training & recognition modes, here.)

In any event, it seems that folks are having success doing either
in different situations.  However, there doesn't seem to be a
compelling argument for preferring one or the other IN PRINCIPLE.

Cheers,
Kamil