batch & on-line training

[xiru Zhang]

   Date: Thu, 17 Oct 91 09:59:21 -0700
   From: "Kamil A. Grajski" <kamil at apple.com>

   The consensus opinion seems to be that on-line learning is preferred
   for situations consisting of a classification problem with a large
   (possibly redundant) dataset.  What appears to have been a common
   experience is that batch-mode training generates impressive MCUP
   statistics, but convergence is slower enough that the net gain is 0.
   It is difficult to make a scientific judgement still, mostly because
   the evidence appears to be largely anecdotal, e.g., "I really tried
   hard to make one (batch, or on-line) work, and it beat the other."

I have used per-epoch training on an auto-association netowrk, to extract
"features" of protein local structures, using as few hidden units as
possible. I spent a lot of time to fine-tune the training process, such as
using different learning rate at different stage of training, different
momentum term, different range of random weights at the beginning, how
large each "batch" is, etc. At the end I got a pretty good convergence
rate. (Maybe I did not spend enough effert to fine-tune the per-sample
training.) My feeling is that training a large network with lots of
examples is still an art. You can almost always improve it if you spend
time on it. Per-epoch training may have somewhat different behavior than
per-sample training. So different training schedule is often needed. And it
takes time to figure out what is a good one. It also critically depends on
the particular problem you want to solve. 

Besides the issue of convergence rate, I wonder if people have compared
networks trained by per-epoch schedule and per-sample schedule, to see if
they have the same level of generalization. One thing I noticed in my work
is that per-sample training tends to make certain weights much larger than in
per-epoch training. But I am not sure if this is true in general. 


- Xiru Zhang