batch-mode parallel implementations

[Peter Wohl]

Dear connectionists,
I have some comments on several of these, so I decided not to include
all the history of this discussion in my reply (you read it anyway).
So here I go:

1. Given per-sample training, one still faces the problem of how to deal
with really large networks (thousands of neurons and hundreds of thousands
connections) on a parallel machine that has far fewer processors.
What has been proposed: a) SIMD (don't cry for unused processors, as long
as you can communicate fast enough); b) MIMD with clustering neurons
somehow together, to increase granularity (SIMD also needs some), problem
here being dependence on VERY particular nets (usually layers with powers
of 2 neurons); c) re-writing the communication of the algorithm (see for
example my paper this coming Nov at ICTAI'91).

2. I agree that epoch-training is probably desirable. How large is a
"typical" epoch for a "large" net (thousands of neurons, 
fraction of million connections at least) ? Tens of vectors, hundreds ?
I would say, no more than few hundreds.

3. "Recall" (forward propagation with no weight update) is far easier to
parallelize, since there is no end-of-epoch bottleneck (barrier synch).
In some results (to be published next year), we achieved (on 32 BBN
Butterfly processors) almost 2 million connec-presen/sec with backprop.,
but over 5 million at recall. (2.5 million if you "adjust" forward-only
by dividing by two, to match the backprop figure more closely).

To summarize, I think the real problem of parallelizing ANNs applies when
at least one of net-size or training-epoch-size is large (and thus slow
when run sequentially). And don't forget: net architecture could change
during training (e.g. cascade corr), and still keep it parallel.

Thanks for your patience,
Peter Wohl
thsspxw at iitmax.iit.edu