Putting a NN on 16k processors

[Technoshaman Tom]

On Mon, 1 Mar 1993 prechelt at ira.uka.de wrote:

> > Apart of that, seems to be quite complicated to put a nn on 16K processors ...
> > how do you do that ?

> 1. Use the Nettalk-architecture which consumes up to two processors per
>    connection.
> 2. Additionally, replicate the net and train on several examples at once.
> 3. Don't use TOO small nets and training sets.

The method you use for parallel NN depend on the size of your nets,
the size of your training set, and restrictions on training mechanisms.

One fairly obvious method, which is easy to code in a high level
language, is using batch training described by linear algebra equations.
(matrix multiplication, transposition, addition, etc.)

However, in many situations the size of your net will be small enough
to replicate the net on each processor, since usually you will have more
examples to train relative to the size of the net.

-Thomas Edwards