The transposed weight matrix hassle

[xiru Zhang]

   Date: 03 Nov 91 19:56:50+0100
   From: Henrik Klagges <uh311ae at sunmanager.lrz-muenchen.de>

   There are some nasty things showing up if you want to fine-tune
   a parallel architecture to algorithms such as backprop. E.g., you
   either get the communications fast fro the forwar phase or the
   backward phase - but if you want to use the same communication
   flow for both, you have to transpose the weight matrices. This
   is on the order of O(forget it). Has anybody cooked up an idea ?

   Cheers, Henrik

   MPCI at LLNL
   IBM Research
   U. of Munich



This is definitely not true for our implementations on CM-2. We have
several ways to run backprop:

1. one big network on the whole CM-2;
2. multiple copies of the same network on CM-2, each copy runs on a group
   of processors (i.e., "batch mode");
3. multiple copies of the same network on CM-2, each copy runs on one
   processor.

In none of the above cases we had the problem you mentioned in your
message.

- Xiru Zhang
  Thinking Machines Corp.