Backprop-simulator for Connection Machine available

[Sebastian.Thrun@B.GP.CS.CMU.EDU]

Folks,

according to the recent discussion concerning parallel implementations
of neural networks, I want to inform you that there is a very fast code
of backprop for the Connection Machine CM-2 public available. This
posting is about a year old, and meanwhile various labs have found and
removed (almost) all the bugs in the code, such that it is very reliable
now. It should be noted that this implementation is the most simple way
of parallelizing backprop (it's data-parallel), and that it works
efficiently with large training sets only (in the order of 500 to
infinity). But then it works great!

                                                  --- Sebastian



------------------------------------------------------------------
                (original message follows)
------------------------------------------------------------------

The following might be interesting for everybody who works with the PDP
backpropagation simulator and has access to a Connection Machine:


        ********************************************************
        **                                                    **
        **   PDP-Backpropagation on the Connection Machine    **
        **                                                    **
        ********************************************************


For testing our new Connection Machine CM/2 I extended the PDP
backpropagation simulator by Rumelhart, McClelland et al. with a parallel
training procedure for the Connection Machine (Interface C/Paris, Version 5).

Following some ideas by R.M. Faber and A. Singer I simply made use of the
inherent parallelism of the training set: Each processor on the connection
machine (there are at most 65536) evaluates the forward and backward
propagation phase for one training pattern only. Thus the whole training set
is evaluated in parallel and the training time does not depend on the size of
this set any longer. Especially at large training sets this reduces the
training time greatly. For example:

I trained a network with 28 nodes, 133 links and 23 biases to approximate the
differential equations for the pole balancing task adopted from Anderson's
dissertation.  With a training set of 16384 patterns, using the conventional
"strain" command, one learning epoch took about 110.6 seconds on a SUN 4/110 -
the connection machine with this SUN on the frontend managed the same in
0.076 seconds.

--> This reduces one week exhaustive training to approximately seven minutes!

(By parallelizing the networks themselves similar acceleration can be
achieved also with smaller training sets.)


--------------

The source is written in C (Interface to Connection Machine: PARIS) and can
easily be embedded into the PDP software package. All origin functions of the
simulator are not touched - it is also still possible to use the extended
version without a Connection Machine. If you want to have the source, please
mail me!


                                             Sebastian Thrun, thrun at cs.cmu.edu


You can also obtain the source via ftp:

            ftp 129.26.1.90
Name:       anonymous
Password:   <transmit your full e-mail address, e.g. thrun at damn&hell.edu>
ftp>        cd pub
ftp>        cd gmd
ftp>        get pdp-cm.c
ftp>        bye