CascadeC in parallel

[Scott_Fahlman@SEF-PMAX.SLISP.CS.CMU.EDU]

    It is less than obvious how to parallelize Cascade Correlation, especially
    if compared to stuff like backprop.

There are several good ways to partition Cascade-Correlation for parallel
execution.  Of course, the choice depends on the dimensions of your
problem, and the dimensions and communication structure of your machine.

One obvious choice is indeed to run each of the candidate units on its own
separate processor.  There is little communication involved: just
broadcasting the incoming values and error to be matched (if these are not
stored locally) and occasional polling to determine quiescence and choose a
winner.  For hard problems, Cascor spends most of its time in the candidate
training phase, so this can give good results.  If there are many output
units, they can also be trained separately, one per processor.  It is
possible to overlap candidate and output training to some degree.

The other obvious choice is to simulate an identical copy of the whole
architecture on different processors, each with 1/N of the training data.
Just before each weight-update phase, you sum up the computed derivative
values from all the processors and update all the copies at once.

Of course, there's much less total work to be done than with backprop, so
you might need a bigger problem to get any real advantage from parallelism.
But I find it hard to think of that as a disadvantage.

Scott Fahlman
School of Computer Science
Carnegie Mellon University