Connections Per Second

[Scott.Fahlman@B.GP.CS.CMU.EDU]

     A 128 input units, 64 hidden units, 128 output units 3-layer backprop
     network (learning the encoder problem, say).
    
     Let's say I can do 50 epochs of 128 exemplars in 41.54 seconds of this
     network.  Is it valid to say:
    
     Number of connections per exemplar = 
            (128 * 64) + (128 * 64) [forward prop] +
            (128 * 64) [back prop] = 24576
    
     Number of connections per epoch = 24576 * 128 = 3145728
    
     Number of connections per 50 epochs = 3145728 * 50 = 157286400
    
     Divide by run time for 50 epochs = 157286400/41.54 = 3786384.2
    
     Now, is it accurate to say my backprop program runs at 3.7 million
     interconnections per second?
    
     Is my claim still acceptable if I am actually performing the 
     neural network calculations by systollic array matrix multiplication?
    
I think that most of the CPS numbers that people quote are derived by
multiplying the total number of connections in the net times the number
pattern presentations divided by the total time required.  You would not
count a connection twice just because the backprop algorithm happens to
use some or all of the weights twice.

For your example, I get (64 * 129) + (128 * 65) = 16,576 connections (don't
forget the bias connections), 6400 pattern presentations, and 41.54
seconds, for a grand total of 2.55 million CPS.

At least, that's how I would compute these numbers.  Your milage may
differ.  I suspect that some of the CPS numbers quoted for commercial
neural net simulators were computed using some other formula.  When I talk
to people selling these simulators, I always try to pin them down on how
their CPS numbers were computed, but the marketing types seldom know the
gory details.

I think that it is reasonable to report "equivalent" CPS numbers, even if
your program actually does the computation in some other way -- computing
many epochs in parallel, for example.

-- Scott