Connections per Second

[Ron Chrisley UNTIL 10/3/88]

Joachim Beer <beer at icsib.Berkeley.EDU> writes:

"The recent discussion about CPS numbers made me
wonder why we need such new metrics at all.
Why not just record the execution time in seconds?...
After all, the user is not interest in how many 
connections are updated per second but 
how long it will take to solve a (benchmark) problem."

The CUPS comparison should be between *implementations*
of standard connectionist networks, not between different
models themselves.  I don't know of any one comparing ART
to backprop using CUPS, but I do see people comparing various
network systems (like SAIC, HNC, Rochecter connectionist simulator
on machine XYZ, etc) that implement standard models.

This measurement, if the calculation procedure is standardized,
is useful.  You're right that the user wants to know how long it
will take, in seconds, for  the network to solve a problem, but
they usually want to know how long it will take for the network
to solve *their* problem, not a benchmark one.  This is where
CUPS ratings come in handy.  If we *banned* [:-)] CUPS
ratings, people would use them themselves anyway:  "Let's see,
if HNC told me that it takes 42.21 seconds for them to do 150
iterations of XOR learning with network configuration Z, then 
that means that they can do N CUPS, so that means that they
could do 100 iterations of my problem with network configuration
X in under 76 seconds."

Of course, this says nothing about how long (in iterations) it takes 
to *solve* a particular problem.  But reporting absolute times for 
benchmarks doesn't either (at least not until we have a good theory
how different problems relate to each other in terms of learning 
difficulty.  I should live so long... ?-)

Ron Chrisley