Connections per Second

[Joachim Beer]

I still believe execution time is a more versatile metric
than CUPS. Of course, it all depends on what you want to
benchmark. You might want to compare different algorithms 
or benchmark connectionist simulators/machines.
It is clear that CUPS don't help you to compare different
connectionist models. However, most simulators try to
support a wide range of different connectionist models.
Knowing their CUPS performance for standard backprop does not
help me to asses their potential performance for different
connectionist models. I would therefore like to see a
connectionist benchmark set that incorporates as many different
models as possible. I realize that this is probably asking
to much, but, for example, even backprop can be implemented
in many alternative ways: on-line, off-line, conjugated gradient
methods, etc. For example, in order to evaluate a backprop
simulator/machine I would like to measure the simulator
on the above alternative implementation models, because
they have slightly different computational requirements.
Just standard (on-line?) backprop CUPS performance is not very
informative for anything that aims to be more than just
a standard backprop machine or simulator.
To make it short, CUPS apply only to one precisely
defined benchmark program (usually standard backprop). In my
opinion this is to narrow a definition of a benchmark metric.

Can one define a (artifical?) benchmark set that reflects
as best as possible the operational and computational
requirements of connectionist networks in general?
Something the connectionist community could agree on.

-Joachim