Connections per Second

[Joachim Beer]

The recent discussion about CPS numbers made me
wonder why we need such new metrics at all.
Why not just record the execution time in seconds?
Here is what I don't like about CPS numbers:
 
  * They don't facilitate the performance comparison of
    connectionist models and non-connectionist models
    (e.g. statistical pattern classifiers).
    Are we afraid of such comparisons?

  * If CPS numbers become the accepted yard stick
    with which to measure execution models then one is
    likely to penalize new approaches which are slower
    per connection but, overall, require fewer connections.

  * CPS numbers are very hard to interpret, as we have seen
    in the recent discussion about CPS. 

Every performance metric is open to abuse, but measuring
performance in absolute execution time seems to me the least
ambiguous metric. After all, the user is not interest in
how many connections are updated per second but how long
it will take to solve a (benchmark) problem.

A similar development has taken place in the field of
Logic Programming, there everything is measured in KLIPS
(Kilo logical inference per second). This has led to
such a confusion that it is now virtually impossible to
fairly compare competing execution models (not to mention
comparison with other programming paradigms).
Let's hope this will not happen to connectionism.

-Joachim Beer