Connections Per Second

[George Cybenko]

The recent discussion about how to measure machine performance
in terms of connections per second, etc. is reminiscent of the
decade old debate about how to report machine performance in general.
Here are three important turning points in the history of measuring
and reporting machine performance for scientific computing.  

Pre 1980  - (Vendor defined and supplied floating point execution rates)
	In order to make megaflop numbers as large as possible,
	people used register arithmetic operations so those 
	rates completely ignored addressing, incrementing, cache effects, etc.
	Consequently, the performance of a machine on a typical 
	scientific program was almost uncorrelated with those MFLOP rates.

Mid-1980's - (Kernels and loops)
	Livermore loops, Whetstones, Linpack kernels were introduced
	because of the problems noted above.  However, these loops and
	kernels are somewhat homogeneous, lack I/O , stressed the
	memory bandwidth in predictable ways and could be
	easily detected and optimized by a compiler.  Consequently,
	there were accusations that compilers and machines were
	being constructed to deliver high performance on those
	benchmark loops and kernels.  

Late-1980's and beyond ?  (Applications based benchmarking -Perfect Club, SPEC)
	Replace kernels and loops with scientific applications 
	codes and data sets that are representative of a high 
	end machine workload.  Replace MFLOPS with absolute execution times.  
	The Perfect (Performance Evaluation through Cost-Effective
	Transformation) Club was a cooperative effort formed in 1987
	that collected 13 scientific codes and data sets to 
	form a benchmark suite.  When porting codes to different 
	machines, changes in the codes were allowed.  The initial 
	effort included researchers from Cray, IBM, Caltech, 
	Princeton, HARC, University of Illinois, and the Institute 
	for Supercomputing Research (Tokyo) - the codes
	include circuit simulation, fluids, physics and signal 
	processing applications.  The Systems Performance Evaluation
	Cooperative (SPEC) is an industry motivated effort started in
	spring 1989 to develop applications based benchmarking for a
	wider range of machines, including workstations.  DEC, IBM,
	and MIPS belong to SPEC for example.

In light of this history, it seems that using MFLOPS or CPS as measures
of machine performance on neural computing applications ignores a decade
of progress and plays right into vendor hands.  Instead, let me suggest
that someone submit a state-of-the-art code solving a representative
problem in one of the major connectionist models, together with
a data set and solution to the Perfect Club or SPEC suite.  This way,
people interested in connectionist computing can simultaneously
contribute to a broader effort in benchmarking and avoid recapitulating
history. 

Information about the Perfect Club effort can be obtained by writing
to
	Lynn Rubarts
	Center for Supercomputing Research and Development
	University of Illinois
	Urbana, IL 61801  USA
	(217) 333-6223

or sending an email request for the Perfect Club reports to 

	rubarts at uicsrd.csrd.uiuc.edu.

Anyone with a code that might be suitable for the Perfect benchmark
can contact me.

George Cybenko
Center for Supercomputing Research and Development
University of Illinois at Urbana
Urbana, IL 61801
(217) 244-4145
gc at uicsrd.csrd.uiuc.edu