Algorithm evaluation over the Internet (papers available)

[Lucas, Simon M]

 Dear All,

 We've been developing a system for
 automatically evaluating algorithms over
 the Internet - in particular this is geared towards evaluation
 of image/signal processing and pattern recognition
 algorithms.  There are many aims of the
 system but the main one is to make life easier
 for the algorithm developer by automating the
 evaluation process and by ensuring that the
 results produced are objective and authentic.

 There's a paper due to appear in ICPR-2000,
 and you can also get a preprint of a more complete
 paper submitted to the Interntation Journal
 of Document Analysis and Recognition (see below).

 The website is in its early testing phase
 and can be found at http://algoval.essex.ac.uk
 Most of the site is being mirrored for the moment
 at http://ace.essex.ac.uk so try that if the first one
 fails.

 The results section shows how potenitally informative
 and effortless evaluation can be when done
 this way (see trainable ROI results, or dictionary
 word recognition for example).

 Also, if there's sufficient interest,
 I wondered if anyone would be interested in
 helping organise a NIPS workshop on this kid of
 stuff.

 Comments welcome.

   Best Regards,

     Simon Lucas
     

Papers

 http://algoval.essex.ac.uk/papers/icpr2000.ps
 title: Automatic evaluation of algorithms over the Internet

 authors:  Simon Lucas and Kostas Sarampalis

 to appear: Proceedings ICPR 2000
 
 abstract:
   This paper describes a system for the automatic evaluation
   of algorithms (especially pattern recognition algorithms) over
   the Internet.  We present the case for such a system,
   discuss the system requirements and potential users and
   present an initial prototype.  We illustrate usage of
   the system with an evaluation of image distance measures
   used for face recognition.


 http://algoval.essex.ac.uk/papers/ijdar.ps
 title: Automatic evaluation of document image analysis
       algorithms over the Internet

Simon Lucas, Paul Beattie and Joanne Coy

Submission for IJDAR special issue on performance evaluation

Abstract:

   We have implemented a system for automatically evaluating
   document image analysis algorithms on datasets
   over the Internet.  In this paper we
   discuss some general issues related to this
   mode of evaluation and describe in particular
   the set-up of a common document image processing
   problem: locating regions of interest within an image.
   Traditionally, the evaluation of these
   and other document processing algorithms has usually been
   done in a rather manner,
   in most cases by the developers
   of the algorithms.  By contrast, our system
   makes the evaluation process simple, consistent, objective and
   automatic.  The system also provides detailed log files
   to give useful feedback to algorithm developers.


  
 

------------------------------------------------
Dr. Simon Lucas
Department of Electronic Systems Engineering
University of Essex
Colchester CO4 3SQ
United Kingdom

Tel:    (+44) 1206 872935
Fax:    (+44) 1206 872900
Email:  sml at essex.ac.uk
http://esewww.essex.ac.uk/~sml
-------------------------------------------------