Connectionists: CFP - Workshop on Evaluation Methods for Machine Learning II

[William Elazmeh]

Call for participation

The AAAI-2007 Workshop on Evaluation Methods for Machine Learning II


Description:
============
The purpose of this workshop is to continue the lively and interesting debate
started last year at the AAAI 2006 workshop on Evaluation Methods for Machine
learning. The previous workshop was successful on the following points:

- It established that the current means of evaluating learning algorithms has
  some serious drawbacks.
- It established that there are many important properties of algorithms that
  should be measured, requiring more than a single evaluation  metric.
- It established that algorithms must be tested under many different conditions.
- It established that the UCI data sets do not reflect the variety of domains
  to which algorithms are applied in practice.

For this year's workshop, we are intending to address, in a more specific
fashion, some of the topics that were raised at last year's workshop and some
new ones. Last year's participants are invited to submit papers reflecting the
evolution of their views within the year, or new ideas. Researchers, or
practitioners, interested in this issue who did not get a chance to submit a
paper last year are invited to do so this year.


Topics:
=======
We invite position papers and technical papers addressing three main topics,
and their subtopics:

Evaluation Metrics:
* The efficacy of existing evaluation metrics
* The need for new metrics
* Useful metrics from the other fields
* Single number summaries versus curves
* New performance visualization methods

Statistical Issues:
* Cross-validation vs. bootstrap
* Bias vs. variance
* Parametric vs. non-parametric
* The power of tests
* Sampling methods
* Multiple comparisons

Data Sets:
* The (over) use of community repositories such as UCI
* Concept Drift
* Synthetic, or semi-synthetic, data sets


Format and Attendance:
======================
The workshop will consist of:

Invited talks: We are planning to have several invited speakers. Some, from
outside of our research community, will be able to criticize our accepted
practices from an external point of view. Some, from inside our community,
will discuss how we could improve on our current practices.

Panel Discussion: Our invited speakers will be asked to engage each other on
the various issues surrounding the problem of evaluation in Machine Learning
at the end of the workshop. The audience will be strongly encouraged to
participate in the discussion.

Presentations: The papers accepted to the workshop will be presented throughout
the day between the various invited talks. Papers will be grouped by theme, in
order to facilitate discussion at the end of each session.

Workshop attendance is open to the public and is estimated at 20-25 attendees.
Priority will be given to those active participants in the workshop (paper
authors or speakers).


Submission:
===========
Authors are invited to submit papers on the topics outlined above or other
related issues. Both position and technical reports will be considered for this
workshop. To promote a lively event, with plenty of discussion, the organizers
are very interested in papers taking strong positions on the issues listed
above. Workshop papers should not exceed 6 pages using the AAAI Style.
Submissions should be made electronically in PDF or Postscript format and
should be sent (no later than April 1, 2007) by email to William Elazmeh:

			welazmeh at site.uottawa.ca

	Resubmissions and position papers (of at least 1 page) are welcome.


Important Dates:
================
* Submission deadline: April 1, 2007
* Notification date: April 25, 2007
* Camera ready deadline: May 15, 2007.
* The workshop date: July 22 or 23, 2007.
* AAAI-07 conference is to be held on July 22-26, 2007, in Vancouver, British
  Columbia, Canada.


Organizers:
===========
Chris Drummond
NRC Institute for Information Technology, Canada
Email: chris.drummond at nrc-cnrc.gc.ca

William Elazmeh (main contact)
School of Information Technology and Engineering, University of Ottawa,
800 King Edward Ave., room 5-105, Ottawa, Ontario, K1N 6N5 Canada
Telephone: +1 (613) 562 5800 Extension 6699
Fax.: +1 (613) 562 5664
Email: welazmeh at site.uottawa.ca

Nathalie Japkowicz
School of Information Technology and Engineering
University of Ottawa, Canada
Email: nat at site.uottawa.ca

Sofus A. Macskassy
Fetch Technologies
United States of America
Email: sofmac at fetch.com


Program Committee:
==================
Nitesh Chawla, University of Notre Dame, USA
Peter Flach, University of Bristol, UK
Alek Kotcz, Microsoft Research, USA
Nicolas Lachiche, Louis Pasteur University, France
Sylvain Letourneau, National Research Council, Canada
Dragos Margineantu, The Boeing Company, Phantom Works, USA
Stan Matwin, University of Ottawa, Canada
Kiri Wagstaff, NASA Jet Propulsion Laboratory, USA
Bianca Zadrozny, Fluminense Federal University, Brazil


Workshop Website:
=================
http://www.site.uottawa.ca/~welazmeh/conferences/AAAI-07/workshop/