NIPS, blind reviewing, and elitism

[Michael C. Mozer]

As NIPS*95 program chair, I want to respond to the issue of blind reviewing.
We have considered this idea over the past few months, and in balance the 
costs seem to outweigh the benefits.  Most of the arguments for and against 
blind reviewing were stated clearly in earlier messages.

NIPS is an elite conference in that researchers tend to self-select their best
work for submission, and even then only 25-30% of the submissions are accepted.
Further, researchers who do good work one year and have papers accepted are 
likely to do good work in the future, so it is not surprising that there is 
a core group of consistent contributors to the conference, even without any 
bias.  (Serving on the program committee before, I witnessed an opposite 
bias -- a bias against accepting multiple papers by an individual who 
had several strong submissions, and against awarding a talk to the same 
individual in successive years.)

Prior to 1994, there was likely some validity to the perception that NIPS 
"insiders" had an edge.  As a reviewer, it was difficult to evaluate work
solely on the basis of extended abstracts; in borderline cases, it helped to 
know the authors' track records.  After Dave Touretzky changed the submission
format to full papers in 1994, reviewers and program committee members to
whom I've spoken seemed satisfied that the reviewing process was objective.  
I suspect that the perception of unfairness may linger for a few years, but 
any such reality was squelched by the new submission format.

If you felt the reviewing process was not objective in 1994, I would like to 
hear your story.  (It is possible to communicate anonymously; to find out more 
about this service, send mail to help at anon.penet.fi.)  Several points of note 
before expressing a complaint:  (1) Many good submissions were ultimately 
rejected, simply because the submission pool tends to be of high quality and 
the number of accepted papers is limited by a maximum page count of the 
proceedings volume.  (2) Most of the people who have served as program and 
general chairs at NIPS have had papers rejected, including the current chairs!
The reviewers are the primary decision makers.  (3) Reviewers are not always
as competent and dilligent as one might like, although NIPS uses three 
reviewers per paper--plus an area chair to arbitrate--to mitigate the 
consequences of an inappropriate review.

I gladly welcome comments and suggestions aimed at broadening the constituency
of NIPS without lowering the meeting's quality.  I will summarize the feedback
I receive to the net.

Cheers, 

Mike Mozer