NIPS & double blind reviewing

[tbreuel@parc.com]

On Wed, Dec 18, 2002 at 06:08:48AM -0800, Sandro Mussa-Ivaldi wrote:
> If you are young and unknown: the paper was rejected because you
> dont  belong to that old boys network. If you are more seasoned and famous:
> the paper was rejected because evidently the reviewer hates my guts. Both way
> it hurts. But it is far better than: my paper was rejected because it was a
> bunch of bull.  Which comes right before suicide.

Actually, I think a far more common situation is that the paper was
rejected because the reviewer didn't understand it despite making
a reasonable effort.

Often, this is because I, the author, didn't explain things well 
enough.  That's fine, I can live with that.  I do expect enough
feedback from the reviewer to be able to fix things, however.

But frequently, it is also because either the reviewer has only a
superficial knowledge of the subject, because the reviewer has a pet
theory and will reject any theory that doesn't agree with it, or because
he just didn't bother reading the paper very well.

Well-known researchers in a field are given the benefit of the doubt,
so reviewers will try harder to understand their papers, while papers by
less well-known researchers are dismissed quickly.  In fact, people on
this list have even expressed the opinion that this is the way things
should be done--taking account the reputation of the researcher in the
assessment of the strength and validity of results.  I strongly
disagree with that view.  Double blind reviewing addresses this specific
problem because reviewers have less opportunity to take into account
the reputation of the authors in their evaluation.

Double blind reviewing doesn't fix many other problems, though.  Reviewers
who just don't know the particular subject well, or reviewers who have
a pet theory and dislike any other theory, simply cannot perform a good
review, double blind or not.

Addressing those issues is the responsibility of editors.  Editors
should examine the reviews returned by the reviewers, and they should
look carefully at the feedback from authors in response to a review.
Based on those, a responsible editor can spot problems with reviewers
help the reviewer to improve his approach to reviewing, or avoid using
the same reviewer again in the future.

While that approach isn't feasible for a conference, CVPR this year,
in addition to using double-blind reviewing, is soliciting feedback from
authors on their reviews (but not revised manuscripts); those are used
to identify out particularly problematic or sloppy reviewers and remove
their recommendations from consideration.

Incidentally, the stated rule of CVPR is also that any papers that
attempt to sneak in identifications of their authors will be rejected;
I assume that also refers to usage like "In my previous paper [3]..." or
posting the paper on a web site prior to reviewing.

It would seem sensible to me for NIPS to adopt the same rules.

Thomas.