No subject

[Chris J.C. Burges]

I strongly support the idea of introducing double blind reviewing at
NIPS.  If bias exists it is insidious and corrosive.  Further since
reviewing is very subjective, detecting bias can be very hard.  Worse,
it can occur unconsciously.  A close friend recently told me a story
about his reviewing two similar papers, one from a group he liked and
one from a group whose work he did not respect as much.  He started with
the paper from the second group, and half way through, he'd already
formed strong negative opinions on the work.  But then he was shocked to
discover that the paper was in fact from the first group.  He felt that
the incident uncovered a bias in his reviewing of which he was not
previously aware.

Let's look at the objections, so far, to blind reviewing:

John Lazzaro uses the example of Jan Hendrik Schon.  John is proposing
rejecting the paper due to the previous history of the author.  This is
exactly the kind of problem blind reviewing addresses.  Suppose that
Schon has mended his ways and his submission is actually ground
breaking, high quality research.  Do you want to reject it out of hand?
No, you want an unbiased, peer reviewed assessment of it.  The problem
of vetoing a given authors work should be decided by the editors, based
on past history, not by the reviewers - unless they themselves find
fraud in the submission.

Grace objects that writing a paper so as not to give a clue as to your
identity distorts the paper.  Also she points out that many authors put
their papers on their home page, so digging up the authorship of the
submission would be easy.  Regarding both of these points: even with
blind reviewing, authors can still leave a trail of bread crumbs as to
their identity if they wish.  No one is suggesting that they be forced
to make their identity as hard as possible to discern.  What is being
suggested, is that a barrier be erected, so that bias in a review would
have to be a much more conscious act that it is now.  I don't have to
put the paper on my home page if I feel that bias may exist.

The only other objection so far is that blind reviewing is costly.  But
that cost is hugely reduced with electronic submissions.  It need not be
cumbersome any more.  Also, coming up with examples of journals /
conferences that do not do blind reviewing is not convincing; one can
equally well come up with ones that do, e.g. ICCV, CHI, JASA (according
to http://www.acm.org/sigmod/record/issues/0003/chair.pdf , about 20% of
ACM sponsored conferences are double blind, so it can't be that hard).

-- Chris Burges