NIPS & double blind reviewing

[Sebastian Thrun]

I have followed the recent public discussion on double blind reviewing
with great interest. Many valid points were made. In many ways, this
discussion parallels a similar (but not so public) discussion that
took place a year ago, when I was elected to serve as the program
chair for NIPS 2002.

At the time, I talked to dozens of people, students, colleagues, NIPS
attendees, and researchers in other communities. I received a large
number of suggestions as to how to improve NIPS, and my policy was to
run them by as many people as I could, to get a general sense of what
to do. I also received a number of unsolicited Emails which I very
much appreciated (and still very much do).  Double blind reviewing was
one of these items that came up repeatedly, and which made it quickly
to the top of my list. Early on, I was quite determined to just do it,
not because I believe there is anything major broken with the current
system, but because it would have helped alleviating a perception that
has been voiced repeatedly in the present debate.

However, I ended up not implementing double blind reviewing, primarily
for two reasons. First, in individual conversations quite a few people
were opposed to it. I'd say about half of the people I asked were in
favor, half were against it.  The arguments in favor were pretty much
covered in the debate of the last days, and I believe many of them are
very valid. Those against included the one articulated by Sue Becker
(this is a very important point: every year we receive several
previously published papers, and sometimes we end up comparing those
word-by-word). They also included voices from the theory community,
who told me that properly checking a proof can take days.  The author
identity helps them to determine the rigor necessary in verifying a
new theoretical result, because there's no way they can afford to
spare several weeks for reviewing. I've never reviewed a lengthy proof
myself, so I took this advice for face value. The second reason,
however, ended up the determining factor: The present NIPS software
does not support double blind reviewing.  I don't want to bore this
list with technical details, but the change to the software would have
been major and would have incurred major costs. I felt the money was
better spent with student travel stipends, of which we gave quite a
few this year.

So in the end, I ended up putting my time into innovations that
received nearly unanimous support. Those included a later deadline,
the online pre-proceedings, the anonymous circulation of reviews among
other reviewers of the same paper, and the acquisition of external
funds used to draw in more students into our community.

One thing that might be less known to the community at large is that
NIPS has policies in place that effectively penalize insiders.  Just
look at this year's program: Not a single oral presentation was
co-authored by a program committee member. Most invited talks over the
past years were given by people outside the NIPS community (5 out of 6
this year).  You rarely find a person presenting orally in two
consecutive years. You rarely find a person serving on the program
committee for more than 3 years.  I've never participated in a NIPS PC
where an author's name or affiliation was cited as a reason to reject
a paper. For obvious reasons we don't make rejections public, but I
can assure you there we rejected quite a number of papers by some of
the best known researchers in the field - you'd be surprised! And
honestly, from all the conferences I attend, NIPS has been the singly
most successful meeting in terms of drawing in outsiders, at least in
my opinion. I don't mean this list to suggest that we should not
consider going to a double blind reviewing system. But I really
believe that NIPS strives to actively counter some of the concerns
voiced in the discussion of the past days.

I really appreciate the discussion of the past days. As in the past, I
very much welcome any suggestion as to how to improve NIPS, with or
without public debate. I believe NIPS has been a successful meeting.
..Okay, I might be a bit biased here ;-). But I probably speak for all
organizers when I say that we are committed to do whatever it takes to
keep it that way.

Happy Holiday Season!

Sebastian Thrun