NIPS & double blind reviewing

[Dale Schuurmans]

Oh this debate again.
I just thought I would save everyone a bunch of time 
by summarizing the standard arguments for and against 
double blind refereeing.  There is no point in seeing
it all come out piecemeal yet again.

(I know we've all seen these arguments before, but it is 
kind of interesting to see them collected in one place.)

-----------------------------------------------------------------------

Standard arguments against double-blind refereeing
--------------------------------------------------

1. It does not work anyway (it is still not double blind).

   a) A referee can usually identify the author(s) by recognizing
      the work or the writing style, or noticing the self-references.

   b) Most authors post their submitted papers on their homepages
      which makes it easy to discover their identities using Google 
      or Citeseer.

   c) Double blind refereeing can therefore be thought of as slightly
      fraudulent because it is not nearly as blind or objective as 
      outsiders would perceive it to be.

2. Knowing the author and institution can help to write a better review.

   a) The credibility and reputation of the author can help to assess 
      whether a complex piece of theoretical or experimental work has 
      been properly conducted.

   b) Someone who has already been publically convicted of scientific 
      fraud can falsify their data, write a convincing paper, and still
      get accepted under double blind refereeing.

3. It implicitly accuses referees of being biased.

   a) The double blind proposal suggests that referees are currently
      influenced inappropriately by knowing the identity of authors and 
      their institutions.  This shows a lack of trust in one's peers.

   b) Any potential bias that favors well known authors and institutions 
      is counter-balanced by an opposing bias against such authors and 
      institutions.

4. It is too much extra work for the author to mask their identity.

   a) Authors should not have to refer to themselves in the third 
      person when writing a scientific paper.  This requires additional 
      effort beyond simply reporting their scientific ideas, which 
      seems to be needless and wasteful.

   b) Authors can circumvent the system by divulging their identity
      anyway.

6. It is too expensive and time consuming to set up.

   a) There is a lot of extra effort required on the part of a program
      committee to set up a double blind reviewing system.

   b) It requires extra effort for referees to assess a paper if they
      do not know who's work they are assessing.

7. The current system is already working well.

   a) Everything is going fine with the single blind reviewing system 
      just the way it is.  Submissions are up.  Attendance is up.  
      Everybody is happy.  Why fix it if it isn't broken?


Standard conclusion against:
The benefits are debatable and do not justify the costs.


Standard arguments for double blind refereeing
----------------------------------------------

1. It objectively improves the reviews and the review system.

   a) Referees are relieved of the distraction of knowing who performed
      the work and where it was performed, which allows them to focus
      more clearly on the science instead of the scientist.

   b) The assessment of quality is based on exposition rather than 
      reputation.

   c) The benefit of the doubt is handed out more fairly to insiders
      and outsiders alike, based only on what was said---not who said it.

   d) Review quality is improved because it is harder for a referee
      to be careless and dismissive when they do not actually know 
      the identity of the author(s) or the institution they submitted from.

2. It greatly improves the perception of fairness.

   a) Authors of rejected papers are more likely to take the referees
      comments at face value and not presume that they were mistreated
      on the basis of who they were as opposed to what they said.

   b) This leads to less overall complaining, less emotionally charged 
      email sent to program chairs, and more effort spent on improving 
      the basic research and expository capabilities.

3. It allows easier access to outsiders and their ideas.

   a) It helps researchers from other communities cross disciplinary 
      boundaries because they can immediately establish their credibility
      based on relevant knowledge, rather than have their viewpoints 
      dismissed merely because their prior reputation has not yet reached 
      the target community.  This can encourage cross fertilization
      between research communities, and mitigate the over-convergence
      effects of small social groups.

   b) It helps young researchers who are just starting out, again because 
      reputation is factored out of the assessment.

   c) It helps researchers at lesser known institutions for the same
      reason.

4. It does not necessarily accuse referees of cognizant bias.

   a) No serious experimental science ignores the effect of subject and 
      experimenter bias in experimental design.  The point is not that
      subjects and experimenters consciously thwart objective investigation, 
      but that they cannot avoid affecting the outcome, even unconsciously, 
      if they know what is being investigated.  Similarly, referees are 
      subject to the same unconscious effects, whether they would like to 
      be or not.

5. Communities that adopt double blind refereeing system do not go back.

   a) Several high quality communities have adopted double blind 
      refereeing in the past decade (e.g. SIGGRAPH, SIGMOD, ICCV, ACL, 
      AAAI, IJCAI).  None of these communities have ever contemplated 
      reverting to a single blind refereeing system.  This suggests that 
      double blind refereeing has conferred a perceived tangible 
      improvement to these communities---at least in their eyes.
      

Standard conclusion for:
The only reasons why this isn't standard practice in every scientific
community are inertia and the fact that the main beneficiaries of 
the single blind refereeing system are exactly the ones responsible 
for changing it.

-----------------------------------------------------------------------

I hope I haven't misrepresented these arguments
or missed any significant ones.

Dale