NIPS & double blind reviewing

[Rushi Bhatt]

Dear Colleagues,

I think Prof. McClelland and Dr. Burges have raised
some very valid points. If nothing else, as (I think)
pointed out in the UAI discussion thread referred to
earlier, an explicitly double-blind review process
will re-affirm the requirement that the review process
needs to be as bias free as possible.

Now to add my $0.02 worth:

One might want to look at JAMA 1994 Jul
13;272(2):147-9 (abstract pasted at the end of this
message) which suggests that double-blind reviewed
journal papers are cited more often. Of course, the
analysis has a bunch of caveats and the analysis
should have been more rigorous. Also, it seems that
citation patterns are different in different
scientific fields, so the conclusions may not
translate. 

Perhaps one could perform a similar (but more
rigorous) analysis using data from conferences that
already have a double-blind reveiw process in our
field.

Would any ex-conference chairs be willing to
contribute some data and help me set up the analysis?
Let me know :-)

Regards,
Rushi Bhatt

PhD student
CNS,
Boston University.

_____________________


 JAMA 1994 Jul 13;272(2):147-9 

    A citation analysis of the impact of blinded peer
review.

    Laband DN, Piette MJ.

    Department of Economics and Finance, Salisbury
State University, Md.

    OBJECTIVE--To determine whether articles published
in journals using blinded peer review receive
significantly more or fewer citations than those
published in journals using nonblinded peer review.
DESIGN--Drawing from a sample of 1051 full articles
published in 28 economics journals during 1984, we
used nonlinear regression and ordered probit
techniques to estimate the impact of blinded peer
review on citations of these articles in 1985 through
1989. OUTCOMES--Citations of articles.
RESULTS--Articles published in journals using blinded
peer review were cited significantly more than
articles published in journals using nonblinded peer
review, controlling for a variety of author, article,
and journal attributes. CONCLUSIONS--Nonblinded peer
review apparently suffers from type I error to a
greater extent than blinded peer review. That is,
journals using nonblinded peer review published a
larger fraction of papers that should not have been
published than do journals using blinded peer review.
When reviewers know the identity of the author(s) of
an article, they are able to (and evidently do)
substitute particularistic criteria for universalistic
criteria in their evaluative process.

PMID: 8015128 [PubMed - indexed for MEDLINE]

--- Jay McClelland <jlm at cnbc.cmu.edu> wrote:
> 
> For what it is worth, the American Psychological
> Association,
> which publishes many journals, allows authors the
> option to submit
> their papers 'blinded' if they choose, in case the
> authors
> are concerned about bias.  I don't know the details
> on how
> many papers are in fact submitted that way.  I
> occasionally
> review for APA journals, and have never seen one
> that was
> blinded.
> 
> My own opinion is that the prior record of the
> individuals
> whose papers are being reviewed provides information
> that
> on balance improves the review process. 
> Unfortunately there's no
> doubt that bias is involved and worthy papers are
> sometimes 
> adversely affected.  The bias includes cultural
> standards of 
> evaluation that may not be objectively optimal along
> with 
> more specific biases based on reactions to the work
> of 
> particular individuals or labs.   It is difficult
> for blinding
> to remove all of this sort of bias, and sometimes as
> others have
> said author identity is still apparent.  As far as I
> can
> tell these issues can only be addressed by keeping
> them in
> mind and carefully considering them in the selection
> of
> reviewers (and of course in selecting those who
> select the 
> reviewers!).  A reputation for fair and open
> mindedness
> is a crucial consideration in editor/reviewer
> selection. Also,
> those who have concerns should feel they have the
> opportunity
> to have their concerns taken into consideration.
> 
>  -- Jay McClelland




__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com