NIPS & double blind reviewing

[Steven M. Kemp]

I have located a second bit of empirical evidence about the effects of open
reviewing.  The study was done in a realm even *more* removed from NIPS
than orchestral music.  This was a study conducted by psychologists about
peer reviewing in psychological journals.  B-)

Peters, D. P., & Ceci, S. J.  (1982).  Peer-review practices of
psychological journals:  The fate of published articles submitted again.
The Behavioral and Brain Sciences, 5, 187-255.

I take the following synopsis from Bordens & Abbott (1996).  Research
Design and Methods: A process approach.  (Mountain View, CA: Mayfield).

Apparently, Peters and Ceci took 12 already published articles, changed the
author names and affiliations to fake (and hence unknown) ones, and added a
few cosmetic alterations to hide the fact that it was the same paper.  All
12 had originally had at least one prestige author from a prestige
institution.  All were re-typed and sent back in to the very same journals
that had originally published the journal (often to the same editor).
Three of the 12 were detected as copies of already published papers and
rejected.  Of the 9 that were left undetected, all but one were rejected by
the very same journals that had published them earlier!  And NONE of those
8 were rejected because they duplicated already published research.

At a very minimum, the fact that the only substantive change in the
manuscripts submitted was the replacement of known authors from prestigious
institutions with names of complete unknowns raises serious questions about
biases in open review.

If folks are concerned that this was psychology and not computer science,
for a modest fee I and some of my fellow psychologists will be delighted to
perform the same experiment on some NIPS volunteers.  B-)

steve kemp


At 9:03 AM 12/18/02, Eric Baum wrote:
>There is at least one historical precedent where evidence indicates
>that bias was influencing selection, in spite of the fact
>that such bias was denied by the referees, and where the bias was
>subsequently ameliorated through blind reviewing.
>
>
>From http://reason.com/9711/ci.ng.orchestral.shtml
>
>
>
>REASON * November 1997
>
>Orchestral Maneuvers
>
>By Nick Gillespie
>
>A recent study from the National Bureau of Economic Research applies
>the concept of a level playing field to the symphonic stage. In
>"Orchestrating Impartiality," economists Claudia Goldin and Cecelia
>Rouse demonstrate that female orchestra musicians have benefitted
>hugely from the use of "blind" auditions, in which candidates perform
>out of the sight of evaluators.
>
>In 1970 female musicians made up only 5 percent of players in the
>country's top orchestras...
>
>But beginning in the '70s and '80s, more and more of the orchestras
>switched to blind auditions, partly to avoid charges of such bias.
>Female musicians currently make up 25 percent of the "Big Five."
>Through an analysis of orchestral management files and audition
>records, Goldin and Rouse conclude that blind auditions increased by
>50 percent the probability that a woman would make it out of early
>rounds. And, they say, the procedure explains between 25 percent
>and 46 percent of the increase in women in orchestras from 1970 to
>1996.



  >>>>>>>>>>>>>>>>>>>>>         <<<<<<<<<<<<<<<<<<<<<<<<

Steven M. Kemp                |
Department of Psychology      | email:  steve_kemp at unc.edu
Davie Hall, CB# 3270          |
University of North Carolina  |
Chapel Hill, NC 27599-3270    |   fax: (919) 962-2537

Visit our WebSite at:  http://www.unc.edu/~skemp/

  >>>>>>>>>>>>>>>>>>>>>         <<<<<<<<<<<<<<<<<<<<<<<<

The laws of mind [are] themselves of so fluid a character as to simulate
divergences from law.  -- C. S. Peirce (Collected Papers, 6.101).