4 ms·

There is no peer reviewed evidence in favor of peer review?

• mattkrause 11y ago
  There are, but they are fairly discouraging. Peters and Ceci (1982) resubmitted a dozen articles after changing the authors' names and affiliations and retyping the manuscript (this is 1982, after all). Only 1/12 was accepted on the second go-around, while 8/12 were rejected for quality-related reasons. Rothwell and Martin (2000) examined data from neuroscience journals and conferences (2 each), where reviewers grouped papers into three categories (high/medium/low), and found virtually no agreement (they calcualted kappa, a measure of agreement that ranges from 0 (none) to 1 (perfect), to be ~0.1). Someone (sorry, can't find the citation; it's in the last year or two) recently analyzed attempted to link grant scores to subsequent publications to see if reviewers could prospectively find "good" proposals. The reviewers could successfully identify terrible proposals and brilliant ones, but had a terrible time with those in the vast murky middle.