3 ms·
Exactly. We owe the present universality of 'peer review' to the post-WWII expansion of government-funded military-industry-university science (see also Vanneva
by aab0 10y ago
Exactly. We owe the present universality of 'peer review' to the post-WWII expansion of government-funded military-industry-university science (see also Vannevar Bush):
> The idea that any legitimate scientific journal ought to implement a formal referee system began to take hold in the decades following the Second World War...The very phrase 'scientific community' dates from this time. Researchers wanted to preserve autonomy while holding on to the massive government funding that had come their way since the Second World War. Allocations for basic research in the United States, for instance, swelled by a factor of 25 in less than a decade9. 'Peer review' was a term borrowed from the procedures that government agencies used to decide who would receive financial support for scientific and medical research
'Peer review', like citation counts, is essentially bureaucratic: it is the scientific form of 'no one was ever fired for buying IBM'. ('But I relied on a peer-reviewed paper!' goes the cry. You also see this excuse often said by authors of rubbish papers: 'you can't criticize my paper, it's been peer-reviewed!') It is an attempt to resolve the principal-agent problem (exacerbated by the huge increase in numbers of researchers and the prospects of power and wealth now offered by science) by serving as a stamp of 'scientific quality' which is easily externally verifiable by ignorant outsiders such as funders.
As such, it is considered indispensable by governments/universities, and it helps build barriers to entry by established publishers like Elsevier. This is why 'peer review', despite the lack of any real evidence and the occasional study showing experimentally that it is an atrocious filter which frequently discriminates against the best research and select for the worst, has throve over the past century.
- jessriedel 10y agoI think you might be conflating assessments of importance with assessments of soundness. Referees often disagree intractably on importance, and its been shown to be a very poor predictor of future important results. But referees disagree much more rarely on soundness, and when they do they are often resolved through discussion. Papers that are sound but deemed uninteresting get published elsewhere. The importance checking is debatable, but soundness checking is crucial and reasonably objective.
- tamana 10y agoThere is no peer reviewed evidence in favor of peer review?
- mattkrause 10y agoThere are, but they are fairly discouraging. Peters and Ceci (1982) resubmitted a dozen articles after changing the authors' names and affiliations and retyping the manuscript (this is 1982, after all). Only 1/12 was accepted on the second go-around, while 8/12 were rejected for quality-related reasons. Rothwell and Martin (2000) examined data from neuroscience journals and conferences (2 each), where reviewers grouped papers into three categories (high/medium/low), and found virtually no agreement (they calcualted kappa, a measure of agreement that ranges from 0 (none) to 1 (perfect), to be ~0.1). Someone (sorry, can't find the citation; it's in the last year or two) recently analyzed attempted to link grant scores to subsequent publications to see if reviewers could prospectively find "good" proposals. The reviewers could successfully identify terrible proposals and brilliant ones, but had a terrible time with those in the vast murky middle.