5 ms·

The history of peer review, and how it was troubled from the start

61 points by Hooke 11y ago

ktRolster 11y ago
The peer review system is not just to help scientists, but to help outsiders have confidence that science is actually working (since outsiders often pay for it), and to help them understand what consensus is. That is my summary of the main point of the article. I think of peer review as kind of a low-pass filter: it keeps out the obvious junk. The problem is when science is hidden behind paywalls.
- return0 11y ago
  Usually the "obvious junk" is discarded by the editor of the journal itself (with help of reviewers if necessary). >since outsiders often pay for it. I doubt the percentage of "outsiders" actually paying for individual papers is significant. Overwhelmingly, it's libraries paying subscriptions.
  - taejo 11y ago
    I think GP is referring to outsiders paying for the science (through public grants) rather than the publication.
- vlehto 11y ago
  I thought peer review and consensus should have nothing to do with each other. Peer review is about how solid the research is, no matter the results. (At least it should be.) Consensus is averaged educated guess of the experts in the field. It doesn't necessarily have anything to do with science. It has been serious problem on several fields when authority figure has strong, but false opinion. "Science advances one funeral at a time." - Max Planck And then there are things like systematic reviews in soft sciences and replications in hard sciences. These are used to find the underlying truth so that it's not biased by anybodies opinion. (At least that's what they aim at.)
  - ktRolster 11y ago
    That sounds reasonable, but the article says it was also about demonstrating consensus: "In the 1960s, refereeing emerged as a symbol of objective judgement and consensus in science."
    - tamana 11y ago
      That statement is absurd on its face, the system is designed to be subjective, not objective.
      - ktRolster 11y ago
        Your assessment would be more convincing if you had actually read the article, or even had any evidence at all to back up your point. Who are people going to believe? An article in Nature about the history of peer review, or you, who has no evidence?
- aab0 11y ago
  Exactly. We owe the present universality of 'peer review' to the post-WWII expansion of government-funded military-industry-university science (see also Vannevar Bush): > The idea that any legitimate scientific journal ought to implement a formal referee system began to take hold in the decades following the Second World War...The very phrase 'scientific community' dates from this time. Researchers wanted to preserve autonomy while holding on to the massive government funding that had come their way since the Second World War. Allocations for basic research in the United States, for instance, swelled by a factor of 25 in less than a decade9. 'Peer review' was a term borrowed from the procedures that government agencies used to decide who would receive financial support for scientific and medical research 'Peer review', like citation counts, is essentially bureaucratic: it is the scientific form of 'no one was ever fired for buying IBM'. ('But I relied on a peer-reviewed paper!' goes the cry. You also see this excuse often said by authors of rubbish papers: 'you can't criticize my paper, it's been peer-reviewed!') It is an attempt to resolve the principal-agent problem (exacerbated by the huge increase in numbers of researchers and the prospects of power and wealth now offered by science) by serving as a stamp of 'scientific quality' which is easily externally verifiable by ignorant outsiders such as funders. As such, it is considered indispensable by governments/universities, and it helps build barriers to entry by established publishers like Elsevier. This is why 'peer review', despite the lack of any real evidence and the occasional study showing experimentally that it is an atrocious filter which frequently discriminates against the best research and select for the worst, has throve over the past century.
  - jessriedel 11y ago
    I think you might be conflating assessments of importance with assessments of soundness. Referees often disagree intractably on importance, and its been shown to be a very poor predictor of future important results. But referees disagree much more rarely on soundness, and when they do they are often resolved through discussion. Papers that are sound but deemed uninteresting get published elsewhere. The importance checking is debatable, but soundness checking is crucial and reasonably objective.
  - tamana 11y ago
    There is no peer reviewed evidence in favor of peer review?
- mattkrause 11y ago
  If "outsiders" actually knew how peer review worked, I don't think it would give them a great deal of confidence. Peer review literally means that a few other people (often two or three, maybe five at the outside) looked at a manuscript and failed to find anything too objectionable. Since these decisions are just based on a few people spending a few hours, you might expect this signal to be a noisy measure of quality, and it is. NIPS, an excellent machine learning/neuroscience conference, ran an experiment where 10% of the submissions were reviewed by two separate program committees. They found considerable disagreement between the two committees (~60% of the selected papers were different) More details here: http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experiment/fulltext http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experime...
  - tamana 11y ago
    More to the point, peer review picks an limited sample of papers to accept, based on interestingness and X printed pages per month per journal. There is no peer review system that will review and accept every valid research project.
adhadh 11y ago
The idea of an anonymous judge speaking for the public just doesn't make much sense these days.
- toomim 11y ago
  The public should be the judge.
  - deleted 11y ago
    [deleted]
  - adrianN 11y ago
    The public has no clue what the papers are about and doesn't really care either. Passing peer review is just the first step for a paper though. It also needs to gather citations to avoid being forgotten. It's up to the scientists to decide which paper to cite and they are usually a lot more qualified to do so.
  - pessimizer 11y ago
    Paging Dr. Wakefield... Paging Dr. Lysenko...
100ideas 11y ago
So much for philosopher-kings...
dang 11y ago
This is an excellent piece about the origins and history of peer review. Everyone interested in the ongoing controversies about replicability and bias in science, which appear frequently on HN's front page (including now at https://news.ycombinator.com/item?id=11535169 https://news.ycombinator.com/item?id=11535169) ought to read it.
methehack 11y ago
Do some of the same problems apply to peer code review?

61 points by Hooke 11y ago

The peer review system is not just to help scientists, but to help outsiders have confidence that science is actually working (since outsiders often pay for it), and to help them understand what consensus is. That is my summary of the main point of the article. I think of peer review as kind of a low-pass filter: it keeps out the obvious junk. The problem is when science is hidden behind paywalls.

return0 11y ago
Usually the "obvious junk" is discarded by the editor of the journal itself (with help of reviewers if necessary). >since outsiders often pay for it. I doubt the percentage of "outsiders" actually paying for individual papers is significant. Overwhelmingly, it's libraries paying subscriptions.
- taejo 11y ago
  I think GP is referring to outsiders paying for the science (through public grants) rather than the publication.
vlehto 11y ago
I thought peer review and consensus should have nothing to do with each other. Peer review is about how solid the research is, no matter the results. (At least it should be.) Consensus is averaged educated guess of the experts in the field. It doesn't necessarily have anything to do with science. It has been serious problem on several fields when authority figure has strong, but false opinion. "Science advances one funeral at a time." - Max Planck And then there are things like systematic reviews in soft sciences and replications in hard sciences. These are used to find the underlying truth so that it's not biased by anybodies opinion. (At least that's what they aim at.)
- ktRolster 11y ago
  That sounds reasonable, but the article says it was also about demonstrating consensus: "In the 1960s, refereeing emerged as a symbol of objective judgement and consensus in science."
  - tamana 11y ago
    That statement is absurd on its face, the system is designed to be subjective, not objective.
    - ktRolster 11y ago
      Your assessment would be more convincing if you had actually read the article, or even had any evidence at all to back up your point. Who are people going to believe? An article in Nature about the history of peer review, or you, who has no evidence?
aab0 11y ago
Exactly. We owe the present universality of 'peer review' to the post-WWII expansion of government-funded military-industry-university science (see also Vannevar Bush): > The idea that any legitimate scientific journal ought to implement a formal referee system began to take hold in the decades following the Second World War...The very phrase 'scientific community' dates from this time. Researchers wanted to preserve autonomy while holding on to the massive government funding that had come their way since the Second World War. Allocations for basic research in the United States, for instance, swelled by a factor of 25 in less than a decade9. 'Peer review' was a term borrowed from the procedures that government agencies used to decide who would receive financial support for scientific and medical research 'Peer review', like citation counts, is essentially bureaucratic: it is the scientific form of 'no one was ever fired for buying IBM'. ('But I relied on a peer-reviewed paper!' goes the cry. You also see this excuse often said by authors of rubbish papers: 'you can't criticize my paper, it's been peer-reviewed!') It is an attempt to resolve the principal-agent problem (exacerbated by the huge increase in numbers of researchers and the prospects of power and wealth now offered by science) by serving as a stamp of 'scientific quality' which is easily externally verifiable by ignorant outsiders such as funders. As such, it is considered indispensable by governments/universities, and it helps build barriers to entry by established publishers like Elsevier. This is why 'peer review', despite the lack of any real evidence and the occasional study showing experimentally that it is an atrocious filter which frequently discriminates against the best research and select for the worst, has throve over the past century.
- jessriedel 11y ago
  I think you might be conflating assessments of importance with assessments of soundness. Referees often disagree intractably on importance, and its been shown to be a very poor predictor of future important results. But referees disagree much more rarely on soundness, and when they do they are often resolved through discussion. Papers that are sound but deemed uninteresting get published elsewhere. The importance checking is debatable, but soundness checking is crucial and reasonably objective.
- tamana 11y ago
  There is no peer reviewed evidence in favor of peer review?
mattkrause 11y ago
If "outsiders" actually knew how peer review worked, I don't think it would give them a great deal of confidence. Peer review literally means that a few other people (often two or three, maybe five at the outside) looked at a manuscript and failed to find anything too objectionable. Since these decisions are just based on a few people spending a few hours, you might expect this signal to be a noisy measure of quality, and it is. NIPS, an excellent machine learning/neuroscience conference, ran an experiment where 10% of the submissions were reviewed by two separate program committees. They found considerable disagreement between the two committees (~60% of the selected papers were different) More details here: http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experiment/fulltext http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experime...
- tamana 11y ago
  More to the point, peer review picks an limited sample of papers to accept, based on interestingness and X printed pages per month per journal. There is no peer review system that will review and accept every valid research project.

adhadh 11y ago

The idea of an anonymous judge speaking for the public just doesn't make much sense these days.

toomim 11y ago
The public should be the judge.
- deleted 11y ago
  [deleted]
- adrianN 11y ago
  The public has no clue what the papers are about and doesn't really care either. Passing peer review is just the first step for a paper though. It also needs to gather citations to avoid being forgotten. It's up to the scientists to decide which paper to cite and they are usually a lot more qualified to do so.
- pessimizer 11y ago
  Paging Dr. Wakefield... Paging Dr. Lysenko...

100ideas 11y ago

So much for philosopher-kings...

dang 11y ago

This is an excellent piece about the origins and history of peer review. Everyone interested in the ongoing controversies about replicability and bias in science, which appear frequently on HN's front page (including now at https://news.ycombinator.com/item?id=11535169 https://news.ycombinator.com/item?id=11535169) ought to read it.

methehack 11y ago

Do some of the same problems apply to peer code review?