4 ms·

The peer review system is not just to help scientists, but to help outsiders have confidence that science is actually working (since outsiders often pay for it)

The peer review system is not just to help scientists, but to help outsiders have confidence that science is actually working (since outsiders often pay for it), and to help them understand what consensus is. That is my summary of the main point of the article. I think of peer review as kind of a low-pass filter: it keeps out the obvious junk. The problem is when science is hidden behind paywalls.

return0 11y ago
Usually the "obvious junk" is discarded by the editor of the journal itself (with help of reviewers if necessary). >since outsiders often pay for it. I doubt the percentage of "outsiders" actually paying for individual papers is significant. Overwhelmingly, it's libraries paying subscriptions.
- taejo 11y ago
  I think GP is referring to outsiders paying for the science (through public grants) rather than the publication.
vlehto 11y ago
I thought peer review and consensus should have nothing to do with each other. Peer review is about how solid the research is, no matter the results. (At least it should be.) Consensus is averaged educated guess of the experts in the field. It doesn't necessarily have anything to do with science. It has been serious problem on several fields when authority figure has strong, but false opinion. "Science advances one funeral at a time." - Max Planck And then there are things like systematic reviews in soft sciences and replications in hard sciences. These are used to find the underlying truth so that it's not biased by anybodies opinion. (At least that's what they aim at.)
- ktRolster 11y ago
  That sounds reasonable, but the article says it was also about demonstrating consensus: "In the 1960s, refereeing emerged as a symbol of objective judgement and consensus in science."
  - tamana 11y ago
    That statement is absurd on its face, the system is designed to be subjective, not objective.
    - ktRolster 11y ago
      Your assessment would be more convincing if you had actually read the article, or even had any evidence at all to back up your point. Who are people going to believe? An article in Nature about the history of peer review, or you, who has no evidence?
aab0 11y ago
Exactly. We owe the present universality of 'peer review' to the post-WWII expansion of government-funded military-industry-university science (see also Vannevar Bush): > The idea that any legitimate scientific journal ought to implement a formal referee system began to take hold in the decades following the Second World War...The very phrase 'scientific community' dates from this time. Researchers wanted to preserve autonomy while holding on to the massive government funding that had come their way since the Second World War. Allocations for basic research in the United States, for instance, swelled by a factor of 25 in less than a decade9. 'Peer review' was a term borrowed from the procedures that government agencies used to decide who would receive financial support for scientific and medical research 'Peer review', like citation counts, is essentially bureaucratic: it is the scientific form of 'no one was ever fired for buying IBM'. ('But I relied on a peer-reviewed paper!' goes the cry. You also see this excuse often said by authors of rubbish papers: 'you can't criticize my paper, it's been peer-reviewed!') It is an attempt to resolve the principal-agent problem (exacerbated by the huge increase in numbers of researchers and the prospects of power and wealth now offered by science) by serving as a stamp of 'scientific quality' which is easily externally verifiable by ignorant outsiders such as funders. As such, it is considered indispensable by governments/universities, and it helps build barriers to entry by established publishers like Elsevier. This is why 'peer review', despite the lack of any real evidence and the occasional study showing experimentally that it is an atrocious filter which frequently discriminates against the best research and select for the worst, has throve over the past century.
- jessriedel 11y ago
  I think you might be conflating assessments of importance with assessments of soundness. Referees often disagree intractably on importance, and its been shown to be a very poor predictor of future important results. But referees disagree much more rarely on soundness, and when they do they are often resolved through discussion. Papers that are sound but deemed uninteresting get published elsewhere. The importance checking is debatable, but soundness checking is crucial and reasonably objective.
- tamana 11y ago
  There is no peer reviewed evidence in favor of peer review?
mattkrause 11y ago
If "outsiders" actually knew how peer review worked, I don't think it would give them a great deal of confidence. Peer review literally means that a few other people (often two or three, maybe five at the outside) looked at a manuscript and failed to find anything too objectionable. Since these decisions are just based on a few people spending a few hours, you might expect this signal to be a noisy measure of quality, and it is. NIPS, an excellent machine learning/neuroscience conference, ran an experiment where 10% of the submissions were reviewed by two separate program committees. They found considerable disagreement between the two committees (~60% of the selected papers were different) More details here: http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experiment/fulltext http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experime...
- tamana 11y ago
  More to the point, peer review picks an limited sample of papers to accept, based on interestingness and X printed pages per month per journal. There is no peer review system that will review and accept every valid research project.

return0 11y ago

Usually the "obvious junk" is discarded by the editor of the journal itself (with help of reviewers if necessary). >since outsiders often pay for it. I doubt the percentage of "outsiders" actually paying for individual papers is significant. Overwhelmingly, it's libraries paying subscriptions.

taejo 11y ago
I think GP is referring to outsiders paying for the science (through public grants) rather than the publication.

vlehto 11y ago

I thought peer review and consensus should have nothing to do with each other. Peer review is about how solid the research is, no matter the results. (At least it should be.) Consensus is averaged educated guess of the experts in the field. It doesn't necessarily have anything to do with science. It has been serious problem on several fields when authority figure has strong, but false opinion. "Science advances one funeral at a time." - Max Planck And then there are things like systematic reviews in soft sciences and replications in hard sciences. These are used to find the underlying truth so that it's not biased by anybodies opinion. (At least that's what they aim at.)

ktRolster 11y ago
That sounds reasonable, but the article says it was also about demonstrating consensus: "In the 1960s, refereeing emerged as a symbol of objective judgement and consensus in science."
- tamana 11y ago
  That statement is absurd on its face, the system is designed to be subjective, not objective.
  - ktRolster 11y ago
    Your assessment would be more convincing if you had actually read the article, or even had any evidence at all to back up your point. Who are people going to believe? An article in Nature about the history of peer review, or you, who has no evidence?

aab0 11y ago

Exactly. We owe the present universality of 'peer review' to the post-WWII expansion of government-funded military-industry-university science (see also Vannevar Bush): > The idea that any legitimate scientific journal ought to implement a formal referee system began to take hold in the decades following the Second World War...The very phrase 'scientific community' dates from this time. Researchers wanted to preserve autonomy while holding on to the massive government funding that had come their way since the Second World War. Allocations for basic research in the United States, for instance, swelled by a factor of 25 in less than a decade9. 'Peer review' was a term borrowed from the procedures that government agencies used to decide who would receive financial support for scientific and medical research 'Peer review', like citation counts, is essentially bureaucratic: it is the scientific form of 'no one was ever fired for buying IBM'. ('But I relied on a peer-reviewed paper!' goes the cry. You also see this excuse often said by authors of rubbish papers: 'you can't criticize my paper, it's been peer-reviewed!') It is an attempt to resolve the principal-agent problem (exacerbated by the huge increase in numbers of researchers and the prospects of power and wealth now offered by science) by serving as a stamp of 'scientific quality' which is easily externally verifiable by ignorant outsiders such as funders. As such, it is considered indispensable by governments/universities, and it helps build barriers to entry by established publishers like Elsevier. This is why 'peer review', despite the lack of any real evidence and the occasional study showing experimentally that it is an atrocious filter which frequently discriminates against the best research and select for the worst, has throve over the past century.

jessriedel 11y ago
I think you might be conflating assessments of importance with assessments of soundness. Referees often disagree intractably on importance, and its been shown to be a very poor predictor of future important results. But referees disagree much more rarely on soundness, and when they do they are often resolved through discussion. Papers that are sound but deemed uninteresting get published elsewhere. The importance checking is debatable, but soundness checking is crucial and reasonably objective.
tamana 11y ago
There is no peer reviewed evidence in favor of peer review?

mattkrause 11y ago

If "outsiders" actually knew how peer review worked, I don't think it would give them a great deal of confidence. Peer review literally means that a few other people (often two or three, maybe five at the outside) looked at a manuscript and failed to find anything too objectionable. Since these decisions are just based on a few people spending a few hours, you might expect this signal to be a noisy measure of quality, and it is. NIPS, an excellent machine learning/neuroscience conference, ran an experiment where 10% of the submissions were reviewed by two separate program committees. They found considerable disagreement between the two committees (~60% of the selected papers were different) More details here: http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experiment/fulltext http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experime...

tamana 11y ago
More to the point, peer review picks an limited sample of papers to accept, based on interestingness and X printed pages per month per journal. There is no peer review system that will review and accept every valid research project.