4 ms·
Why do we need the process of peer review? Peer review is not robust against even low levels of collusion (http://arxiv.org/abs/1008.4324v1 http://arxiv.org/ab
by gwern 15y ago
Why do we need the process of peer review?
Peer review is not robust against even low levels of collusion (http://arxiv.org/abs/1008.4324v1 http://arxiv.org/abs/1008.4324v1). Scientists who win the Nobel Prize find their other work suddenly being heavily cited (http://www.nature.com/news/2011/110506/full/news.2011.270.html http://www.nature.com/news/2011/110506/full/news.2011.270.ht...), suggesting either that the community either badly failed in recognizing the work's true value or that they are now sucking up & attempting to look better by the halo effect. (A mathematician once told me that often, to boost a paper's acceptance chance, they would add citations to papers by the journal's editors - a practice that will surprise none familiar with Goodhart's law and the use of citations in tenure & grants.)
Physicist Michael Nielsen points out (http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review/ http://michaelnielsen.org/blog/three-myths-about-scientific-...) that peer review is historically rare (just one of Einstein's 300 papers was peer reviewed! the famous _Nature_ did not institute peer review until 1967), has been poorly studied (http://jama.ama-assn.org/cgi/content/abstract/287/21/2784 http://jama.ama-assn.org/cgi/content/abstract/287/21/2784) & not shown to be effective, is nationally biased (http://jama.ama-assn.org/cgi/content/full/295/14/1675 http://jama.ama-assn.org/cgi/content/full/295/14/1675), erroneously rejects many historic discoveries (one study lists "34 Nobel Laureates whose awarded work was rejected by peer review" (http://www.canonicalscience.org/publications/canonicalsciencereports/20082.html http://www.canonicalscience.org/publications/canonicalscienc...); Horribin 1990 (http://jama.ama-assn.org/content/263/10/1438.abstract http://jama.ama-assn.org/content/263/10/1438.abstract) lists others like the discovery of quarks), and catches only a small fraction (http://jama.ama-assn.org/cgi/content/abstract/280/3/237 http://jama.ama-assn.org/cgi/content/abstract/280/3/237) of errors. And fraud, like the one we just saw in psychology? Forget about it (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0005738 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna...);
> "A pooled weighted average of 1.97% (N = 7, 95%CI: 0.86–4.45) of scientists admitted to have fabricated, falsified or modified data or results at least once –a serious form of misconduct by any standard– and up to 33.7% admitted other questionable research practices. In surveys asking about the behaviour of colleagues, admission rates were 14.12% (N = 12, 95% CI: 9.91–19.72) for falsification, and up to 72% for other questionable research practices....When these factors were controlled for, misconduct was reported more frequently by medical/pharmacological researchers than others."
No, peer review is not the secret sauce of science. Replication is more like it.
- snowwrestler 15y agoYour conclusion seems to be: why do we need peer review if the results are this bad? But an alternative perspective might be: think of how much worse things would be _without_ peer review.
- gwern 15y ago> But an alternative perspective might be: think of how much worse things would be _without_ peer review. Hey, I have this elephant repellant for sale. I know it works awesome because I haven't ever seen any elephants around here. (I also cited historical data about the absence of peer review working, you know, pretty well.)
- mironathetin 15y agoIn order to check, if what you write makes sense, we have to read at least all texts you link to, probably also links in these text. This will take perhaps one day, if done accurately. Do you prefer that all readers spend the time, or wouldn't it be an advantage, if a few knowledgeable experts would do the job and signal to everybody: what he writes makes sense? Sure, peer review is not flawless. But flawed procedures can either be replaced or improved. Improvement keeps what is good and changes the rest. Complete replacement may come with new problems. Don't think that you can escape vanity where people are involved. Not even in science. There are also always people who make use of system errors. As long as that is only a few percent, we do pretty well. And yes, I think snowwrestler also has a very good point.
- gwern 15y ago> Do you prefer that all readers spend the time, or wouldn't it be an advantage, if a few knowledgeable experts would do the job and signal to everybody: what he writes makes sense? Division of labor does not imply peer review. Peer review is merely an ad hoc, unproven, flawed way of implementing division of labor.