3 ms·
Researchers as a whole need to do more checking. While I agree that errors like the one identified in the link are rare, they are not so rare that one shouldn't
by btrettel 4y ago
Researchers as a whole need to do more checking. While I agree that errors like the one identified in the link are rare, they are not so rare that one shouldn't spend the time looking for them or assume that everything was done properly.
I've speculated before that peer review gives researchers false confidence in published results [0]. A lot of academics seems to believe that peer review is much better at finding errors than it actually is. (Here's one example of a conversation I had on HN that unfortunately was not productive: [1].) To be clear, I think getting through peer review is evidence that a paper is good, albeit weak evidence. I would give the fact that a paper is peer reviewed little weight compared against my own evaluation of the paper.
[0] https://news.ycombinator.com/item?id=22290907 https://news.ycombinator.com/item?id=22290907
[1] https://news.ycombinator.com/item?id=31485701 https://news.ycombinator.com/item?id=31485701
- NegativeLatency 4y agoSomewhere near 100% of my shipped bugs have been peer reviewed so that makes a lot of sense to me.
- michaelmior 4y ago> To be clear, I think getting through peer review is evidence that a paper is good I think this depends on how you define good. I'm sure there's some variation across fields, but peer review generally seeks to establish that what is presented in the paper is plausible, logically consistent, well-presented, meaningful, and novel. That list is non-exhaustive, but correct is very hard to establish in a peer review process. In my experience, it would be rare for a reviewer to repeat calculations in a paper unless something seems fairly obviously off. As a computer scientist, it would be even more rare for a peer reviewer to examine the code written for a paper (if it is available) to check for bugs. Point being, there are a lot of reasons a paper that appears good may be completely incorrect. Although this is typically for reasons that I as a casual reader would be even less likely to distinguish than a reviewer who is particularly knowledgeable about that particular field.
- magicalhippo 4y agoI've got a close relative who reviews papers all the time in their field (not CS). Based on that my take is that if a paper passes peer review it is a good indicator there's nothing egregiously wrong with the stuff that's written.
- 11101010001100 4y agoI just completed a paper review as a reviewer. After I think 4 rounds, the author finally ran the calculation I had asked for in the initial review and admitted I was right. We got there in the end, but I had to sit on my hands.
- pcrh 4y agoPeer review can help improve a paper (and it has improved some of mine); however, contrary to some popular notions, it doesn't lend "truth" to a paper. Peer reviewers are not monitoring how experiments were conducted, they only have access to a data set that is by necessity already highly selected from all the work that went into producing the final manuscript. The authors thus bear ultimate responsibility. When considering published work close to mine, I use my own judgement of the work, regardless of peer review or which journal it is published in (for example it may be in a PhD thesis). For work where I am not so familiar with the methodologies, I prefer to wait for independent verification/replication (direct or indirect) from a different research group, which ideally used different methods.
- throwawaymaths 4y agoWell, to be fair, there is the journal "organic syntheses" https://en.m.wikipedia.org/wiki/Organic_Syntheses https://en.m.wikipedia.org/wiki/Organic_Syntheses