2 ms·
Honestly the esteem of peer-review does not hold up to the reality of peer-review. It's elevated to this gold standard for quality science, but in practice, it'
by marginalia_nu 5y ago
Honestly the esteem of peer-review does not hold up to the reality of peer-review. It's elevated to this gold standard for quality science, but in practice, it's just not. Not only is some of the real landmarks in science older than peer review as we know it today (until the middle of 20th century, articles were reviewed by an editor), in several fields, 50-90% of peer-reviewed studies fail to replicate.
It's very possible to just keep submitting papers to journals until you get someone who shrugs and lets it through. Reviewers aren't paid, and while reviewing is encouraged it basically does bupkis to further your career (and academia is very cut-throat).
- hasjekyll 5y agoCompletely agree - the quality of peer-review is a lottery. Sometimes you get an excellent referee making good points, other times you get a referee solely focussed on making sure you cite their work. And, occasionally, you get a referee who doesn't even appear to read the paper and just waves it through.
- Karellen 5y ago> in several fields, 50-90% of peer-reviewed studies fail to replicate. Is that a failure of peer-review though? Or is it more of a consequence that even if you do everything right you'll still get some false positives some of the time, and negative results tend not to get published at all? (i.e. the problem that pre-registration for medical trials hopes to solve - https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science) )
- marginalia_nu 5y ago> Is that a failure of peer-review though? It raises questions around using peer-review as a mark of scientific quality. If it is, why does peer reviewed science time and time again turn out to have enormous glaring quality problems?
- YeGoblynQueenne 5y agoI think this is like the difference between homeopathy and medicine. If you're sick and go to a doctor, you have a chance that the doctor will find a way to cure you. If you're sick and go to a homeopath, there's no chance that the homeopath will find a way to cure you. Similarly, peer review has a chance to catch errors before they make it to print, while absence of peer review has no such chance. Peer review is not infallible, neither is it a mark of scientific quality, as you say. I appreciate that people tend to use it like that- you read, in articles in the lay press, that such-and-such study was "not published in a peer-reviewed journal" as if to say that we can't be sure of the quality of the study and that, conversely, if it was peer-reviewed, we could. That's not right. Peer review is not a sufficient condition to guarantee correct results. It's not even a necessary condition. Like I say in my comment above it is one mechanism of modern research practices that helps improve the quality of published papers.
- Cupertino95014 5y ago> If you're sick and go to a homeopath, there's no chance that the homeopath will find a way to cure you Wrong, sorry. Disclaimer: I'm not a homeopath and don't go to any. Most of them are, in fact, quacks. There are, unfortunately, chronic diseases which can be treated but not cured. Mainstream doctors and Big Pharma make a living off of those. If you have one of those, "well, why NOT try a homeopath?" is a perfectly rationale response. "Sinus rinsing" is something that might be termed "homeopathy." It IS medically respectable, unlike most of their "treatments" (like magnets).
- UncleMeat 5y agoI think the core problem here is the layperson perception of peer review. People often say "it is a peer-reviewed paper!" when trying to claim that a paper is definitive and powerful. But when you speak to scientists they have a much more muted understanding of peer review. And I think that is okay. The amount of effort it would take to do a very thorough analysis of a paper is large, sometimes even approaching the cost to do the research in the first place. In fact, peer review is often more about novelty and importance rather than rigor. Very well structured research will get rejected if it isn't seen as contributing to the field in a nontrivial way. A few fields (psych is the big one) are funding replication grants. That'd be the true mark of quality.