4 ms·

You're focusing on exactly one aspect of peer review, determining whether the statistical methods used were valid, which I agree can be done by pretty much anyo

You're focusing on exactly one aspect of peer review, determining whether the statistical methods used were valid, which I agree can be done by pretty much anyone with a background in statistics (given that these days it's pretty easy to get in touch with the authors to ask for clarification). But peer review is about more than just determining whether the paper is internally coherent (or directly lifts text from other papers by the same author, which can be determined by a machine). I shouldn't have to say this, but a former physics Ph.D. is emphatically not qualified to perform peer review on papers on cell biology solely because physics and cell biology are both science. Papers don't exist in a vacuum, and by engaging in practices like selective citation (and others I mentioned in the previous post), it's possible to convince intelligent and reasonable people of a point that is completely wrong. In fact, it's so easy it even happens somewhat routinely within narrow subspecialties, where there may only be a handful of experts on a phenomenon worldwide. People who aren't up to date with the field already usually don't have time to spend weeks, months, or years trudging through the literature to gain enough background knowledge to make those kinds of decisions.

apathy 10y ago
So, peer review doesn't scale, and some alternative is needed. Sounds about right. How about we crowdsource the task? Since all the qualified specialists are busy writing grants against a 15% payline, we can farm it out to people without those pressures. Oh... wait...
wjnc 10y ago
You're right (and we probably agree). I did focus on the number 1 problem in social science / pop psychology today -- sloppy statistics. But you do make a convincing point. The 'stink' that is uncovered nowadays is mostly in those sciences where the layman / statistically schooled can make a difference. We hear a lot less about the replication crisis in those sciences where one indeed needs to have a lot of background knowledge before being able to do peer review. That's a problem. There is no reason to assume that the same problems are not paramount in cell biology and other fields. We actually have enough reason to believe that many trials in medicine are flawed. Given that researchers in all fields face the same incentives (publish or perish) we are probably looking at at least 20 years of false, peer-reviewed and published results in all fields. That's a huge problem for mankind.

apathy 10y ago

So, peer review doesn't scale, and some alternative is needed. Sounds about right. How about we crowdsource the task? Since all the qualified specialists are busy writing grants against a 15% payline, we can farm it out to people without those pressures. Oh... wait...

wjnc 10y ago

You're right (and we probably agree). I did focus on the number 1 problem in social science / pop psychology today -- sloppy statistics. But you do make a convincing point. The 'stink' that is uncovered nowadays is mostly in those sciences where the layman / statistically schooled can make a difference. We hear a lot less about the replication crisis in those sciences where one indeed needs to have a lot of background knowledge before being able to do peer review. That's a problem. There is no reason to assume that the same problems are not paramount in cell biology and other fields. We actually have enough reason to believe that many trials in medicine are flawed. Given that researchers in all fields face the same incentives (publish or perish) we are probably looking at at least 20 years of false, peer-reviewed and published results in all fields. That's a huge problem for mankind.