3 ms·

I can sympathize with the idea of a "scientific police force" looking for fraud, but the fact is that we already attempt the appearance of that through peer rev

I can sympathize with the idea of a "scientific police force" looking for fraud, but the fact is that we already attempt the appearance of that through peer review. The problem is that peer review right now only provides the appearance of oversight. Referees are not given the time, money, or resources to really dig into papers and look for issues. I have peer reviewed 4 papers this year, averaging around 3 hours per review. The onus is largely on the referees/editors/publishers to immediately find something wrong in the paper to reject it rather than on the authors to really prove their case. And I mean "immediately". One journal that I review for requires peer reviews within 2 weeks of accepting the assignment, and they will nag you if you do not have it done within a week. If the onus must be on the referees, they need to be given much more time and resources, and that might even include money, and also might include waiting for reproduction of the results before publishing. "Publish or perish" also plays a big role here, because scientists need to get papers published to prove they are productive and worthy of funding and employment. Authors will just re-submit their manuscripts to different journals until they are published rather than re-evaluating the research in any fundamental manner (is my methodology flawed? etc.). So putting the onus on referees isn't going to work in the first place, since peer reviews are not always shared.

vlovich123 3y ago
Peer review’s fundamental flaw is that science only works through replication and peer review tries to sidestep that as a cost saving measure. It doesn’t mean there’s no value - having peers review your work can be valuable. But as any professional coder can tell you, a code review is an extremely low quality signal that can only catch obvious bugs, typos, and stylistic/basic rule conformance issues. It can’t explain why you made various design decisions and whether those were any good to begin with. I think its current role as gatekeeper is probably overall more harmful than helpful as it lends legitimacy it doesn’t have a right to (ie this paper is right) but I don’t have a better suggestion other than independent replication. Maybe more separation between who designs, conducts and analyses experiments with the analysis people being rewarded if there’s no result while the design and experiment people being paid if there is to set up a feedback loop?

vlovich123 3y ago

Peer review’s fundamental flaw is that science only works through replication and peer review tries to sidestep that as a cost saving measure. It doesn’t mean there’s no value - having peers review your work can be valuable. But as any professional coder can tell you, a code review is an extremely low quality signal that can only catch obvious bugs, typos, and stylistic/basic rule conformance issues. It can’t explain why you made various design decisions and whether those were any good to begin with. I think its current role as gatekeeper is probably overall more harmful than helpful as it lends legitimacy it doesn’t have a right to (ie this paper is right) but I don’t have a better suggestion other than independent replication. Maybe more separation between who designs, conducts and analyses experiments with the analysis people being rewarded if there’s no result while the design and experiment people being paid if there is to set up a feedback loop?