5 ms·

The publishers' peer review process doesn't work. They pull in half a dozen unpaid "volunteers" (who mostly don't have time and don't care) -- to skim a paper

by ohazi 6y ago

The publishers' peer review process doesn't work. They pull in half a dozen unpaid "volunteers" (who mostly don't have time and don't care) -- to skim a paper and then give a quick thumbs up or down. That's not real peer review, that's just holding up an arbitrary, meaningless bar. There are tons of papers that pass this bar and are either complete bullshit, or that are so poorly written that there was nothing meaningful for "peer reviewers" to actually assess. You shouldn't be able to pass peer review without a methods section that allows a third party to fully recreate your experiment, but it turns out it's easier to pass peer review by leaving out sketchy details so that there's less for a reviewer to flag as sketchy! That's so obviously broken! We spend a lot of time talking about the outrageous cost structure of the traditional publishing industry, but the other problem is that they never actually gave a shit about real review. The model that should be preferred for open access journals should also be open access. Anybody should be able to publish pre-prints, like they can today on arxiv and similar. Then the entire community can choose to accept, reject, or ignore your work. The best regarded works can then be selected for publication.

redis_mlc 6y ago
> You shouldn't be able to pass peer review without a methods section that allows a third party to fully recreate your experiment No. Peer review, anywhere, just means somebody read your paper. It doesn't mean somebody replicated your experiment, and almost never happens that way.
teorema 6y ago
I agree with Moldoveanu that the incentive structure at many open access journals is backwards (even though I empathize with the intent of the journals). But at the same time I don't think this has anything to do with this particular crack (assuming it is a crack) in peer review, which would have happened anyway at any number of journals. I agree with you that it might be better to just publish in archives or use academic organization fees or something to cover costs of some kind of peer review system. On the other hand, I've seen both sides of this coin. Assuming Moldoveanu is correct, every once in awhile this happens and it's frustrating as a reviewer. I was once in a very similar position as a reviewer at a statistics journal, where someone submitted and eventually published something that flew in the face of well-established findings and proofs, due to a simple but easily overlooked and consequential error in one of the derivations. When I reached out to the journal about a commentary or letter to the editor, I was stonewalled in a similar way and got the sense the journal was trying to save face over the error. I actually respected the author and think they made a simple mistake but still think it was worth a correction or at least discussion. At the same time, I've seen cases that look and sound just like this to an outsider, where some contingent of researchers gangs up on someone with a sense of self-righteousness to assert their authority over something, but they're actually completely one hundred percent wrong about it. A smear campaign that looks just like these published refutations of Christian's claims, that look very intelligent and rigorous (and in some sense are), that are nonetheless completely dead wrong. Someone on HN would see a blog post just like this, by the equivalent of Moldoveanu complaining about the incompetence of the journal etc, how Christian is incompetent, etc, and would just assume that because Moldoveanu is asserting some claim to rigor, they must be correct, and the person not doing so is incorrect. This case is different because of course we're talking about Bell's Theorem, so our prior about it being wrong is skewed strongly in the contrary direction. But in many cases, most cases in academics, which side is right or wrong isn't always clear. There's just one party who claims the other party is incorrect, and others just kind of adopt one side or another. Do we assume the majority is correct? I'm not really sure what my point is, other than that indignation doesn't really persuade me so much anymore. I can empathize with it, I've been there -- but really in the end all that can be done is to make your case and just make it again whenever you're asked or the topic comes up. Really the broader issue is about gish gallop and Brandolini's law which seems to be the norm now in many facets of life, and what, if anything, to do about it.
abdullahkhalids 6y ago
I can make a number of counter-arguments. 1. The volunteer nature of peer review is not a direct cause of bad peer review. In some sub-fields, peer reviews works very well at providing feedback or stopping bad papers from being published. In others, not so much. The difference is the norms of the community. 2. There are plenty of licensing bodies in other areas (medicine/plumbing/etc) where the reviewing body lets some of the chaff through with the wheat. Again, paying people can help but is not sufficient. What matters are the norms of the community - which take a lot of hard work to change/build. 3. Please note that the peer review process will always have errors (it's a noisy classification algorithm). The debate should be on the magnitude of acceptable Type I and II errors. 4. Asking papers to have 100% details of the experiment is impossible. It takes 20% effort to get to 80% details. And an additional 4x effort to get the last 20% of the details. A lot of details are often the psycho-motor skills of the grad student, and can't always be written down. I agree that experimentalists should include more details, and release their data, but 100% is too high a bar to reach. > Then the entire community can choose to accept, reject, or ignore your work. 5. The Facebook architecture fails at separating true news from fake news, and a similar architecture for science will similarly be worse than the current system. Popularity contests rarely work at reliably selecting the best.
- praestigiare 6y ago
  4 is a strawman. The comment you are replying to did not say 100% details, it said "a methods section that allows a third party to fully recreate your experiment."
  - abdullahkhalids 6y ago
    I took their meaning to be that the paper should have enough details be such that say a junior grad student could set up the same equipment and do the exact process described in the paper, and get pretty much the same results. I don't think this is possible. What is possible is the following. It's sufficient that the paper has the crucial details. And whoever tries to replicate will still have to apply some creativity, change some details, and learn some skills, and ultimately get slightly different results. In other words, enough detail to "fully recreate [any sufficiently complicated] experiment" is not possible to write down. Obviously, I am not providing any evidence for my position, besides my personal observations.
dekhn 6y ago
I published a paper that was effectively not reproducible by third parties, simply because nobody else in the world had the computing power to reproduce it. In fact, several of my papers have this problem. It's entirely unclear what to put in a methods section. Note that in at least one of the papers, the results are now used in nearly all protein design projects, so hopefully they were right.

by ohazi 6y ago

redis_mlc 6y ago

> You shouldn't be able to pass peer review without a methods section that allows a third party to fully recreate your experiment No. Peer review, anywhere, just means somebody read your paper. It doesn't mean somebody replicated your experiment, and almost never happens that way.

teorema 6y ago

I agree with Moldoveanu that the incentive structure at many open access journals is backwards (even though I empathize with the intent of the journals). But at the same time I don't think this has anything to do with this particular crack (assuming it is a crack) in peer review, which would have happened anyway at any number of journals. I agree with you that it might be better to just publish in archives or use academic organization fees or something to cover costs of some kind of peer review system. On the other hand, I've seen both sides of this coin. Assuming Moldoveanu is correct, every once in awhile this happens and it's frustrating as a reviewer. I was once in a very similar position as a reviewer at a statistics journal, where someone submitted and eventually published something that flew in the face of well-established findings and proofs, due to a simple but easily overlooked and consequential error in one of the derivations. When I reached out to the journal about a commentary or letter to the editor, I was stonewalled in a similar way and got the sense the journal was trying to save face over the error. I actually respected the author and think they made a simple mistake but still think it was worth a correction or at least discussion. At the same time, I've seen cases that look and sound just like this to an outsider, where some contingent of researchers gangs up on someone with a sense of self-righteousness to assert their authority over something, but they're actually completely one hundred percent wrong about it. A smear campaign that looks just like these published refutations of Christian's claims, that look very intelligent and rigorous (and in some sense are), that are nonetheless completely dead wrong. Someone on HN would see a blog post just like this, by the equivalent of Moldoveanu complaining about the incompetence of the journal etc, how Christian is incompetent, etc, and would just assume that because Moldoveanu is asserting some claim to rigor, they must be correct, and the person not doing so is incorrect. This case is different because of course we're talking about Bell's Theorem, so our prior about it being wrong is skewed strongly in the contrary direction. But in many cases, most cases in academics, which side is right or wrong isn't always clear. There's just one party who claims the other party is incorrect, and others just kind of adopt one side or another. Do we assume the majority is correct? I'm not really sure what my point is, other than that indignation doesn't really persuade me so much anymore. I can empathize with it, I've been there -- but really in the end all that can be done is to make your case and just make it again whenever you're asked or the topic comes up. Really the broader issue is about gish gallop and Brandolini's law which seems to be the norm now in many facets of life, and what, if anything, to do about it.

abdullahkhalids 6y ago

I can make a number of counter-arguments. 1. The volunteer nature of peer review is not a direct cause of bad peer review. In some sub-fields, peer reviews works very well at providing feedback or stopping bad papers from being published. In others, not so much. The difference is the norms of the community. 2. There are plenty of licensing bodies in other areas (medicine/plumbing/etc) where the reviewing body lets some of the chaff through with the wheat. Again, paying people can help but is not sufficient. What matters are the norms of the community - which take a lot of hard work to change/build. 3. Please note that the peer review process will always have errors (it's a noisy classification algorithm). The debate should be on the magnitude of acceptable Type I and II errors. 4. Asking papers to have 100% details of the experiment is impossible. It takes 20% effort to get to 80% details. And an additional 4x effort to get the last 20% of the details. A lot of details are often the psycho-motor skills of the grad student, and can't always be written down. I agree that experimentalists should include more details, and release their data, but 100% is too high a bar to reach. > Then the entire community can choose to accept, reject, or ignore your work. 5. The Facebook architecture fails at separating true news from fake news, and a similar architecture for science will similarly be worse than the current system. Popularity contests rarely work at reliably selecting the best.

praestigiare 6y ago
4 is a strawman. The comment you are replying to did not say 100% details, it said "a methods section that allows a third party to fully recreate your experiment."
- abdullahkhalids 6y ago
  I took their meaning to be that the paper should have enough details be such that say a junior grad student could set up the same equipment and do the exact process described in the paper, and get pretty much the same results. I don't think this is possible. What is possible is the following. It's sufficient that the paper has the crucial details. And whoever tries to replicate will still have to apply some creativity, change some details, and learn some skills, and ultimately get slightly different results. In other words, enough detail to "fully recreate [any sufficiently complicated] experiment" is not possible to write down. Obviously, I am not providing any evidence for my position, besides my personal observations.

dekhn 6y ago

I published a paper that was effectively not reproducible by third parties, simply because nobody else in the world had the computing power to reproduce it. In fact, several of my papers have this problem. It's entirely unclear what to put in a methods section. Note that in at least one of the papers, the results are now used in nearly all protein design projects, so hopefully they were right.