4 ms·

I wish there was a way for me to transform this statement. The peer review process is a joke. Preprints are every bit as good as peer-reviewed publications, for

I wish there was a way for me to transform this statement. The peer review process is a joke. Preprints are every bit as good as peer-reviewed publications, for now. This will change if pre-prints ever offer bona fide academic points, but for now, there is no difference in quality.

sterlind 5y ago
peer review or not, when reading you have to: * determine whether the numbers are intentionally fudged. * read the methods and results and check for possible systematic errors. * see if the data supports the conclusions of the authors. * make sure you understand the limits and implications of what the authors claim. as a layperson, I can't really do this for biology. so inevitably, I give things a quick once-over, come up with questions and look to the people I trust for answers or contradictory data. so it's basically an ad-hoc peer review.
rolph 5y ago
the whole purpose of a preprint is to confer with others having experience or specialty regarding the subject of thesis. Noone knows it all, or has a perfect version of the contemporary facts. peer review is quality control
Retric 5y ago
That’s like saying QA testing is a joke because it doesn’t catch everything so why not skip it. Peer review catches a lot of issues early but it’s based around the assumption papers are submitted in good faith. Compared to most things you read in say magazines they might be good enough, but it’s really not the same.
- woofie11 5y ago
  Have you seen peer review catch issues in papers you've submitted? Of the papers I've submitted, exactly one received a quality peer review. The rest were nonsense. Both accepted and rejected papers clearly weren't read. All incentives align to giving a paper a quick skim and being done with it. NIPS showed the process is indistinguishable from noise in an experiment a few years back. It's like QA testing with a six-sided die.
- kaoD 5y ago
  QA testing would be replication. Peer review is more like code reviews... but the code is hidden and you can only see the comments.
fragmede 5y ago
hydroxycholoroquine and invermectin as treatment for Covid-19, as well as a vaccine->autism link, are all well supported by preprints. They have all since been disproven, but pre-prints aren't just as good.
- bsaul 5y ago
  Slightly related : how do you explain the (obviously) fraudulent lancet paper claiming for hcq toxicity, and retracted one week later ? (but that was enough time to have hcq be banned in a few countries, including france). Since this story ( which smells a lot like corruption and lobbyist work) i must say i'm a bit confused about the true value of publication for things related to covid
  - woofie11 5y ago
    Editor of BMJ estimates about 20% of scientific publications are fraudulent. https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-health-research-is-fraudulent-until-proved-otherwise/ https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-hea... Fewer than half replicate. I would estimate a lot of that comes from softer research malpractice. Being confused about the value is a fair reaction. It is high; without scientific research, though, we wouldn't have vaccines or monoclonal antibodies. On the other hand, it is lower than assumed, and trusting individual papers and peer review processes is a mistake. How much value would Facebook have if it were your only source of information? A lot, actually. Imperfect information is valuable too.
    - bsaul 5y ago
      Yet this article was published in the middle of the covid crisis on of the hot topics. I can't imagine the journal's reviewers being careless.
      - woofie11 5y ago
        Have you published much? I haven't seen many careful reviewers. It's an anonymous process, and no reason to put in any effort at all. On the procrastination list, peer reviews are the very last thing scientists do.
janto 5y ago
I like to say that peer review is what happens after a paper is published.
- rolph 5y ago
  no peer review is what happens after a manuscript is submitted, after a preprint is circulated, after a paper is submitted to a journal and after publication. it is an ongoing continuum of process
  - janto 5y ago
    meh, if you're going to publish something then I'd rather just have people get their papers hosted on arxiv asap and then get feedback from everyone. I have not seen significant value added by journals and the review process, in my field.

sterlind 5y ago

peer review or not, when reading you have to: * determine whether the numbers are intentionally fudged. * read the methods and results and check for possible systematic errors. * see if the data supports the conclusions of the authors. * make sure you understand the limits and implications of what the authors claim. as a layperson, I can't really do this for biology. so inevitably, I give things a quick once-over, come up with questions and look to the people I trust for answers or contradictory data. so it's basically an ad-hoc peer review.

rolph 5y ago

the whole purpose of a preprint is to confer with others having experience or specialty regarding the subject of thesis. Noone knows it all, or has a perfect version of the contemporary facts. peer review is quality control

Retric 5y ago

That’s like saying QA testing is a joke because it doesn’t catch everything so why not skip it. Peer review catches a lot of issues early but it’s based around the assumption papers are submitted in good faith. Compared to most things you read in say magazines they might be good enough, but it’s really not the same.

woofie11 5y ago
Have you seen peer review catch issues in papers you've submitted? Of the papers I've submitted, exactly one received a quality peer review. The rest were nonsense. Both accepted and rejected papers clearly weren't read. All incentives align to giving a paper a quick skim and being done with it. NIPS showed the process is indistinguishable from noise in an experiment a few years back. It's like QA testing with a six-sided die.
kaoD 5y ago
QA testing would be replication. Peer review is more like code reviews... but the code is hidden and you can only see the comments.

fragmede 5y ago

hydroxycholoroquine and invermectin as treatment for Covid-19, as well as a vaccine->autism link, are all well supported by preprints. They have all since been disproven, but pre-prints aren't just as good.

bsaul 5y ago
Slightly related : how do you explain the (obviously) fraudulent lancet paper claiming for hcq toxicity, and retracted one week later ? (but that was enough time to have hcq be banned in a few countries, including france). Since this story ( which smells a lot like corruption and lobbyist work) i must say i'm a bit confused about the true value of publication for things related to covid
- woofie11 5y ago
  Editor of BMJ estimates about 20% of scientific publications are fraudulent. https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-health-research-is-fraudulent-until-proved-otherwise/ https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-hea... Fewer than half replicate. I would estimate a lot of that comes from softer research malpractice. Being confused about the value is a fair reaction. It is high; without scientific research, though, we wouldn't have vaccines or monoclonal antibodies. On the other hand, it is lower than assumed, and trusting individual papers and peer review processes is a mistake. How much value would Facebook have if it were your only source of information? A lot, actually. Imperfect information is valuable too.
  - bsaul 5y ago
    Yet this article was published in the middle of the covid crisis on of the hot topics. I can't imagine the journal's reviewers being careless.
    - woofie11 5y ago
      Have you published much? I haven't seen many careful reviewers. It's an anonymous process, and no reason to put in any effort at all. On the procrastination list, peer reviews are the very last thing scientists do.

janto 5y ago

I like to say that peer review is what happens after a paper is published.

rolph 5y ago
no peer review is what happens after a manuscript is submitted, after a preprint is circulated, after a paper is submitted to a journal and after publication. it is an ongoing continuum of process
- janto 5y ago
  meh, if you're going to publish something then I'd rather just have people get their papers hosted on arxiv asap and then get feedback from everyone. I have not seen significant value added by journals and the review process, in my field.