8 ms·

I think the important story here isn't that some controversy addicts/anti-vaxxers wrote a bad paper, but that the review process at a journal called Vaccines le

I think the important story here isn't that some controversy addicts/anti-vaxxers wrote a bad paper, but that the review process at a journal called Vaccines let trough a 4chan level analysis. And this during a world wide drive to vaccinate everybody no less! Sadly it's too hard to attribute the deaths via vaccine hesitancy that this paper will cause, so none of the reviewers can get charged with involuntary manslaughter.

f6v 5y ago
If you dig deep enough, many of the peer-reviewed papers are unreproducible.
- SiempreViernes 5y ago
  In some fields sure, and whenever you have small sample sizes you'll unavoidably have many contradictory results (unless everyone agrees on priors beforehand), but I don't think vaccines is a field with a replication crisis comparable to psychology.
  - username90 5y ago
    Medicine is absolutely a field full of bullshit papers that misuses methods and statistics. You'd think they would be better than psychology but not really, they are often even worse at math and statistics than the psychology papers. You'd have to go to biology papers to get reliable rigor, they do a lot on vaccines.
gizmo 5y ago
Manslaughter for publishing a bad scientific paper? Have you lost your mind? We don't want to contribute to vaccine hesitancy, but we also don't want to adopt a totalitarian worldview where any deviation from "yay vaccines!" is tantamount to murder.
- radu_floricica 5y ago
  For the reviewers? No. But if we find out people in the journal hand picked reviewers likely to pass the papers in order to see it published, well, I'd like to see at least a civil suit against them. That's not science when you load the dice, it's fraud.
  - gizmo 5y ago
    This kind of retribution mindset doesn't help. Will scientific progress increase when we get more lawyers involved? Is that what society needs? No. Science should be a process where eventually truth wins out. For this to happen scientists need to be able to publish their findings without fear of retribution. They already risk losing funding, they already have poor job security, they already need to meet a quota for papers and citations. Adding more negative incentives on top of this won't do society any good.
  - unishark 5y ago
    Bad reviewers are extremely common. I wouldn't read too much into lazy people not double-checking the stats carefully. The one highly-enthusiastic reviewer is a bit suspicious, but I'd more suspect it was a case of the author recommending a friend as a reviewer (a common practice). But it could also just be by luck they got a person who was negative towards covid vaccinations so wanted to be very supportive.
    - aj3 5y ago
      Isn't peer review usually blind?
      - unishark 5y ago
        It's very common for journals to ask for reviewer suggestions. For example I guess this is the journal in question: https://www.mdpi.com/journal/vaccines/instructions https://www.mdpi.com/journal/vaccines/instructions "Reviewer Suggestions During the submission process, please suggest three potential reviewers with the appropriate expertise to review the manuscript. The editors will not necessarily approach these referees. Please provide detailed contact information (address, homepage, phone, e-mail address). The proposed referees should neither be current collaborators of the co-authors nor have published with any of the co-authors of the manuscript within the last five years. Proposed reviewers should be from different institutions to the authors. You may identify appropriate Editorial Board members of the journal as potential reviewers. You may suggest reviewers from among the authors that you frequently cite in your paper." Obviously people are supposed to do so ethically, but cabals of friends can potentially hack the process by suggesting each other and working together.
impendia 5y ago
> a journal called Vaccines I'm an academic mathematician. Your pointing out the name of the journal is an interesting point. Pop quiz: out of the journals Aequationes mathematicae, Indagationes mathematicae, Inventiones mathematicae, Quaestiones mathematicae, three are mediocre and one is absolutely top-notch. Without googling can you guess which? MDPI is a publisher that was started recently, has apparently tried to grow very fast, and engages in questionable practices. I peer-reviewed for one of their journals once and had a very negative experience. Never. Again. They've also started journals with good sounding names. For example there was no mathematics journal with the title "Mathematics" so they started one. I've instructed my mail client that *@mdpi.com is to go straight to spam. I got invited to be on the editorial board of one of their journals, and I felt no guilt about not responding. Not too surprised to see them publish crackpottery.
- pmyteh 5y ago
  I think this is a fascinating question. There does seem to be an association between 'better' names and higher quality, and I've been wondering why that might be. Older journals have a wider range of available names so choose better? Association only in our heads between the journal quality and its name subsequent to reading the content? (I got your pop quiz completely wrong; would a junior mathematician who hadn't read or heard of any of them think that 'inventiones' was inherently a better name?) Or is a 'better' name likely, ceteribus paribus, to attract better manuscripts - starting a feedback loop that drives it up? I can't help thinking that a journal called 'Mathematics' has to have better prospects than the average questionable new journal. My own field is political communication. One of our leading journals is 'Political Communication', which could exemplify any of the above. But another is 'The International Journal of Press/Politics', which should by its name be complete shit and turns out to be excellent.
- SiempreViernes 5y ago
  Hm, Science describes it as a "reputable open-access journal" which based on your account I now must assume to be low key gloating. Certainly this debacle should lower their rating to "low repute journal".

f6v 5y ago

If you dig deep enough, many of the peer-reviewed papers are unreproducible.

SiempreViernes 5y ago
In some fields sure, and whenever you have small sample sizes you'll unavoidably have many contradictory results (unless everyone agrees on priors beforehand), but I don't think vaccines is a field with a replication crisis comparable to psychology.
- username90 5y ago
  Medicine is absolutely a field full of bullshit papers that misuses methods and statistics. You'd think they would be better than psychology but not really, they are often even worse at math and statistics than the psychology papers. You'd have to go to biology papers to get reliable rigor, they do a lot on vaccines.

gizmo 5y ago

Manslaughter for publishing a bad scientific paper? Have you lost your mind? We don't want to contribute to vaccine hesitancy, but we also don't want to adopt a totalitarian worldview where any deviation from "yay vaccines!" is tantamount to murder.

radu_floricica 5y ago
For the reviewers? No. But if we find out people in the journal hand picked reviewers likely to pass the papers in order to see it published, well, I'd like to see at least a civil suit against them. That's not science when you load the dice, it's fraud.
- gizmo 5y ago
  This kind of retribution mindset doesn't help. Will scientific progress increase when we get more lawyers involved? Is that what society needs? No. Science should be a process where eventually truth wins out. For this to happen scientists need to be able to publish their findings without fear of retribution. They already risk losing funding, they already have poor job security, they already need to meet a quota for papers and citations. Adding more negative incentives on top of this won't do society any good.
- unishark 5y ago
  Bad reviewers are extremely common. I wouldn't read too much into lazy people not double-checking the stats carefully. The one highly-enthusiastic reviewer is a bit suspicious, but I'd more suspect it was a case of the author recommending a friend as a reviewer (a common practice). But it could also just be by luck they got a person who was negative towards covid vaccinations so wanted to be very supportive.
  - aj3 5y ago
    Isn't peer review usually blind?
    - unishark 5y ago
      It's very common for journals to ask for reviewer suggestions. For example I guess this is the journal in question: https://www.mdpi.com/journal/vaccines/instructions https://www.mdpi.com/journal/vaccines/instructions "Reviewer Suggestions During the submission process, please suggest three potential reviewers with the appropriate expertise to review the manuscript. The editors will not necessarily approach these referees. Please provide detailed contact information (address, homepage, phone, e-mail address). The proposed referees should neither be current collaborators of the co-authors nor have published with any of the co-authors of the manuscript within the last five years. Proposed reviewers should be from different institutions to the authors. You may identify appropriate Editorial Board members of the journal as potential reviewers. You may suggest reviewers from among the authors that you frequently cite in your paper." Obviously people are supposed to do so ethically, but cabals of friends can potentially hack the process by suggesting each other and working together.

impendia 5y ago

> a journal called Vaccines I'm an academic mathematician. Your pointing out the name of the journal is an interesting point. Pop quiz: out of the journals Aequationes mathematicae, Indagationes mathematicae, Inventiones mathematicae, Quaestiones mathematicae, three are mediocre and one is absolutely top-notch. Without googling can you guess which? MDPI is a publisher that was started recently, has apparently tried to grow very fast, and engages in questionable practices. I peer-reviewed for one of their journals once and had a very negative experience. Never. Again. They've also started journals with good sounding names. For example there was no mathematics journal with the title "Mathematics" so they started one. I've instructed my mail client that *@mdpi.com is to go straight to spam. I got invited to be on the editorial board of one of their journals, and I felt no guilt about not responding. Not too surprised to see them publish crackpottery.

pmyteh 5y ago
I think this is a fascinating question. There does seem to be an association between 'better' names and higher quality, and I've been wondering why that might be. Older journals have a wider range of available names so choose better? Association only in our heads between the journal quality and its name subsequent to reading the content? (I got your pop quiz completely wrong; would a junior mathematician who hadn't read or heard of any of them think that 'inventiones' was inherently a better name?) Or is a 'better' name likely, ceteribus paribus, to attract better manuscripts - starting a feedback loop that drives it up? I can't help thinking that a journal called 'Mathematics' has to have better prospects than the average questionable new journal. My own field is political communication. One of our leading journals is 'Political Communication', which could exemplify any of the above. But another is 'The International Journal of Press/Politics', which should by its name be complete shit and turns out to be excellent.
SiempreViernes 5y ago
Hm, Science describes it as a "reputable open-access journal" which based on your account I now must assume to be low key gloating. Certainly this debacle should lower their rating to "low repute journal".