4 ms·

"Science is facing a 'reproducibility crisis' where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, rese

"Science is facing a 'reproducibility crisis' where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests." https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778 As such, this would seem to be a common occurrence and Science would have to decide whether to devote equal attention to every failed reproduction.

hartator 7y ago
The thing is this study is featured a lot everywhere.
- ajross 7y ago
  Maybe so, but Science featured it only once. I think it's reasonable to demand that the journal regularly note corrections to stuff that it's published. It's absolutely insane to demand (via a giant article on Slate, no less) that they publish every refutation of anything they ever ran. The only reason this is generating clicks is the politics angle. This is boring science. The original study was probably bad (as, let's face it, most psychology studies are), but they ran it anyway (because clicks). Now the other side gets its clicks on the "censorship" angle. The science? Still boring.
  - CDRdude 7y ago
    > I think it's reasonable to demand that the journal regularly note corrections to stuff that it's published. It's absolutely insane to demand (via a giant article on Slate, no less) that they publish every refutation of anything they ever ran. I don’t understand the distinction you are making here. Why would it be insane to publish refutations? In this case, they refused to even review the refutation.
    - ajross 7y ago
      > Why would it be insane to publish refutations? It's not at all insane to publish refutations. A really good refutation of a long-accepted result is great science, and this kind of thing makes news and runs in major journals all the time. It's insane to demand that Science publish a refutation of anything they happened to run in the past just because your results (or, let's be honest here, your personal politics) disagree. Most refutations, too, are bad science!
      - mentat 7y ago
        They expected to replicate the findings. They fail to do so. Just because this contradicts your personal politics isn't a reason to ignore or suppress the information. They promoted it, they need to set the record straight.
        ajross 7y ago
        "The information" is literally plastered all over a giant article on Slate and we're yelling about it as we speak. That's not very effective suppression. The demand seems to be that "If Science ran an article they're obligated to publish anyone's refutation of that article". And that's insane. It only seems sane to you because your political outrage glands demand justice for someone using the previous study as spin, but that's not Science's problem, and it shouldn't be. Go run the outrage pieces on Slate, that's what it's for.
        kadoban 7y ago
        I would say that if Science ran an article they are morally obligated to seriously consider a refutation of that article, and if no problems can be found, publish it. I did not get the sense that the refutation was seriously considered, and my impression is that the original article has not been retracted in any way, correct? So Science's public stance could be seen as the original article is still the best truth they've published on the topic, which feels wrong. Wiggle room may be appropriate if the article is already retracted or has already been refuted.
Veedrac 7y ago
> this would seem to be a common occurrence and Science would have to decide whether to devote equal attention to every failed reproduction I think the answer to this is a rather blatant “of course!” If Science has the time to support false studies that retrograde the scientific cause, it certainly has the time to publish their corrections and refutations. The scientific process is not about how much stuff you can publish, it's about how much true information you can convey. A journal half the size of Science but twice as trustworthy is at least as valuable, if not vastly more.
- modriano 7y ago
  > I think the answer to this is a rather blatant “of course!” It's not as blatant as you think. It takes experienced scientists to review submitted papers, and their time is an expensive resource. Is it interesting that a research team with a sloppy adherence to a methodology failed to replicate a study? Not really. Should Science give resources and a platform to people who are just careless researchers? Obviously not. It's easy to fail to replicate an experiment; just do a bad job.
  - Sacho 7y ago
    There's an implication in your argument that the science posted in the journal is of a good enough quality to be published - otherwise why wouldn't you apply the same standard you are applying here? Doing a bad job isn't limited to just reproduction. Considering the reproducibility crisis, is it your position that people are worse at reproducing research than they are at conducting it? An alternative interpretation would be that people have more incentive to publish hastily-made "failed to reproduce" papers than they have "findings" papers, but I don't think that's the case either, especially considering the reply from Science to this case.
    - brighter2morrow 7y ago
      I think the rationale for Science to say they already fulfilled their obligations the first time around is that they thoroughly scrutinized the methodology used the first time around. This is asking them to now scrutinize two methodologies used and for less payoff than the first time.
pdonis 7y ago
> Science would have to decide whether to devote equal attention to every failed reproduction. Not to every failed reproduction. Only to failures to reproduce results that Science had previously published. And if that means they end up publishing a lot of failures to reproduce, then perhaps they need to re-think their criteria for publishing the original results. After all, why do you think this "reproducibility crisis" exists in the first place?

hartator 7y ago

The thing is this study is featured a lot everywhere.

ajross 7y ago
Maybe so, but Science featured it only once. I think it's reasonable to demand that the journal regularly note corrections to stuff that it's published. It's absolutely insane to demand (via a giant article on Slate, no less) that they publish every refutation of anything they ever ran. The only reason this is generating clicks is the politics angle. This is boring science. The original study was probably bad (as, let's face it, most psychology studies are), but they ran it anyway (because clicks). Now the other side gets its clicks on the "censorship" angle. The science? Still boring.
- CDRdude 7y ago
  > I think it's reasonable to demand that the journal regularly note corrections to stuff that it's published. It's absolutely insane to demand (via a giant article on Slate, no less) that they publish every refutation of anything they ever ran. I don’t understand the distinction you are making here. Why would it be insane to publish refutations? In this case, they refused to even review the refutation.
  - ajross 7y ago
    > Why would it be insane to publish refutations? It's not at all insane to publish refutations. A really good refutation of a long-accepted result is great science, and this kind of thing makes news and runs in major journals all the time. It's insane to demand that Science publish a refutation of anything they happened to run in the past just because your results (or, let's be honest here, your personal politics) disagree. Most refutations, too, are bad science!
    - mentat 7y ago
      They expected to replicate the findings. They fail to do so. Just because this contradicts your personal politics isn't a reason to ignore or suppress the information. They promoted it, they need to set the record straight.
      - ajross 7y ago
        "The information" is literally plastered all over a giant article on Slate and we're yelling about it as we speak. That's not very effective suppression. The demand seems to be that "If Science ran an article they're obligated to publish anyone's refutation of that article". And that's insane. It only seems sane to you because your political outrage glands demand justice for someone using the previous study as spin, but that's not Science's problem, and it shouldn't be. Go run the outrage pieces on Slate, that's what it's for.
        kadoban 7y ago
        I would say that if Science ran an article they are morally obligated to seriously consider a refutation of that article, and if no problems can be found, publish it. I did not get the sense that the refutation was seriously considered, and my impression is that the original article has not been retracted in any way, correct? So Science's public stance could be seen as the original article is still the best truth they've published on the topic, which feels wrong. Wiggle room may be appropriate if the article is already retracted or has already been refuted.

Veedrac 7y ago

> this would seem to be a common occurrence and Science would have to decide whether to devote equal attention to every failed reproduction I think the answer to this is a rather blatant “of course!” If Science has the time to support false studies that retrograde the scientific cause, it certainly has the time to publish their corrections and refutations. The scientific process is not about how much stuff you can publish, it's about how much true information you can convey. A journal half the size of Science but twice as trustworthy is at least as valuable, if not vastly more.

modriano 7y ago
> I think the answer to this is a rather blatant “of course!” It's not as blatant as you think. It takes experienced scientists to review submitted papers, and their time is an expensive resource. Is it interesting that a research team with a sloppy adherence to a methodology failed to replicate a study? Not really. Should Science give resources and a platform to people who are just careless researchers? Obviously not. It's easy to fail to replicate an experiment; just do a bad job.
- Sacho 7y ago
  There's an implication in your argument that the science posted in the journal is of a good enough quality to be published - otherwise why wouldn't you apply the same standard you are applying here? Doing a bad job isn't limited to just reproduction. Considering the reproducibility crisis, is it your position that people are worse at reproducing research than they are at conducting it? An alternative interpretation would be that people have more incentive to publish hastily-made "failed to reproduce" papers than they have "findings" papers, but I don't think that's the case either, especially considering the reply from Science to this case.
  - brighter2morrow 7y ago
    I think the rationale for Science to say they already fulfilled their obligations the first time around is that they thoroughly scrutinized the methodology used the first time around. This is asking them to now scrutinize two methodologies used and for less payoff than the first time.

pdonis 7y ago

> Science would have to decide whether to devote equal attention to every failed reproduction. Not to every failed reproduction. Only to failures to reproduce results that Science had previously published. And if that means they end up publishing a lot of failures to reproduce, then perhaps they need to re-think their criteria for publishing the original results. After all, why do you think this "reproducibility crisis" exists in the first place?