3 ms·

One factor that might increase reproducibility is to enhance negative reputational effects of publishing an un-reproducible study. That is, it should be so rare

One factor that might increase reproducibility is to enhance negative reputational effects of publishing an un-reproducible study. That is, it should be so rare, so outré, to publish something unreproducible, your professional reputation is severely damaged, and if you do it again, you will leave academia. It sounds harsh, but severe self-regulation is what prevents the catastrophic failure of any institution that is really only answerable to itself. Professors aren't doing any favors by rubberstamping weak papers, or giving weak students PhD's (or BS's for that matter). EDIT: down-votes and no replies, as expected. Hey academics, sometimes to save the patient you have to amputate the limb. I care about science and want it to survive, so you guys continuing to be nice to each other even when you publish BS isn't serving that purpose.

flukus 7y ago
The problem is not being reproducible is not necessarily a failure and does not necessarily indicate a bad actor, it may be bunk but it may show a worthwhile direction for others to investigate further. The problems come when an non-reproducible study gains some authority, citations in other studies or published in the media without proper vetting. It's probably worth punishing people that consistently publish UN-reproducible work, but it shouldn't a rare career destroying event like you are suggesting.
- javajosh 7y ago
  >not being reproducible is not necessarily a failure and does not necessarily indicate a bad actor I disagree with the first part and agree with the second. I would argue that failure to reproduce is perhaps the worst kind of scientific failure, because the activity cannot be called science anymore. And no, I don't mean anything personal when I say 'failure', you don't have to be malicious to do bad science -- after all 'doing bad science' is the default human condition. An analogy with classical music: A musician that fails to reproduce the score with his violin consistently, will lose his job quickly. It doesn't mean he's a bad person, but it does mean he's not good enough for the orchestra. The metaphor for the "reproducibility crisis" in an orchestra is what happens if they let bad players stay, the orchestra sounds worse and worse, and finally the audience stops coming. In the apocalyptic scenario, cultural forces cause ALL orchestras to stop honestly evaluating the skill of their players, and all orchestras simultaneously lose their ability to accurately play any but the simplest music. Standards are painful for those that can't meet them, but they make the world better, overall.
  - inimino 7y ago
    You can judge a musician in a 20-minute audition, but you can't tell if research will reproduce without doing the work. However, there are papers that are so vague and poorly written that they cannot possibly be reproducible, and this is becoming the norm in some fields, so I think you're right that standards would help there. Also, sometimes strange things just happen randomly (like the CERN faster-than-light neutrinos) and there's no shame in publishing results that aren't correct as long as you're honest about it. Some early-stage, low-power studies are going to randomly show results that don't reproduce, and that's also fine. Suppressing these results would hurt progress just as much as making the opposite error.
  - Fomite 7y ago
    'I would argue that failure to reproduce is perhaps the worst kind of scientific failure, because the activity cannot be called science anymore.' This is only true if the results of research are deterministic. In many fields, if a study is estimating an effect that exists in the world as a distribution, a failure to reproduce might not be indicative of...anything.
  - scarejunba 7y ago
    No, that's unfair. If I pre-register my hypothesis, run good methods, and conclude at the p < 0.01, I did things well. I should expect to get some 1% of my studies to fail to reproduce. That means that I routinely do not fail to reproduce but failing a few repros over my career is normal. The real problem is the other guy who didn't pre-register the same hypothesis that he tested. So we don't know that I got it by chance until the repro failed.
itcrowd 7y ago
There is definitely an argument to be made for increased negative (reputational) effects. However, many unreproducable results are only discovered years, if not decades, after initial publication. This makes the discovery less impactfull for the original researchers. In that timespan, many have left academia; others have more recent "shiny" publications to point to instead. It's sad, but in addition to the publish-or-perish, there's also a publish-and-forget mentality in (some) academia, in which your project is finished and you move on to the next "hot" topic. Therefore, my concern is with the efficacy of such measures. What are your thoughts on how to implement such a reputational effect? Public shaming? Should we register fraudulentprofessor.com? > Professors aren't doing any favors by rubberstamping weak papers, or giving weak students PhD's Agreed. However, I think this is a sort of tragedy of the commons, in which "good" behavior of some professors will not stop the onslaught of "bad" behavior of others. Certainly the good ones will be outnumbered in terms of graduates and since university performance reviews rely mostly on sheer numbers (students, papers..), there is an argument to be made that universities self-select for the rubber-stamp professor over the honest ones. I feel a fair solution is to re-imagine the universities' performance review system, although I have a hard time coming up with concrete and good measures of how to select the good from the bad professors.
- javajosh 7y ago
  >What are your thoughts on how to implement such a reputational effect? I think it should be similar to how journalists are treated after they are involved in a major scandal (usually reporting something speculative as fact). Personally, I don't think there's any excuse for unreproducible results. They are worse than silence, they are lies that spread other lies.
  - itcrowd 7y ago
    (To continue your journalist analogy) So what happens to "the journalist" when something speculative is reported in a newspaper piece that is written by 7 people five years ago? In addition, five of the seven writers are now working different jobs in the private sector, no longer writing in any newspaper.
anjc 7y ago
Non-reproducibility doesn't imply shoddy work so it isn't fair to damage people's reputations for this reason. A paper may be non-reproducible because it isn't descriptive enough, because their data is not published yet, because a software library is deprecated, because a URL has changed etc, all of which are only discovered when somebody goes to reproduce it. Researchers can strive to make their work reproducible and still unwittingly fail. If Github goes bust in 20 years you'll find a significant chunk of current work is non-reproducible despite author's best efforts.
TrackerFF 7y ago
I think one problem with this approach is that maybe less scientists will take on fields / topics where the data is inherently elusive or just very hard to get. Could be rare natural phenomena, highly sensitive data, or whatnot. Maybe the collection of data is so prohibitively expensive or specialized, that other researchers are not able to recreate the process accurately enough, which in turn leads to irreproducibility.

flukus 7y ago

The problem is not being reproducible is not necessarily a failure and does not necessarily indicate a bad actor, it may be bunk but it may show a worthwhile direction for others to investigate further. The problems come when an non-reproducible study gains some authority, citations in other studies or published in the media without proper vetting. It's probably worth punishing people that consistently publish UN-reproducible work, but it shouldn't a rare career destroying event like you are suggesting.

javajosh 7y ago
>not being reproducible is not necessarily a failure and does not necessarily indicate a bad actor I disagree with the first part and agree with the second. I would argue that failure to reproduce is perhaps the worst kind of scientific failure, because the activity cannot be called science anymore. And no, I don't mean anything personal when I say 'failure', you don't have to be malicious to do bad science -- after all 'doing bad science' is the default human condition. An analogy with classical music: A musician that fails to reproduce the score with his violin consistently, will lose his job quickly. It doesn't mean he's a bad person, but it does mean he's not good enough for the orchestra. The metaphor for the "reproducibility crisis" in an orchestra is what happens if they let bad players stay, the orchestra sounds worse and worse, and finally the audience stops coming. In the apocalyptic scenario, cultural forces cause ALL orchestras to stop honestly evaluating the skill of their players, and all orchestras simultaneously lose their ability to accurately play any but the simplest music. Standards are painful for those that can't meet them, but they make the world better, overall.
- inimino 7y ago
  You can judge a musician in a 20-minute audition, but you can't tell if research will reproduce without doing the work. However, there are papers that are so vague and poorly written that they cannot possibly be reproducible, and this is becoming the norm in some fields, so I think you're right that standards would help there. Also, sometimes strange things just happen randomly (like the CERN faster-than-light neutrinos) and there's no shame in publishing results that aren't correct as long as you're honest about it. Some early-stage, low-power studies are going to randomly show results that don't reproduce, and that's also fine. Suppressing these results would hurt progress just as much as making the opposite error.
- Fomite 7y ago
  'I would argue that failure to reproduce is perhaps the worst kind of scientific failure, because the activity cannot be called science anymore.' This is only true if the results of research are deterministic. In many fields, if a study is estimating an effect that exists in the world as a distribution, a failure to reproduce might not be indicative of...anything.
- scarejunba 7y ago
  No, that's unfair. If I pre-register my hypothesis, run good methods, and conclude at the p < 0.01, I did things well. I should expect to get some 1% of my studies to fail to reproduce. That means that I routinely do not fail to reproduce but failing a few repros over my career is normal. The real problem is the other guy who didn't pre-register the same hypothesis that he tested. So we don't know that I got it by chance until the repro failed.

itcrowd 7y ago

There is definitely an argument to be made for increased negative (reputational) effects. However, many unreproducable results are only discovered years, if not decades, after initial publication. This makes the discovery less impactfull for the original researchers. In that timespan, many have left academia; others have more recent "shiny" publications to point to instead. It's sad, but in addition to the publish-or-perish, there's also a publish-and-forget mentality in (some) academia, in which your project is finished and you move on to the next "hot" topic. Therefore, my concern is with the efficacy of such measures. What are your thoughts on how to implement such a reputational effect? Public shaming? Should we register fraudulentprofessor.com? > Professors aren't doing any favors by rubberstamping weak papers, or giving weak students PhD's Agreed. However, I think this is a sort of tragedy of the commons, in which "good" behavior of some professors will not stop the onslaught of "bad" behavior of others. Certainly the good ones will be outnumbered in terms of graduates and since university performance reviews rely mostly on sheer numbers (students, papers..), there is an argument to be made that universities self-select for the rubber-stamp professor over the honest ones. I feel a fair solution is to re-imagine the universities' performance review system, although I have a hard time coming up with concrete and good measures of how to select the good from the bad professors.

javajosh 7y ago
>What are your thoughts on how to implement such a reputational effect? I think it should be similar to how journalists are treated after they are involved in a major scandal (usually reporting something speculative as fact). Personally, I don't think there's any excuse for unreproducible results. They are worse than silence, they are lies that spread other lies.
- itcrowd 7y ago
  (To continue your journalist analogy) So what happens to "the journalist" when something speculative is reported in a newspaper piece that is written by 7 people five years ago? In addition, five of the seven writers are now working different jobs in the private sector, no longer writing in any newspaper.

anjc 7y ago

Non-reproducibility doesn't imply shoddy work so it isn't fair to damage people's reputations for this reason. A paper may be non-reproducible because it isn't descriptive enough, because their data is not published yet, because a software library is deprecated, because a URL has changed etc, all of which are only discovered when somebody goes to reproduce it. Researchers can strive to make their work reproducible and still unwittingly fail. If Github goes bust in 20 years you'll find a significant chunk of current work is non-reproducible despite author's best efforts.

TrackerFF 7y ago

I think one problem with this approach is that maybe less scientists will take on fields / topics where the data is inherently elusive or just very hard to get. Could be rare natural phenomena, highly sensitive data, or whatnot. Maybe the collection of data is so prohibitively expensive or specialized, that other researchers are not able to recreate the process accurately enough, which in turn leads to irreproducibility.