5 ms·

I don't think this has to be adversary. Journals choose to set a standard for publication, which is generally peer-review. If journals instead set a standard

I don't think this has to be adversary. Journals choose to set a standard for publication, which is generally peer-review. If journals instead set a standard which was that the work must be reproducible (in other words: there must be instruction enough for an independent team to reasonably be able to follow and repeat what you did), then that sets a higher standard for science. Note that the peers in peer review are likely to mutually ignore that type of requirement - rather than face that requirement themselves.

bonoboTP 6y ago
Peer review in practice often means you got 2 or 3 grad students who skimmed the paper and found it to have the right gestalt an impressive enough results. Reviewers aren't some magical wise expert committee with infinite focus and they don't do any kind of background "auditing". They get a stack of papers to review and don't usually spend more than a few hours on each. Judging by the quality of some reviews I've seen, many grad students take even less than an hour to write a "review". I think the general public has a way overinflated idea of what peer review means in day to day practice.
jrumbut 6y ago
This would push the dial in the direction of incremental, very conservative research and discourage novel methods. What if this was in place when someone performed the first open heart surgeries? "To reproduce these results, put the patient to sleep and make you've got lots of towels..." Even interviewing techniques, specific cell lines, access to patients with a rare disease, none of these things are easily transferable but they are among the sources of the greatest new discoveries.
- yokaze 6y ago
  > This would push the dial in the direction of incremental, very conservative research and discourage novel methods. Yes, there there are obviously edge cases, where reproduction is difficult or impossible. I think, the effort needed to do so should also be honoured, not simply skipped over. I disagree though, that it will stop innovation. On the contrary, if the capabilities and knowledge are spread, it allows people to build on that. Currently, I think, we vastly favour novelty over reproducibility. We still can have "innovation", but I think, it is right to take the novelty under consideration, if it hasn't been reproduced. In my opinion, exceptionalism is used more often as not as protectionism. If results in physics produced at the LHC are held to that standard, I do not see, why we let anyone else of the hook. Unfortunately, in the current "publish-or-die" environment, there is next to no incentive to reproduce results or to make them reproducible. I do not get academic jobs for merely reproducing results, you cannot publish that stuff, or only on low impact papers. And if I do produce something new and others can easily reproduce my results, then they can simply pick up the work where I started, and I lose my edge. Even if I am more open to competition, it still may be a lot of work to make it reproducible for others, time I rather can spend on furthering my research. E.g. there is no excuse, that in computer science practically all the results can be easily reproduced, but how many are?
  - jrumbut 6y ago
    I think, far from edge cases, there are entire, large fields that would be hampered by these guidelines. There could be parts of machine learning and the social sciences where these guidelines would work (and maybe others, I have no idea and it feels like the authors don't either). Articles like this need to get a lot more specific about what fields they're talking about. I think the best idea would be to name specific publication venues or funding opportunities that should adopt these standards. It should be that granular. Otherwise, this is one way that bureaucratic overhead, nonsensical policies, and perverse incentives are created.
- learnstats2 6y ago
  > What if this was in place when someone performed the first open heart surgeries? I'm unclear if you're imagining here a scholarly article about a heart surgery, or a press release from the university hospital saying it was done. Research happens all the time without formal publication - it makes sense that there are different standards. I am proposing that there it would be beneficial for science to have journals with a modestly higher standard.
  - jrumbut 6y ago
    The idea is that the first open heart surgury was truly innovative, but how would you describe that method for replication? If someone agreed to do a peer review they're not gonna test out brand new extremely dangerous surgical techniques, but the case studies and outcome data ought to be published as soon as possible so it can be evaluated. This type of higher standard places a heavier burden on the most innovative science. Meanwhile simple surveys or small nudge interventions are easy to replicate.
    - deleted 6y ago
      [deleted]
    - learnstats2 6y ago
      Given that an open heart surgery is a surgery, and not a published statistical hypothesis, I don't find it to be within the scope of this discussion. But, if you developed a new surgical technique today, and other surgeons were unable to reproduce that technique afterward, that would surely NOT be considered an innovation in surgery.
      - jrumbut 6y ago
        I mentioned in the original comment, open heart surgery at the time that it was a novel technique. The statistical hypothesis is "this new surgical intervention is better than the current standard of care." Instructions in a single paper publishing the results of the pilot cases will not be sufficient for another surgeon to try to replicate. A replication would take perhaps years of training and a massive budget to achieve, meanwhile the field is in the dark about the development because this is all pre-publication. If the heart surgery example isn't working for you just imagine something else that takes years of training and millions of dollars to set up, plenty of those things exist in medicine and elsewhere in science. What I am getting at is that many innovations can not be readily replicated by many or any others (in the early days after it is discovered or invented) or it would be massively expensive to do so and hard to justify when no results have been published and publicly discussed. Meanwhile simple, unambitious interventions are very easy to replicate. This proposed standard increases the burden on researchers in proportion to how truly innovative their research is, thereby incentivizing simple, easy to replicate research. It's possible it would be a good standard in some fields, but would be counterproductive in others. I believe the author should consider this and be more specific about where it should be applied.

bonoboTP 6y ago

Peer review in practice often means you got 2 or 3 grad students who skimmed the paper and found it to have the right gestalt an impressive enough results. Reviewers aren't some magical wise expert committee with infinite focus and they don't do any kind of background "auditing". They get a stack of papers to review and don't usually spend more than a few hours on each. Judging by the quality of some reviews I've seen, many grad students take even less than an hour to write a "review". I think the general public has a way overinflated idea of what peer review means in day to day practice.

jrumbut 6y ago

This would push the dial in the direction of incremental, very conservative research and discourage novel methods. What if this was in place when someone performed the first open heart surgeries? "To reproduce these results, put the patient to sleep and make you've got lots of towels..." Even interviewing techniques, specific cell lines, access to patients with a rare disease, none of these things are easily transferable but they are among the sources of the greatest new discoveries.

yokaze 6y ago
> This would push the dial in the direction of incremental, very conservative research and discourage novel methods. Yes, there there are obviously edge cases, where reproduction is difficult or impossible. I think, the effort needed to do so should also be honoured, not simply skipped over. I disagree though, that it will stop innovation. On the contrary, if the capabilities and knowledge are spread, it allows people to build on that. Currently, I think, we vastly favour novelty over reproducibility. We still can have "innovation", but I think, it is right to take the novelty under consideration, if it hasn't been reproduced. In my opinion, exceptionalism is used more often as not as protectionism. If results in physics produced at the LHC are held to that standard, I do not see, why we let anyone else of the hook. Unfortunately, in the current "publish-or-die" environment, there is next to no incentive to reproduce results or to make them reproducible. I do not get academic jobs for merely reproducing results, you cannot publish that stuff, or only on low impact papers. And if I do produce something new and others can easily reproduce my results, then they can simply pick up the work where I started, and I lose my edge. Even if I am more open to competition, it still may be a lot of work to make it reproducible for others, time I rather can spend on furthering my research. E.g. there is no excuse, that in computer science practically all the results can be easily reproduced, but how many are?
- jrumbut 6y ago
  I think, far from edge cases, there are entire, large fields that would be hampered by these guidelines. There could be parts of machine learning and the social sciences where these guidelines would work (and maybe others, I have no idea and it feels like the authors don't either). Articles like this need to get a lot more specific about what fields they're talking about. I think the best idea would be to name specific publication venues or funding opportunities that should adopt these standards. It should be that granular. Otherwise, this is one way that bureaucratic overhead, nonsensical policies, and perverse incentives are created.
learnstats2 6y ago
> What if this was in place when someone performed the first open heart surgeries? I'm unclear if you're imagining here a scholarly article about a heart surgery, or a press release from the university hospital saying it was done. Research happens all the time without formal publication - it makes sense that there are different standards. I am proposing that there it would be beneficial for science to have journals with a modestly higher standard.
- jrumbut 6y ago
  The idea is that the first open heart surgury was truly innovative, but how would you describe that method for replication? If someone agreed to do a peer review they're not gonna test out brand new extremely dangerous surgical techniques, but the case studies and outcome data ought to be published as soon as possible so it can be evaluated. This type of higher standard places a heavier burden on the most innovative science. Meanwhile simple surveys or small nudge interventions are easy to replicate.
  - deleted 6y ago
    [deleted]
  - learnstats2 6y ago
    Given that an open heart surgery is a surgery, and not a published statistical hypothesis, I don't find it to be within the scope of this discussion. But, if you developed a new surgical technique today, and other surgeons were unable to reproduce that technique afterward, that would surely NOT be considered an innovation in surgery.
    - jrumbut 6y ago
      I mentioned in the original comment, open heart surgery at the time that it was a novel technique. The statistical hypothesis is "this new surgical intervention is better than the current standard of care." Instructions in a single paper publishing the results of the pilot cases will not be sufficient for another surgeon to try to replicate. A replication would take perhaps years of training and a massive budget to achieve, meanwhile the field is in the dark about the development because this is all pre-publication. If the heart surgery example isn't working for you just imagine something else that takes years of training and millions of dollars to set up, plenty of those things exist in medicine and elsewhere in science. What I am getting at is that many innovations can not be readily replicated by many or any others (in the early days after it is discovered or invented) or it would be massively expensive to do so and hard to justify when no results have been published and publicly discussed. Meanwhile simple, unambitious interventions are very easy to replicate. This proposed standard increases the burden on researchers in proportion to how truly innovative their research is, thereby incentivizing simple, easy to replicate research. It's possible it would be a good standard in some fields, but would be counterproductive in others. I believe the author should consider this and be more specific about where it should be applied.