5 ms·

Personally, I am not a fan of an adversary approach. I think, reproducing (or failing to reproduce) scientific results simply should be given more value/money

by yokaze 6y ago

Personally, I am not a fan of an adversary approach. I think, reproducing (or failing to reproduce) scientific results simply should be given more value/money without incentives for disproving the result. The ability to reproduce a result drives also research as currently often the authors are not motivated to make it easy. And you don't get payed for simply redoing what was already published.

learnstats2 6y ago
I don't think this has to be adversary. Journals choose to set a standard for publication, which is generally peer-review. If journals instead set a standard which was that the work must be reproducible (in other words: there must be instruction enough for an independent team to reasonably be able to follow and repeat what you did), then that sets a higher standard for science. Note that the peers in peer review are likely to mutually ignore that type of requirement - rather than face that requirement themselves.
- bonoboTP 6y ago
  Peer review in practice often means you got 2 or 3 grad students who skimmed the paper and found it to have the right gestalt an impressive enough results. Reviewers aren't some magical wise expert committee with infinite focus and they don't do any kind of background "auditing". They get a stack of papers to review and don't usually spend more than a few hours on each. Judging by the quality of some reviews I've seen, many grad students take even less than an hour to write a "review". I think the general public has a way overinflated idea of what peer review means in day to day practice.
- jrumbut 6y ago
  This would push the dial in the direction of incremental, very conservative research and discourage novel methods. What if this was in place when someone performed the first open heart surgeries? "To reproduce these results, put the patient to sleep and make you've got lots of towels..." Even interviewing techniques, specific cell lines, access to patients with a rare disease, none of these things are easily transferable but they are among the sources of the greatest new discoveries.
  - yokaze 6y ago
    > This would push the dial in the direction of incremental, very conservative research and discourage novel methods. Yes, there there are obviously edge cases, where reproduction is difficult or impossible. I think, the effort needed to do so should also be honoured, not simply skipped over. I disagree though, that it will stop innovation. On the contrary, if the capabilities and knowledge are spread, it allows people to build on that. Currently, I think, we vastly favour novelty over reproducibility. We still can have "innovation", but I think, it is right to take the novelty under consideration, if it hasn't been reproduced. In my opinion, exceptionalism is used more often as not as protectionism. If results in physics produced at the LHC are held to that standard, I do not see, why we let anyone else of the hook. Unfortunately, in the current "publish-or-die" environment, there is next to no incentive to reproduce results or to make them reproducible. I do not get academic jobs for merely reproducing results, you cannot publish that stuff, or only on low impact papers. And if I do produce something new and others can easily reproduce my results, then they can simply pick up the work where I started, and I lose my edge. Even if I am more open to competition, it still may be a lot of work to make it reproducible for others, time I rather can spend on furthering my research. E.g. there is no excuse, that in computer science practically all the results can be easily reproduced, but how many are?
    - jrumbut 6y ago
      I think, far from edge cases, there are entire, large fields that would be hampered by these guidelines. There could be parts of machine learning and the social sciences where these guidelines would work (and maybe others, I have no idea and it feels like the authors don't either). Articles like this need to get a lot more specific about what fields they're talking about. I think the best idea would be to name specific publication venues or funding opportunities that should adopt these standards. It should be that granular. Otherwise, this is one way that bureaucratic overhead, nonsensical policies, and perverse incentives are created.
  - learnstats2 6y ago
    > What if this was in place when someone performed the first open heart surgeries? I'm unclear if you're imagining here a scholarly article about a heart surgery, or a press release from the university hospital saying it was done. Research happens all the time without formal publication - it makes sense that there are different standards. I am proposing that there it would be beneficial for science to have journals with a modestly higher standard.

by yokaze 6y ago

learnstats2 6y ago

I don't think this has to be adversary. Journals choose to set a standard for publication, which is generally peer-review. If journals instead set a standard which was that the work must be reproducible (in other words: there must be instruction enough for an independent team to reasonably be able to follow and repeat what you did), then that sets a higher standard for science. Note that the peers in peer review are likely to mutually ignore that type of requirement - rather than face that requirement themselves.

bonoboTP 6y ago
Peer review in practice often means you got 2 or 3 grad students who skimmed the paper and found it to have the right gestalt an impressive enough results. Reviewers aren't some magical wise expert committee with infinite focus and they don't do any kind of background "auditing". They get a stack of papers to review and don't usually spend more than a few hours on each. Judging by the quality of some reviews I've seen, many grad students take even less than an hour to write a "review". I think the general public has a way overinflated idea of what peer review means in day to day practice.
jrumbut 6y ago
This would push the dial in the direction of incremental, very conservative research and discourage novel methods. What if this was in place when someone performed the first open heart surgeries? "To reproduce these results, put the patient to sleep and make you've got lots of towels..." Even interviewing techniques, specific cell lines, access to patients with a rare disease, none of these things are easily transferable but they are among the sources of the greatest new discoveries.
- yokaze 6y ago
  > This would push the dial in the direction of incremental, very conservative research and discourage novel methods. Yes, there there are obviously edge cases, where reproduction is difficult or impossible. I think, the effort needed to do so should also be honoured, not simply skipped over. I disagree though, that it will stop innovation. On the contrary, if the capabilities and knowledge are spread, it allows people to build on that. Currently, I think, we vastly favour novelty over reproducibility. We still can have "innovation", but I think, it is right to take the novelty under consideration, if it hasn't been reproduced. In my opinion, exceptionalism is used more often as not as protectionism. If results in physics produced at the LHC are held to that standard, I do not see, why we let anyone else of the hook. Unfortunately, in the current "publish-or-die" environment, there is next to no incentive to reproduce results or to make them reproducible. I do not get academic jobs for merely reproducing results, you cannot publish that stuff, or only on low impact papers. And if I do produce something new and others can easily reproduce my results, then they can simply pick up the work where I started, and I lose my edge. Even if I am more open to competition, it still may be a lot of work to make it reproducible for others, time I rather can spend on furthering my research. E.g. there is no excuse, that in computer science practically all the results can be easily reproduced, but how many are?
  - jrumbut 6y ago
    I think, far from edge cases, there are entire, large fields that would be hampered by these guidelines. There could be parts of machine learning and the social sciences where these guidelines would work (and maybe others, I have no idea and it feels like the authors don't either). Articles like this need to get a lot more specific about what fields they're talking about. I think the best idea would be to name specific publication venues or funding opportunities that should adopt these standards. It should be that granular. Otherwise, this is one way that bureaucratic overhead, nonsensical policies, and perverse incentives are created.
- learnstats2 6y ago
  > What if this was in place when someone performed the first open heart surgeries? I'm unclear if you're imagining here a scholarly article about a heart surgery, or a press release from the university hospital saying it was done. Research happens all the time without formal publication - it makes sense that there are different standards. I am proposing that there it would be beneficial for science to have journals with a modestly higher standard.