4 ms·

That's like saying money spent fighting arson is well invested because you have extinguish accidental fires sometimes anyway.

waffleiron 5y ago
Closely related: https://en.wikipedia.org/wiki/Parable_of_the_broken_window https://en.wikipedia.org/wiki/Parable_of_the_broken_window
- rmbyrro 5y ago
  I don't think it's related. We need to reproduce every single study, including the fraudulent ones. Gosh, ESPECIALLY the fraudulent ones. But we can't possibly know in advance which are which. That's the whole purpose of trying to reproduce. Without reproduction efforts, scientific papers are of no value, as we can't trust them. In this sense, a paper without reproduction effort is not a "window" in the analogy. Only after reproduction efforts it becomes a window. So saying it's a waste to try reproducing an article is just absurd. It's like saying it's a waste to produce the "window" in the first place because a kid might break it in the future (in case of the paper, that we might figure out the paper was fraudulent). It's still valuable to find out a paper was fraudulent. It builds trust in the whole scientific endeavor.
  - _Nat_ 5y ago
    Then how do you trust the replication-studies? Just keep replicating them, too? The overall process assumes trustworthiness. The better that assumption holds, the better science performs.
    - EricE 5y ago
      I don't think many serious scientists would consider assumptions a valuable part of the scientific method.
      - _Nat_ 5y ago
        There wouldn't be a point to reading papers/textbooks, buying equipment from others, etc., if you thought that it was all part of some elaborate deception. The general presumption would tend to be that other scientists are trying to do good, honest work. Scientific progress does better when that presumption turns out to be more true. To be clear, I was responding to their apparent disregard for the harm done by the original alleged-fraud.
    - rmbyrro 5y ago
      It's not a flawless system. Science is highly reliant on humans and we're not reliable. But it's reasonable to think that some reproduction is better than no reproduction.
      - _Nat_ 5y ago
        Checks are good, definitely. And they help to limit the damage caused by stuff like fraud. You were right about that stuff. The point I think folks were contesting was the tone some may've inferred from [this comment](https://news.ycombinator.com/item?id=30787447 https://news.ycombinator.com/item?id=30787447 ), which may sound a bit like downplaying the damage caused by fraud under the premise that reproduction ought to catch such things anyway. For a computing analogy, TCP uses mechanisms that enable it to tolerate packet-loss. But packet-loss still slows things down, limiting bandwidth and increasing latency, until a connection would be impractical. For a connection with many hops, even a small packet-loss-rate along each hop would be a huge problem, even though TCP actively checks for packet-loss and tries to fix it. Likewise, science-with-replication presumably ought to be able to fix a lot of fraud and errors -- but those problems take a heavy, debilitating toll on scientific progress; it's not a small problem that replication-studies alone could satisfactorily rectify.
        rmbyrro 5y ago
        What the people inferring from my comment are missing is that the reproduction efforts actually saved us from damage. The real damage would have been everyone believing the paper conclusions were solid and start to act according to this false belief. Reproduction efforts saved us from that damage. These efforts were not the damage. Whoever is seeing differently doesn't understand one of the basic foundations of how the scientific endeavor works. And again: anyone expecting a world where there are no dishonest scientists or where the dishonest ones call themselves out in advance to save from the "damage" of people spending time reproducing their papers, is just playing wishful thinking, which is not a recipe for success...
        _Nat_ 5y ago
moonchrome 5y ago
The difference is that extinguishing a fire has no net benefit other than fixing the problem caused. The data collected in attempting to reproduce something is valuable even without original claims (especially is shared appropriately). So I agree with OP the time wasn't wasted, but it could have been spent on better opportunities.
rmbyrro 5y ago
This is wishful thinking. There will always be dishonest scientists and we need to keep trying to reproduce everything to filter them out.
- himinlomax 5y ago
  The more dishonest scientists, the more time we waste on them.
  - rmbyrro 5y ago
    No, the same time will be spent. You can't flag "this scientist is honest, never reproduce anything from him anymore". He can turn dishonest the next day you do this. If there are more dishonest scientists, we'll just find more useless papers. But the time to reproduce will be the same.

waffleiron 5y ago

Closely related: https://en.wikipedia.org/wiki/Parable_of_the_broken_window https://en.wikipedia.org/wiki/Parable_of_the_broken_window

rmbyrro 5y ago
I don't think it's related. We need to reproduce every single study, including the fraudulent ones. Gosh, ESPECIALLY the fraudulent ones. But we can't possibly know in advance which are which. That's the whole purpose of trying to reproduce. Without reproduction efforts, scientific papers are of no value, as we can't trust them. In this sense, a paper without reproduction effort is not a "window" in the analogy. Only after reproduction efforts it becomes a window. So saying it's a waste to try reproducing an article is just absurd. It's like saying it's a waste to produce the "window" in the first place because a kid might break it in the future (in case of the paper, that we might figure out the paper was fraudulent). It's still valuable to find out a paper was fraudulent. It builds trust in the whole scientific endeavor.
- _Nat_ 5y ago
  Then how do you trust the replication-studies? Just keep replicating them, too? The overall process assumes trustworthiness. The better that assumption holds, the better science performs.
  - EricE 5y ago
    I don't think many serious scientists would consider assumptions a valuable part of the scientific method.
    - _Nat_ 5y ago
      There wouldn't be a point to reading papers/textbooks, buying equipment from others, etc., if you thought that it was all part of some elaborate deception. The general presumption would tend to be that other scientists are trying to do good, honest work. Scientific progress does better when that presumption turns out to be more true. To be clear, I was responding to their apparent disregard for the harm done by the original alleged-fraud.
  - rmbyrro 5y ago
    It's not a flawless system. Science is highly reliant on humans and we're not reliable. But it's reasonable to think that some reproduction is better than no reproduction.
    - _Nat_ 5y ago
      Checks are good, definitely. And they help to limit the damage caused by stuff like fraud. You were right about that stuff. The point I think folks were contesting was the tone some may've inferred from [this comment](https://news.ycombinator.com/item?id=30787447 https://news.ycombinator.com/item?id=30787447 ), which may sound a bit like downplaying the damage caused by fraud under the premise that reproduction ought to catch such things anyway. For a computing analogy, TCP uses mechanisms that enable it to tolerate packet-loss. But packet-loss still slows things down, limiting bandwidth and increasing latency, until a connection would be impractical. For a connection with many hops, even a small packet-loss-rate along each hop would be a huge problem, even though TCP actively checks for packet-loss and tries to fix it. Likewise, science-with-replication presumably ought to be able to fix a lot of fraud and errors -- but those problems take a heavy, debilitating toll on scientific progress; it's not a small problem that replication-studies alone could satisfactorily rectify.
      - rmbyrro 5y ago
        What the people inferring from my comment are missing is that the reproduction efforts actually saved us from damage. The real damage would have been everyone believing the paper conclusions were solid and start to act according to this false belief. Reproduction efforts saved us from that damage. These efforts were not the damage. Whoever is seeing differently doesn't understand one of the basic foundations of how the scientific endeavor works. And again: anyone expecting a world where there are no dishonest scientists or where the dishonest ones call themselves out in advance to save from the "damage" of people spending time reproducing their papers, is just playing wishful thinking, which is not a recipe for success...
        _Nat_ 5y ago

moonchrome 5y ago

The difference is that extinguishing a fire has no net benefit other than fixing the problem caused. The data collected in attempting to reproduce something is valuable even without original claims (especially is shared appropriately). So I agree with OP the time wasn't wasted, but it could have been spent on better opportunities.

rmbyrro 5y ago

This is wishful thinking. There will always be dishonest scientists and we need to keep trying to reproduce everything to filter them out.

himinlomax 5y ago
The more dishonest scientists, the more time we waste on them.
- rmbyrro 5y ago
  No, the same time will be spent. You can't flag "this scientist is honest, never reproduce anything from him anymore". He can turn dishonest the next day you do this. If there are more dishonest scientists, we'll just find more useless papers. But the time to reproduce will be the same.