4 ms·

This is why funding replication studies and letting people publish null results and reproductions of important results is fundamental. It will introduce a stro

by nextos 2y ago

This is why funding replication studies and letting people publish null results and reproductions of important results is fundamental. It will introduce a strong incentive to be honest. Liars will get caught rather quickly. Right now, it often takes decades to uncover fraud.

ZYbCRq22HbJ2y7 2y ago
> Right now, it often takes decades to uncover fraud. Where? Many in this thread are talking as if there is monoculture in academia.
mike_hearn 2y ago
Unfortunately as you spend more time investigating this problem it becomes clear that replication studies aren't the answer. They're a bandage over the bleeding but don't address the root causes, and would have nearly no impact even if funded at a much larger scale. Because this suggestion comes up in every single HN thread about scientific fraud I eventually wrote an essay on why this is the case: https://blog.plan99.net/replication-studies-cant-fix-science-0e195234a280 https://blog.plan99.net/replication-studies-cant-fix-science... (press escape to dismiss the banner). If you're really interested in the topic please read it but here's a brief summary: • Replication studies don't solve many of the most common types of scientific fraud. Instead, you just end up replicating the fraud itself. This is usually because the methodology is bad, but if you try to fix the methodology to be scientific the original authors just claim you didn't do a genuine replication. • Many papers can't be replicated by design because the methodology either isn't actually described at all, or doesn't follow logically from the hypothesis. Any attempt would immediately fail after the first five minutes of reading the paper because you wouldn't know what to do. It's not clear what happens to the money if someone gets funded to replicate such a paper. Today it's not a problem because replicators choose which papers to replicate themselves, it's not a systematic requirement. • The idea implicitly assumes that very few researchers are corrupt thus the replicators are unlikely to also be. This isn't the case because replication failures are often due to field-wide problems, meaning replications will be done by the same insiders who benefit from the status quo and who signed off on the bad papers in the first place. This isn't an issue today because the only people who do replication studies are genuinely interested in whether the claims are true, it's not just a procedural way to get grant monies. • Many papers aren't worth replicating because they make trivial claims. If you punish non-replication without fixing the other incentive problems, you'll just pour accelerant on the problem of academics making obvious claims (e.g. the average man would like to be more muscular), and that just replaces one trust destroying problem with another. Replication failure is a symptom not a cause. The cause is systematically bad incentives.
- boxed 2y ago
  Incentives are themselves dangerous. We should treat incentives like guns. Instead we apply incentives to all manner of problems and are surprised they backfire and destroy everything. You have to give people less incentives and more just time to do their basic job.
  - foldr 2y ago
    I think this hits the nail on the head. Academics have been treated like assembly line workers for decades now. So they’ve collectively learned how to consistently place assembled product on the conveyor belt. The idea that scientific output is a stack of publications is pretty absurd if you think about it for a few minutes. But try telling that to the MBA types who now run universities.
    - mike_hearn 2y ago
      You do need to incentivize something. If you incentivize nothing that's the same thing as an institution not existing and science being done purely as a hobby. You can get some distance that way - it's how science used to work - but the moment you want the structure and funding an institution can provide you must set incentives. Otherwise people could literally just stop turning up for work and still get paid, which is obviously not going to be acceptable to whoever is funding it.
      - foldr 2y ago
        I think it’s an interesting feature of current culture that we take it as axiomatic that people need to be ‘incentivized’. I’m not sure I agree. To me that axiom seems to be at the root of a lot of the problems we’re talking about in this thread. (Yes, everyone is subject to incentives in some broad sense, but acknowledging that doesn’t mean that we have to engineer specific incentives as a means to desired outcomes.)
        krisoft 2y ago
hwillis 2y ago
That's a tremendously expensive way to detect fraud. There's funding, but also the people replicating need to have at least near the expertise of the original authors, and the rate of false positives is clearly going to be high. Mistakes on the part of the replicator will look the same as scientific fraud. Maybe most worryingly, the negative impact of human studies would be doubled. More than doubled, probably- look at how many people claimed to successfully replicate LK-99. A paper may need to be replicated many time to identify unknown. Maybe first we could just try specifically looking for fraud? Like recording data with 3rd parties (ensuring that falsified data will at least be recorded suspiciously), or a body that looks for fabricated data or photoshopped figures?

by nextos 2y ago

ZYbCRq22HbJ2y7 2y ago

> Right now, it often takes decades to uncover fraud. Where? Many in this thread are talking as if there is monoculture in academia.

mike_hearn 2y ago

Unfortunately as you spend more time investigating this problem it becomes clear that replication studies aren't the answer. They're a bandage over the bleeding but don't address the root causes, and would have nearly no impact even if funded at a much larger scale. Because this suggestion comes up in every single HN thread about scientific fraud I eventually wrote an essay on why this is the case: https://blog.plan99.net/replication-studies-cant-fix-science-0e195234a280 https://blog.plan99.net/replication-studies-cant-fix-science... (press escape to dismiss the banner). If you're really interested in the topic please read it but here's a brief summary: • Replication studies don't solve many of the most common types of scientific fraud. Instead, you just end up replicating the fraud itself. This is usually because the methodology is bad, but if you try to fix the methodology to be scientific the original authors just claim you didn't do a genuine replication. • Many papers can't be replicated by design because the methodology either isn't actually described at all, or doesn't follow logically from the hypothesis. Any attempt would immediately fail after the first five minutes of reading the paper because you wouldn't know what to do. It's not clear what happens to the money if someone gets funded to replicate such a paper. Today it's not a problem because replicators choose which papers to replicate themselves, it's not a systematic requirement. • The idea implicitly assumes that very few researchers are corrupt thus the replicators are unlikely to also be. This isn't the case because replication failures are often due to field-wide problems, meaning replications will be done by the same insiders who benefit from the status quo and who signed off on the bad papers in the first place. This isn't an issue today because the only people who do replication studies are genuinely interested in whether the claims are true, it's not just a procedural way to get grant monies. • Many papers aren't worth replicating because they make trivial claims. If you punish non-replication without fixing the other incentive problems, you'll just pour accelerant on the problem of academics making obvious claims (e.g. the average man would like to be more muscular), and that just replaces one trust destroying problem with another. Replication failure is a symptom not a cause. The cause is systematically bad incentives.

boxed 2y ago
Incentives are themselves dangerous. We should treat incentives like guns. Instead we apply incentives to all manner of problems and are surprised they backfire and destroy everything. You have to give people less incentives and more just time to do their basic job.
- foldr 2y ago
  I think this hits the nail on the head. Academics have been treated like assembly line workers for decades now. So they’ve collectively learned how to consistently place assembled product on the conveyor belt. The idea that scientific output is a stack of publications is pretty absurd if you think about it for a few minutes. But try telling that to the MBA types who now run universities.
  - mike_hearn 2y ago
    You do need to incentivize something. If you incentivize nothing that's the same thing as an institution not existing and science being done purely as a hobby. You can get some distance that way - it's how science used to work - but the moment you want the structure and funding an institution can provide you must set incentives. Otherwise people could literally just stop turning up for work and still get paid, which is obviously not going to be acceptable to whoever is funding it.
    - foldr 2y ago
      I think it’s an interesting feature of current culture that we take it as axiomatic that people need to be ‘incentivized’. I’m not sure I agree. To me that axiom seems to be at the root of a lot of the problems we’re talking about in this thread. (Yes, everyone is subject to incentives in some broad sense, but acknowledging that doesn’t mean that we have to engineer specific incentives as a means to desired outcomes.)
      - krisoft 2y ago

hwillis 2y ago

That's a tremendously expensive way to detect fraud. There's funding, but also the people replicating need to have at least near the expertise of the original authors, and the rate of false positives is clearly going to be high. Mistakes on the part of the replicator will look the same as scientific fraud. Maybe most worryingly, the negative impact of human studies would be doubled. More than doubled, probably- look at how many people claimed to successfully replicate LK-99. A paper may need to be replicated many time to identify unknown. Maybe first we could just try specifically looking for fraud? Like recording data with 3rd parties (ensuring that falsified data will at least be recorded suspiciously), or a body that looks for fabricated data or photoshopped figures?