3 ms·

> • Many papers can't be replicated by design because the methodology either isn't actually described at all, or doesn't follow logically from the hypothesis. A

> • Many papers can't be replicated by design because the methodology either isn't actually described at all, or doesn't follow logically from the hypothesis. Any attempt would immediately fail after the first five minutes of reading the paper because you wouldn't know what to do. This is indeed a problem. But it can be solved by journals not accepting any paper that is not 1) described in such a way that it can be fully replicated, and 2) include the code and the data used to generate the published results Just that would go a long way. And then fund replication studies for consequential papers. Not just because of fraud but because of unintended flaws in studies, which can happen of course.

mike_hearn 2y ago
Journals can't police academia because their customers are university libraries. It's just not going to happen: journals are so far gone that they routinely publish whole editions filled with fake AI generated content and nobody even notices, so expecting them to enforce scientific rigor on the entire grant-funded science space is a non-starter ... and very indirect. Why should journals do this anyway? It's the people paying who should be checking that the work is done to a sufficient standard.
- insane_dreamer 2y ago
  You’re right but it’s what journals are supposed to do - that’s why we pay. Otherwise we don’t need journals and everyone can just self publish to bioRxiv.
  - mike_hearn 2y ago
    They'll only do it if the customers demand it. But the customers are universities, if they cared they could just fix the problem at the source by auditing and validating studies themselves. They don't need to pay third parties to do it. Journals often don't have sufficient lab access to detect fraud anyway. I do think the journal ecosystem can just disappear and nobody would care. It only exists to provide a pricing mechanism within the Soviet-style planned economy that academia has created. If that's replaced by a different pricing mechanism academic publishing could be rapidly re-based on top of Substack blogs and nothing of value would be lost.

mike_hearn 2y ago

Journals can't police academia because their customers are university libraries. It's just not going to happen: journals are so far gone that they routinely publish whole editions filled with fake AI generated content and nobody even notices, so expecting them to enforce scientific rigor on the entire grant-funded science space is a non-starter ... and very indirect. Why should journals do this anyway? It's the people paying who should be checking that the work is done to a sufficient standard.

insane_dreamer 2y ago
You’re right but it’s what journals are supposed to do - that’s why we pay. Otherwise we don’t need journals and everyone can just self publish to bioRxiv.
- mike_hearn 2y ago
  They'll only do it if the customers demand it. But the customers are universities, if they cared they could just fix the problem at the source by auditing and validating studies themselves. They don't need to pay third parties to do it. Journals often don't have sufficient lab access to detect fraud anyway. I do think the journal ecosystem can just disappear and nobody would care. It only exists to provide a pricing mechanism within the Soviet-style planned economy that academia has created. If that's replaced by a different pricing mechanism academic publishing could be rapidly re-based on top of Substack blogs and nothing of value would be lost.