18 ms·

I should point out that the notion of "replication" can often be way more difficult and nuanced than people expect. For one, what is the scope of the replicatio

I should point out that the notion of "replication" can often be way more difficult and nuanced than people expect. For one, what is the scope of the replication? Would it be simply to re-run the analysis on the data and make sure the math checks out? Or would it be to re-collect the data according to the methods described by the original researchers? The former is pretty easy, but only catches errors in the analysis phase (i.e., the data itself could be flawed). The latter is very comprehensive, but you essentially have to double up the effort on re-doing the study---which may not always be possible if you're studying a moving target (e.g., how the original SARS-CoV-2 variant spread through the initial set of hosts).

epgui 5y ago
Here's an even easier set of requirement to simplify the first case: - Require all research to publish their source code. - Require all research to publish their raw data minus "PII". * Note: I use "PII" here with the intention of it taking the most liberal meaning possible, where privacy trumps transparency absolutely and where de-anonymization is impossible. This would rule out a lot of data, and personally I think we could take a more balanced approach, but even this minimalist approach would be a vast improvement on the current situation.
- bjelkeman-again 5y ago
  When I learned at university that not all published research, especially government funded, didn’t do this already I was dumbfounded.
  - epgui 5y ago
    "Not all" is a big understatement... I would estimate that less than 0.00001% of published research does this. Every time I talk about this to someone (colleagues in adjacent fields, PIs...), they seem to give zero pucks. It's really mind-boggling.
OrderlyTiamat 5y ago
> re-run the analysis on the data and make sure the math checks out? That isn't a replication in any meaningful sense. But a replication can certainly take many forms. An exact replication is one, another could be to do a conceptual replication, so studying the same effect but with a different design, or combining these with a new analysis pooling the data from both the study and the new study with (possibly) improved statistical analysis.

epgui 5y ago

Here's an even easier set of requirement to simplify the first case: - Require all research to publish their source code. - Require all research to publish their raw data minus "PII". * Note: I use "PII" here with the intention of it taking the most liberal meaning possible, where privacy trumps transparency absolutely and where de-anonymization is impossible. This would rule out a lot of data, and personally I think we could take a more balanced approach, but even this minimalist approach would be a vast improvement on the current situation.

bjelkeman-again 5y ago
When I learned at university that not all published research, especially government funded, didn’t do this already I was dumbfounded.
- epgui 5y ago
  "Not all" is a big understatement... I would estimate that less than 0.00001% of published research does this. Every time I talk about this to someone (colleagues in adjacent fields, PIs...), they seem to give zero pucks. It's really mind-boggling.

OrderlyTiamat 5y ago

> re-run the analysis on the data and make sure the math checks out? That isn't a replication in any meaningful sense. But a replication can certainly take many forms. An exact replication is one, another could be to do a conceptual replication, so studying the same effect but with a different design, or combining these with a new analysis pooling the data from both the study and the new study with (possibly) improved statistical analysis.