6 ms·

One thing I'd like to see is a requirement that for all government funded research, a certain percentage of that funding, say 30%, must go toward replicating ot

by 0x0203 5y ago

One thing I'd like to see is a requirement that for all government funded research, a certain percentage of that funding, say 30%, must go toward replicating other publicly funded research that has had less than 2 independent and non-affiliated labs replicate. Any original research couldn't be published until at least two independent and non-affiliated labs replicate based on the submitted paper and report on the results that can then be included with the original research. I'd like to see this across all of academia, but I imagine there are enough challenges with enforcing this in a productive manner already that doing it across all research becomes both impractical and difficult to prevent abuse. But at least with public funds, it would be nice to put in some checks to reduce the amount of fraudulent or sloppy research that tax payers pay for.

the_snooze 5y ago
I should point out that the notion of "replication" can often be way more difficult and nuanced than people expect. For one, what is the scope of the replication? Would it be simply to re-run the analysis on the data and make sure the math checks out? Or would it be to re-collect the data according to the methods described by the original researchers? The former is pretty easy, but only catches errors in the analysis phase (i.e., the data itself could be flawed). The latter is very comprehensive, but you essentially have to double up the effort on re-doing the study---which may not always be possible if you're studying a moving target (e.g., how the original SARS-CoV-2 variant spread through the initial set of hosts).
- epgui 5y ago
  Here's an even easier set of requirement to simplify the first case: - Require all research to publish their source code. - Require all research to publish their raw data minus "PII". * Note: I use "PII" here with the intention of it taking the most liberal meaning possible, where privacy trumps transparency absolutely and where de-anonymization is impossible. This would rule out a lot of data, and personally I think we could take a more balanced approach, but even this minimalist approach would be a vast improvement on the current situation.
  - bjelkeman-again 5y ago
    When I learned at university that not all published research, especially government funded, didn’t do this already I was dumbfounded.
    - epgui 5y ago
      "Not all" is a big understatement... I would estimate that less than 0.00001% of published research does this. Every time I talk about this to someone (colleagues in adjacent fields, PIs...), they seem to give zero pucks. It's really mind-boggling.
- OrderlyTiamat 5y ago
  > re-run the analysis on the data and make sure the math checks out? That isn't a replication in any meaningful sense. But a replication can certainly take many forms. An exact replication is one, another could be to do a conceptual replication, so studying the same effect but with a different design, or combining these with a new analysis pooling the data from both the study and the new study with (possibly) improved statistical analysis.
mike_hearn 5y ago
Be aware that despite how much focus replicability gets, it's only one of many things that goes wrong with research papers. Even if you somehow waved a magic wand and fixed replicability perfectly tomorrow, entire academic fields would still be worthless and misleading. How can replicable research go wrong? Here's just a fraction of the things I've seen reading papers: 1. Logic errors. So many logic errors. Replicating something that doesn't make sense leaves you with two things that don't make sense: a waste of time and money. 2. Tiny effect sizes. Often an effect will "replicate" but with a smaller effect than the one claimed; is this a successful replication or not? 3. Intellectual fraud. Often this works by taking a normal English term and then at the start of your paper giving it an incorrect definition. Again this will replicate just fine but the result is still misinformation. 4. Incoherent concepts. What exactly does R0 mean in epidemiology and precisely how is it determined? You can replicate the calculations that are used but you won't be calculating what you think you are. 5. A lot of research isn't experimental, it's purely observational. You can't go back and re-observe the things being studied, only re-analyze the data they originally collected. Does this count? 6. Incredibly obvious findings. Wealthy parents have more successful children, etc. It'll replicate all right but so what? Why are taxpayers being made to fund this stuff? 7. Fraudulent practices that are nonetheless normalized within a field. The article complains about scientists Photoshopping western blots (a type of artifact produced in biology experiments). That's because editing your data in ways that make it fit your theory is universally understood to be fraud ... except in climatology, where scientists have developed a habit of constantly rewriting the databases that contain historical temperature records. And by "historical" we mean "last year" here, not 1000 years ago. These edits always make global warming more pronounced, and sometimes actually create warming trends where previously there were none (e.g. [1]). Needless to say climatologists don't consider this fraud. It means if you're trying to replicate a claim from climatology, even an apparently factual claim about a certain fixed year, you may run into the problem that it was "true" at the time it was made and may even have been replicated, but is now "false" because the data has been edited since. Epidemiology has a somewhat similar problem - they don't consider deterministic models to be important, i.e. it may be impossible to get the same numbers out of a model as a paper presents, even if you give it identical inputs, due to race conditions/memory corruption bugs in the code. They do not consider this a problem and will claim it doesn't matter because the model uses a PRNG somewhere, or that they "replicated" the model outputs because they got numbers only 25% different. What does it even mean to say a claim does or does not replicate, in fields like these? All this takes place in an environment of near total institutional indifference. Paper replicates? Great. Nobody cares, because they all assumed it would. Paper doesn't replicate, or has methodological errors making replication pointless? Nobody cares about that either. Your proposal suggests blocking publication until replication is done by independent labs. That won't work, because even if you found some way to actually enforce that (not all grants come from the government!), you'll just end up with lots of papers that can be replicated but are still nonsensical for other reasons. [1] https://nature.com/articles/nature.2015.17700 https://nature.com/articles/nature.2015.17700

by 0x0203 5y ago

the_snooze 5y ago

I should point out that the notion of "replication" can often be way more difficult and nuanced than people expect. For one, what is the scope of the replication? Would it be simply to re-run the analysis on the data and make sure the math checks out? Or would it be to re-collect the data according to the methods described by the original researchers? The former is pretty easy, but only catches errors in the analysis phase (i.e., the data itself could be flawed). The latter is very comprehensive, but you essentially have to double up the effort on re-doing the study---which may not always be possible if you're studying a moving target (e.g., how the original SARS-CoV-2 variant spread through the initial set of hosts).

epgui 5y ago
Here's an even easier set of requirement to simplify the first case: - Require all research to publish their source code. - Require all research to publish their raw data minus "PII". * Note: I use "PII" here with the intention of it taking the most liberal meaning possible, where privacy trumps transparency absolutely and where de-anonymization is impossible. This would rule out a lot of data, and personally I think we could take a more balanced approach, but even this minimalist approach would be a vast improvement on the current situation.
- bjelkeman-again 5y ago
  When I learned at university that not all published research, especially government funded, didn’t do this already I was dumbfounded.
  - epgui 5y ago
    "Not all" is a big understatement... I would estimate that less than 0.00001% of published research does this. Every time I talk about this to someone (colleagues in adjacent fields, PIs...), they seem to give zero pucks. It's really mind-boggling.
OrderlyTiamat 5y ago
> re-run the analysis on the data and make sure the math checks out? That isn't a replication in any meaningful sense. But a replication can certainly take many forms. An exact replication is one, another could be to do a conceptual replication, so studying the same effect but with a different design, or combining these with a new analysis pooling the data from both the study and the new study with (possibly) improved statistical analysis.

mike_hearn 5y ago

Be aware that despite how much focus replicability gets, it's only one of many things that goes wrong with research papers. Even if you somehow waved a magic wand and fixed replicability perfectly tomorrow, entire academic fields would still be worthless and misleading. How can replicable research go wrong? Here's just a fraction of the things I've seen reading papers: 1. Logic errors. So many logic errors. Replicating something that doesn't make sense leaves you with two things that don't make sense: a waste of time and money. 2. Tiny effect sizes. Often an effect will "replicate" but with a smaller effect than the one claimed; is this a successful replication or not? 3. Intellectual fraud. Often this works by taking a normal English term and then at the start of your paper giving it an incorrect definition. Again this will replicate just fine but the result is still misinformation. 4. Incoherent concepts. What exactly does R0 mean in epidemiology and precisely how is it determined? You can replicate the calculations that are used but you won't be calculating what you think you are. 5. A lot of research isn't experimental, it's purely observational. You can't go back and re-observe the things being studied, only re-analyze the data they originally collected. Does this count? 6. Incredibly obvious findings. Wealthy parents have more successful children, etc. It'll replicate all right but so what? Why are taxpayers being made to fund this stuff? 7. Fraudulent practices that are nonetheless normalized within a field. The article complains about scientists Photoshopping western blots (a type of artifact produced in biology experiments). That's because editing your data in ways that make it fit your theory is universally understood to be fraud ... except in climatology, where scientists have developed a habit of constantly rewriting the databases that contain historical temperature records. And by "historical" we mean "last year" here, not 1000 years ago. These edits always make global warming more pronounced, and sometimes actually create warming trends where previously there were none (e.g. [1]). Needless to say climatologists don't consider this fraud. It means if you're trying to replicate a claim from climatology, even an apparently factual claim about a certain fixed year, you may run into the problem that it was "true" at the time it was made and may even have been replicated, but is now "false" because the data has been edited since. Epidemiology has a somewhat similar problem - they don't consider deterministic models to be important, i.e. it may be impossible to get the same numbers out of a model as a paper presents, even if you give it identical inputs, due to race conditions/memory corruption bugs in the code. They do not consider this a problem and will claim it doesn't matter because the model uses a PRNG somewhere, or that they "replicated" the model outputs because they got numbers only 25% different. What does it even mean to say a claim does or does not replicate, in fields like these? All this takes place in an environment of near total institutional indifference. Paper replicates? Great. Nobody cares, because they all assumed it would. Paper doesn't replicate, or has methodological errors making replication pointless? Nobody cares about that either. Your proposal suggests blocking publication until replication is done by independent labs. That won't work, because even if you found some way to actually enforce that (not all grants come from the government!), you'll just end up with lots of papers that can be replicated but are still nonsensical for other reasons. [1] https://nature.com/articles/nature.2015.17700 https://nature.com/articles/nature.2015.17700