3 ms·

If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's m

If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.

Osmium 13y ago
> I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result when studied (usually due to minute trace impurities, or marginally different temperatures or flow rates during growth, or a host of other things–all of which need to be analysed and vetted–and this happens even if they're using growth reactors made by the same manufacturer!). It makes replication very difficult and often results in slow progress. And it's expensive too. So it's not just medicine that has that problem. Edit: I should add that obviously in the end it works out because these materials eventually make it into industry and into millions of devices. So it's ultimately reproducible. But that gap between first discovering a process or a mechanism or just something interesting and making it reproducible consistently enough and at a large enough scale for industry is just huge.
- mathattack 13y ago
  Indeed - thank you for clarifying. This does sound expensive, though all the more necessary.
- deleted 13y ago
  [deleted]
Fomite 13y ago
Beyond it being expensive to replicate once, for observational studies and clinical trials, you need many repetitions because of statistical variability. That gets expensive in a hurry.
- bmahmood 13y ago
  You are right that clinical trial studies would be much more time intensive and costly. The studies we will be replicating through the Reproducibility Initiative are more preclinical cancer studies, rather than later-stage clinical trials. We felt the preclinical stage was more important to validate, as much of the research is based on academic studies with over 60% failure rate for replication (see http://www.nature.com/nature/journal/v483/n7391/full/483531a.html#t1 http://www.nature.com/nature/journal/v483/n7391/full/483531a...) By clinical trials, a lot of the pharma companies have already done initial validation, so extra validation is redundant. We hope a reproducibility system to validate preclinical research can help patient groups, foundations, and industry better identify reproducible oncology targets, and hence the focus on preclinical studies (which in turn are less expensive).
  - Fomite 13y ago
    Agreed - and this is a hugely important effort. This is more me musing about replication vs. reproducibility (on the computational/statistical end) vs. repeatability.

Osmium 13y ago

> I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result when studied (usually due to minute trace impurities, or marginally different temperatures or flow rates during growth, or a host of other things–all of which need to be analysed and vetted–and this happens even if they're using growth reactors made by the same manufacturer!). It makes replication very difficult and often results in slow progress. And it's expensive too. So it's not just medicine that has that problem. Edit: I should add that obviously in the end it works out because these materials eventually make it into industry and into millions of devices. So it's ultimately reproducible. But that gap between first discovering a process or a mechanism or just something interesting and making it reproducible consistently enough and at a large enough scale for industry is just huge.

mathattack 13y ago
Indeed - thank you for clarifying. This does sound expensive, though all the more necessary.
deleted 13y ago
[deleted]

Fomite 13y ago

Beyond it being expensive to replicate once, for observational studies and clinical trials, you need many repetitions because of statistical variability. That gets expensive in a hurry.

bmahmood 13y ago
You are right that clinical trial studies would be much more time intensive and costly. The studies we will be replicating through the Reproducibility Initiative are more preclinical cancer studies, rather than later-stage clinical trials. We felt the preclinical stage was more important to validate, as much of the research is based on academic studies with over 60% failure rate for replication (see http://www.nature.com/nature/journal/v483/n7391/full/483531a.html#t1 http://www.nature.com/nature/journal/v483/n7391/full/483531a...) By clinical trials, a lot of the pharma companies have already done initial validation, so extra validation is redundant. We hope a reproducibility system to validate preclinical research can help patient groups, foundations, and industry better identify reproducible oncology targets, and hence the focus on preclinical studies (which in turn are less expensive).
- Fomite 13y ago
  Agreed - and this is a hugely important effort. This is more me musing about replication vs. reproducibility (on the computational/statistical end) vs. repeatability.