3 ms·

It is a terrible thing, and it is absolutely nothing like finding bugs. Reproducibility is a core requirement of good science, and if we need to compare it to

It is a terrible thing, and it is absolutely nothing like finding bugs. Reproducibility is a core requirement of good science, and if we need to compare it to software engineering, the reproducibility crisis is like the adage "many eyes make all bugs shallow", when the assumption that there is many eyes even looking is often untrue. Most studies are never reproduced, but are held as true under the belief that if someone tried they could. EDIT: You claimed that in an ideal world, 100% of experiments/studies would not be reproducible. This denotes a profound misunderstanding of the scientific process, or the whole basis of reproducibility. In an idea world, 100% of studies would be vetted through reproduction, and 100% of them would be reproducible. This is essentially the fundamental assumption of the scientific process.

therpe1 10y ago
An experiment should be reproducible. The fact that there exist many experiments that are not reproducible is not a problem and, as the parent post indicates, the fact that these experiments are being discovered as non-reprodicible is a good thing. This is science at work. The system is error-correcting itself in real-time. We are already seeing laboratories devise new approaches and innovations that will, over time, reduce non-reproducibility errors. The "reproducibility crisis" -- isn't a crisis at all. There's nothing fundamentally wrong with science. Historians of science will find this era amusing though. When the reproducibility crisis was limited to psychology and the "soft sciences" nobody really took it serious, in fact many people leaped at the opportunity to point out that these sciences aren't "washy." Now, interestingly enough, we're seeing essentially everything but physics (the 'hardest', but really the 'simplest' and most 'stable') science suffer the same fate and this idea of science as a socio-historical process is finally starting to sink in. It only took a good 80 years but here we are.
ZeroGravitas 10y ago
No, I claimed that all scientist would have had the experience of not reproducing something. Because if they do it a lot, as part of a regular process then they will eventually find something that doesn't work because the original scientist didn't document a step correctly or misread the results or just got lucky due to random chance. Just like all developers will eventually find a bug in code they code review. This is different from all code they review having bugs.
- endorphone 10y ago
  While the wording may be vague, they aren't talking about the experiences of a subset of researchers -- they are saying that of the experiments they tried to replicate, 2/3rds weren't reproduceable. That is terrible, and has absolutely nothing to do with finding bugs.

therpe1 10y ago

An experiment should be reproducible. The fact that there exist many experiments that are not reproducible is not a problem and, as the parent post indicates, the fact that these experiments are being discovered as non-reprodicible is a good thing. This is science at work. The system is error-correcting itself in real-time. We are already seeing laboratories devise new approaches and innovations that will, over time, reduce non-reproducibility errors. The "reproducibility crisis" -- isn't a crisis at all. There's nothing fundamentally wrong with science. Historians of science will find this era amusing though. When the reproducibility crisis was limited to psychology and the "soft sciences" nobody really took it serious, in fact many people leaped at the opportunity to point out that these sciences aren't "washy." Now, interestingly enough, we're seeing essentially everything but physics (the 'hardest', but really the 'simplest' and most 'stable') science suffer the same fate and this idea of science as a socio-historical process is finally starting to sink in. It only took a good 80 years but here we are.

ZeroGravitas 10y ago

No, I claimed that all scientist would have had the experience of not reproducing something. Because if they do it a lot, as part of a regular process then they will eventually find something that doesn't work because the original scientist didn't document a step correctly or misread the results or just got lucky due to random chance. Just like all developers will eventually find a bug in code they code review. This is different from all code they review having bugs.

endorphone 10y ago
While the wording may be vague, they aren't talking about the experiences of a subset of researchers -- they are saying that of the experiments they tried to replicate, 2/3rds weren't reproduceable. That is terrible, and has absolutely nothing to do with finding bugs.