4 ms·
"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, rese
by ZeroGravitas 10y ago
"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. "
Ironically, it says this as a bad thing, but in an ideal world this would be 100%.
It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't found any points more in the direction that they're not looking hard enough.
edit: pretty much everyone seems to have taken this the opposite way to how I intended it, but re-reading I can't figure out why that is the case. I'll try to re-phrase:
Science cannot be perfect every time. It's just too complex. This is why you need thorough peer review including reproduction. But if that peer review/reproduction is thorough, then it's going to find problems. When the system is working well, basically everyone will have at some point found a problem in something they are reproducing. This is good because that problem can then be fixed and it will become reproducible or be withdrawn. The current situation is that people don't even look for the problems and no-one can trust results.
edited again to change "peer-reviewed" -> "peer-reviewed including reproduction"
- droopyEyelids 10y agoThe whole point of an experiment is to isolate a single variable so you can test a falsifiable statement about it. It's the exact opposite of building a system, which is what coders do.
- tgb 10y agoI don't think that analogy is at all accurate and I think the conclusion that you reach from it is completely incorrect.
- ZeroGravitas 10y agoI think I must have expressed myself poorly, as I think my conclusion is the same as the article suggests i.e. that the science/code shouldn't be considered "done", until it's been peer-reviewed, since it's easy to fool yourself and others if you're not actually reviewing and testing your code. What did you take away from my comment?
- gcp 10y agoPeer review does not imply reproducibility, and it's the latter that is the problem. I can confirm, as a reviewer, that your methodology and analysis looks sensible, but the flaws may be deeper, and the fact that you didn't publish the 19 other studies that failed, but that this is the "lucky one", or that you simply cherry picked the data, is not something I can see as a reviewer. This is especially true if the experiment is nontrivial to re-do.
- ZeroGravitas 10y ago" Peer review does not imply reproducibility, and it's the latter that is the problem." I think this is the key to it, I'm suggesting that reproducibility should be part of considering something peer-reviewed, but of course as currently practised, that isn't true. Of course in a software metaphor, that would probably cover both code review and QA, which is sometimes done by a different job role which further muddies the water.
- pvaldes 10y agoThis would be like expecting a car brand to open its code to its competitors before to release a new product. Will lead typically also to the peers rushing to publish the same discovering on disguise before the original.
- tgb 10y agoAfter reading your explanations elsewhere, I take back my statements. Your statements are literally correct, although easy to take incorrectly, and I support them: in particular you seem to be supporting that 100% of researchers should attempt to replicate studies and do so sufficiently that all of them will eventually fail to replicate at least some studies. I think most of us took this to mean that you thought every study should fail to replicate (by analogy that every software has at least some bugs), but I see now your intent and that your original wording backed that up.
- endorphone 10y agoIt is a terrible thing, and it is absolutely nothing like finding bugs. Reproducibility is a core requirement of good science, and if we need to compare it to software engineering, the reproducibility crisis is like the adage "many eyes make all bugs shallow", when the assumption that there is many eyes even looking is often untrue. Most studies are never reproduced, but are held as true under the belief that if someone tried they could. EDIT: You claimed that in an ideal world, 100% of experiments/studies would not be reproducible. This denotes a profound misunderstanding of the scientific process, or the whole basis of reproducibility. In an idea world, 100% of studies would be vetted through reproduction, and 100% of them would be reproducible. This is essentially the fundamental assumption of the scientific process.
- therpe1 10y agoAn experiment should be reproducible. The fact that there exist many experiments that are not reproducible is not a problem and, as the parent post indicates, the fact that these experiments are being discovered as non-reprodicible is a good thing. This is science at work. The system is error-correcting itself in real-time. We are already seeing laboratories devise new approaches and innovations that will, over time, reduce non-reproducibility errors. The "reproducibility crisis" -- isn't a crisis at all. There's nothing fundamentally wrong with science. Historians of science will find this era amusing though. When the reproducibility crisis was limited to psychology and the "soft sciences" nobody really took it serious, in fact many people leaped at the opportunity to point out that these sciences aren't "washy." Now, interestingly enough, we're seeing essentially everything but physics (the 'hardest', but really the 'simplest' and most 'stable') science suffer the same fate and this idea of science as a socio-historical process is finally starting to sink in. It only took a good 80 years but here we are.
- ZeroGravitas 10y agoNo, I claimed that all scientist would have had the experience of not reproducing something. Because if they do it a lot, as part of a regular process then they will eventually find something that doesn't work because the original scientist didn't document a step correctly or misread the results or just got lucky due to random chance. Just like all developers will eventually find a bug in code they code review. This is different from all code they review having bugs.
- eclee2 10y agoIf there are errors in a study's methods that make it unreplicable, then it shouldn't have passed peer review or been published.
- DanBC 10y agoHow do you know it's unreplicable until someone tries to replicate it?
- rokosbasilisk 10y agoI work with principal investigators phd/mds at upenn automating some of their data analysis pipelines. They have all have secret checklists for bs detection in papers they read. Certain labs set off red flags to them or certain techniques being too fuzzy or easy to mess up. Every one seems to have their own heuristics, and no one seems to take any article at face value anymore. I hear PI's say stuff is unreplicable all the time.
- pvaldes 10y agoThen you risk to lose all einstein work unless you have two einsteins at the same time in fact. Genious are scarce. And you could not publish nothing about comets for example, because this would be unreplicable until the next 20 years. Is not so simple as that.
- pvaldes 10y agoScience publishing is based in a review by peers system. All evident bugs (and many non so evident bugs) should be catched before to appear in a journal. Is totally different to standard journalism.
- TorKlingberg 10y agoReplication happens after publication. Why is everyone misunderstanding ZeroGravitas' point?
- pmontra 10y agoI agree that there are no test suites to provide experiments to run and no make test to repeat an experiment with little effort. This accounts for the "too complex" thing. However many experiments should be reproducible. Not making results testable is against the goal of sharing knowledge. But I understand that's an extra effort compared the current state of the art, and that must be rewarded and acknowledged. In another comment I proposed to include reproducibility in the h-index.