3 ms·
Publishing failures to replicate without blame seems like a good way forward, similar to the practice of publishing postmortems in the computer industry.
by skybrian 11y ago
Publishing failures to replicate without blame seems like a good way forward, similar to the practice of publishing postmortems in the computer industry.
- will_pseudonym 11y agoI would say that this is more important than postmortems. This is new science that's being published. It's overturning knowledge assumed to be true.
- _delirium 11y agoWell, casting doubt on knowledge assumed to be true, which may lead to overturning, but is still provisional when you're at the point where you have one published experiment claiming a result, and one published failure to replicate that result. Even correct results are not always successfully replicated on the first replication attempt, including in "hard" areas like physics/chemistry. Lots of possible reasons. One frequent case is that the original results were, even if broadly correct, underspecified, i.e. there was some key component of the original experimental apparatus or method that wasn't sufficiently described in the paper, either due to an oversight or because the original authors didn't know it actually mattered to the outcome. Sometimes figuring out what that was makes it possible to replicate the result, although other times it turns out to undermine the original results (e.g. the entire result turns out to be due to some specific impurity in a specific supplier's equipment). But in any case still very useful.
- mirimir 11y agoYes! Been there, done that. So much of lab technique is hard to explain in words. People do postdocs and sabbaticals in order to learn technique. And then there are uncontrolled variables. Classic example: > When a student had difficulty in crystallizing a compound [professors] would simply shake their beards over the flask containing the offending substance. Then, lo and behold, after nucleation had done its job, crystallization set in. Gerhard believed that the beard of an Adolf von Baeyer or an Otto Wallach could indeed be a source of crystals of every conceivable space group. http://www.improbable.com/2011/02/09/legend-of-chrystallographers-beards/ http://www.improbable.com/2011/02/09/legend-of-chrystallogra... https://www.tes.com/teaching-resource/are-bearded-chemists-better-at-crystallization-6397626 https://www.tes.com/teaching-resource/are-bearded-chemists-b... http://tdl.libra.titech.ac.jp/hkshi/xc/contents/pdf/11709748X/5 http://tdl.libra.titech.ac.jp/hkshi/xc/contents/pdf/11709748...
- erikpukinskis 11y agoEverything should be published: positive results, negative results, positive confirmations, negative confirmations. Journals should only judge studies based on the quality of the science, not based on the "interestingness" of the outcome.
- sdabdoub 11y agoPLoS One and Nature Scientific Reports do exactly this (and I'm sure there are others I'm not aware of), but the list does need to be expanded.
- maxander 11y agoIts not merely the way forward- its the whole problem. Nothing of this is at all challenging methodologically or technologically- scientists, as a group, are masters of such things. The challenge is social. If you've failed to reproduce a result, this could be due to the original experimenter's flawed work, or your own mistakes, or unnoticed ambiguities in the experimental method, etc. There's no way to publish a negative replication without implicitly asserting, in the eyes of your fellows, that the original researcher did it wrong- its not what a negative result logically shows, but this is just how humans work. Jumping the gap between what you've shown and how your publication will be socially received takes substantial courage- scientific careers are fragile!