13 ms·

I hope "Recomputability" emerges as a distinct term. At least in the biological sciences, I'm seeing the term "reproducibility" used a lot where the meaning is

by dasmoth 13y ago

I hope "Recomputability" emerges as a distinct term. At least in the biological sciences, I'm seeing the term "reproducibility" used a lot where the meaning is much closer to "recomputability", i.e. "you can repeat the exact computational steps we performed" -- without necessarily saying much about either the lab-work and/or sample-collection parts of the project, or the possibility of performing similar analyses using different tools/platforms. (I'd also like to see a bit more recognition of the importance of full reproduction -- i.e. someone starts with the same hypothesis or idea and does their own experiment -- in modern science).

turingfan 13y ago
I do agree ... I'd love "recomputation" and variants to catch on. I also agree completely about distinguishing between recomputation and a proper reproduction. It's not what I emphasise in the manifesto but it is true that a recomputation doesn't really say anything about the generality of a result.
_delirium 13y ago
That's where I see recomputation as not quite pushing the same goals as reproducibility, even though its advocates often couch them as the same goal. Recomputation can be useful, but re-running the exact same code in the same virtual machine isn't really an independent reproduction of the claimed scientific result. That often benefits from not using the original source code; two independently written implementations claiming to implement the same approach and achieving the same results is a much better reproduction. In the natural sciences, independent reproduction often finds subtle dependencies on the original apparatus that change the interpretation: when lab B tries to reproduce lab A's results on slightly different equipment and can't, it can highlight an unnoticed dependency on some specific feature of the original equipment, and may throw into question the original paper's conclusions. You would never have found that if, instead of lab B independently reproducing the result, lab A just packed up their equipment into a shipping container and shipped it to lab B, who unpacked and ran it unchanged. That's what the VM approach is arguing for, and that's not really reproducibility.
- turingfan 13y ago
  I completely agree. A recomputation of an experiment is not ensuring reproducibility of the scientific result. It's ensuring reproducibility of the individual experiment. The analogy I have given is with cold fusion. If we could reproduce their exact lab setup then we could find out if the results were real - i.e. were not misread or anything, and assuming they were, have a chance of explaining the anomaly. But no, it's not the same as reproducibility.

by dasmoth 13y ago

turingfan 13y ago

I do agree ... I'd love "recomputation" and variants to catch on. I also agree completely about distinguishing between recomputation and a proper reproduction. It's not what I emphasise in the manifesto but it is true that a recomputation doesn't really say anything about the generality of a result.

_delirium 13y ago

That's where I see recomputation as not quite pushing the same goals as reproducibility, even though its advocates often couch them as the same goal. Recomputation can be useful, but re-running the exact same code in the same virtual machine isn't really an independent reproduction of the claimed scientific result. That often benefits from not using the original source code; two independently written implementations claiming to implement the same approach and achieving the same results is a much better reproduction. In the natural sciences, independent reproduction often finds subtle dependencies on the original apparatus that change the interpretation: when lab B tries to reproduce lab A's results on slightly different equipment and can't, it can highlight an unnoticed dependency on some specific feature of the original equipment, and may throw into question the original paper's conclusions. You would never have found that if, instead of lab B independently reproducing the result, lab A just packed up their equipment into a shipping container and shipped it to lab B, who unpacked and ran it unchanged. That's what the VM approach is arguing for, and that's not really reproducibility.

turingfan 13y ago
I completely agree. A recomputation of an experiment is not ensuring reproducibility of the scientific result. It's ensuring reproducibility of the individual experiment. The analogy I have given is with cold fusion. If we could reproduce their exact lab setup then we could find out if the results were real - i.e. were not misread or anything, and assuming they were, have a chance of explaining the anomaly. But no, it's not the same as reproducibility.