5 ms·
This is why publishing code and data, together, is so important. Irreproducibile results in computing have no justification in this day and age.
by worik 6y ago
This is why publishing code and data, together, is so important.
Irreproducibile results in computing have no justification in this day and age.
- StandardFuture 6y agoIrreproducibile results should have zero place in science or academia. Hell, it has no place in politics, the media, etc. "Irreproducibile results" defeats the purpose of proving that it is not bullshit or a farce.
- projektfu 6y agoIf you run their code on their data you haven't reproduced their result. You have merely copied it. If their code works on another very similar data set to produce a very similar result, then you have something. Or if you're trying to understand the technique, you have to write your code and see if it reproduces their result on their data.
- bordercases 6y ago> If you run their code on their data you haven't reproduced their result. You have merely copied it. That's like saying you can't learn from exercises in the book if the answers are in the back.
- canjobear 6y agoYou certainly don't learn anything by running `sh run.sh`.
- asdff 6y agoyou learn run.sh actually works
- bordercases 6y agoYou learn if it's deterministic or not under the data given as input and the operating environment.
- worik 6y agoWhat nonsense! Copying something is reproducing it. If it cannot be copied then it cannot be reproduced. If it can be copied then it can be investigated. I ran into exactly this problem with: White's Reality Check (WRC), described in White's (2000) Econometrica paper titled "A Reality Check for Data Snooping", The algorithms that were being tested were so poorly described that less than half could be reproduced. Once we got there for what we could we found two fundamental problems with "Whites Reality Check", one theoretical that is blindingly obvious once you see it and the other that they ignored trading costs. When we measured the levels of trading costs that would permit the algorithms to work it was clear they could not. That we could not deduce without reproducing the results. If we could have copied the original code it would have shaved about eight weeks from my research. BTW "Whites Reality Check" was at the time pone of the most cited paper in its, rather narrow, field.
- projektfu 6y agoBut running their code, presumably, would get their result on their data, unless it was a fraud or weird misconfiguration. We’re not talking about fraud, we’re talking about reproducibility supporting a theory. You suspected the result was wrong, so your interest was in debugging their code to contradict their result. That has value but also is not a reproduction of the study.
- fsflover 6y agoIf you agree that publicly funded code must be free, sign the petition: https://publiccode.eu https://publiccode.eu.
- Heliosmaster 6y agoThis is the main reason why we are building Nextjournal [0]. By using immutability of Datomic + Clojure the code (and the results) are immutable and permanent. [0] https://nextjournal.com https://nextjournal.com
- lambdatronics 6y agoIrreproducibility can result from irreplicability or many other reasons. Publishing the code & data helps rule out the trivial replicability failure: that the authors made an implementation error, so the result is not reproducible. Establishing replicability is essential, but that won't prevent all irreproducible results. Attempts to reproduce results under different conditions are still necessary for the functioning of the scientific process - but they are a waste of time if the original result is not replicable.