3 ms·
This article harkens back to discussions of the "reproducibility crisis" in science, as discussed extensively here just recently (see link below). Where, in thi
by southern_cross 8y ago
This article harkens back to discussions of the "reproducibility crisis" in science, as discussed extensively here just recently (see link below). Where, in this case, not coughing up the code used in the simulations in a timely manner led to an apparently unnecessary multi-year dispute.
https://news.ycombinator.com/item?id=17702380 https://news.ycombinator.com/item?id=17702380
- BeetleB 8y agoI'm glad this is getting to the fore. Depending on which forum I'm in, I'm heavily downvoted when I talk about reproducibility problems in hard sciences - particularly physics. My thesis was in condensed matter, and (in those days), no one published code. There were a lot of pointless arguments like the one in this submission (although usually not as heated) - all because professors were against sharing their code. It was routine to hear researchers discuss whether they believed a particular paper. Why should belief be a factor? Simulation code contains a lot of numerical tricks and hacks - and the papers never discuss them.[1] So reproducing the results of a paper were mostly pointless, and the general practice was "Let's see if we can reproduce their findings using our technique". If you got somewhat similar results, you'd publish. If you didn't, you'd sometimes publish (remember, there's a bias against negative results). [1]In fact, one reviewer even insisted that those details be removed because the journal was about publishing interesting new findings in science - not interesting techniques to make the computation more amenable.