3 ms·

> Why on earth would anyone trust descriptions that they cannot verify? > > Trusting without the ability to verify goes against everything scientific. Hasn't

> Why on earth would anyone trust descriptions that they cannot verify? > > Trusting without the ability to verify goes against everything scientific. Hasn't this always been true about scientific papers? Descriptions can be verified by reproducing the experiment. Why is a paper any less trustworthy just because there's code involved?

beagle3 14y ago
> Hasn't this always been true about scientific papers? Descriptions can be verified by reproducing the experiment. Why is a paper any less trustworthy just because there's code involved? It was always true to an extent. Code is a force multiplier that makes it significantly harder to evaluate the paper with out it (and without reproducing an equivalent). I'm not in academia myself, but I've heard from friends more than once that when they actually received code (and/or data) they requested from an author, the code turned out to be not precisely described in the paper, and the data is often massaged to fit in a way that's not precisely described either. The question shouldn't be "why aren't you satisfied with what was good 20 years ago?", but rather "when sharing the bits that makes everything reproducible is a 'git push' away, why isn't it considered mandatory?" It is a common error that science is about proving things; The scientific method is actually about trying to disprove things and failing to do so. If what you want to do is science, why don't you make it easiest to disprove your results?
jerf 14y ago
The need for reproducibility in experiments is an accident of the fact that our universe is horrifically complicated and true reproducibility is a myth, thus we must make a deliberate, conscious effort to come as close as possible, or no progress can be made. When that is no longer true and it becomes possible to run (under certain constrained circumstances) fully deterministic experiments that can be freely replicated to the bit by anybody, it's time to rethink the assumptions made lo these many centuries ago. People arguing against source code release often argue as if those of us in favor think that re-running the original simulation is the end-all, be-all of reproducibility. Clearly that is not the case. No one simulation can truly prove anything, and independent reverification will always have a place. But since we do have the source artifacts and original data, why not release them and show exactly what was done and how it was done? Again, the idea that experiments should not do so is merely an artifact of the fact that scientific papers could only be 10 very expensive pages or so in a journal; why carry unexamined assumptions based on that now outdated fact forward into the future? Accidents of the past are nothing more than accidents of the past, not holy writ. And I'm not aware of a good argument against release of source code that doesn't boil down to well, that's just not how we do it when deeply examined.

beagle3 14y ago

> Hasn't this always been true about scientific papers? Descriptions can be verified by reproducing the experiment. Why is a paper any less trustworthy just because there's code involved? It was always true to an extent. Code is a force multiplier that makes it significantly harder to evaluate the paper with out it (and without reproducing an equivalent). I'm not in academia myself, but I've heard from friends more than once that when they actually received code (and/or data) they requested from an author, the code turned out to be not precisely described in the paper, and the data is often massaged to fit in a way that's not precisely described either. The question shouldn't be "why aren't you satisfied with what was good 20 years ago?", but rather "when sharing the bits that makes everything reproducible is a 'git push' away, why isn't it considered mandatory?" It is a common error that science is about proving things; The scientific method is actually about trying to disprove things and failing to do so. If what you want to do is science, why don't you make it easiest to disprove your results?

jerf 14y ago

The need for reproducibility in experiments is an accident of the fact that our universe is horrifically complicated and true reproducibility is a myth, thus we must make a deliberate, conscious effort to come as close as possible, or no progress can be made. When that is no longer true and it becomes possible to run (under certain constrained circumstances) fully deterministic experiments that can be freely replicated to the bit by anybody, it's time to rethink the assumptions made lo these many centuries ago. People arguing against source code release often argue as if those of us in favor think that re-running the original simulation is the end-all, be-all of reproducibility. Clearly that is not the case. No one simulation can truly prove anything, and independent reverification will always have a place. But since we do have the source artifacts and original data, why not release them and show exactly what was done and how it was done? Again, the idea that experiments should not do so is merely an artifact of the fact that scientific papers could only be 10 very expensive pages or so in a journal; why carry unexamined assumptions based on that now outdated fact forward into the future? Accidents of the past are nothing more than accidents of the past, not holy writ. And I'm not aware of a good argument against release of source code that doesn't boil down to well, that's just not how we do it when deeply examined.