> Third-parties validating that the code underlying an academic paper executes as expected is no different than third-parties replicating the results of a physi

> Third-parties validating that the code underlying an academic paper executes as expected is no different than third-parties replicating the results of a physical experiment. First, it's not no different--it's completely different. Third parties have always constructed their own apparatus to reproduce an experiment. They don't go to the original author's lab to perform the experiment! Second, a lot of scientific code won't run at all outside the environment it was developed in. If it's HPC code, it's very likely that the code makes assumptions about the HPC cluster that will cause it to break on a different cluster. If it's experiment control / data-acquisition code, you'll almost certainly need the exact same peripherals for the program to do anything at all sensible. I see a lot of people here on HN vastly over-estimating the value of bit-for-bit reproducibility of one implementation, and vastly underestimating the value of having a diversity of implementations to test an idea.