3 ms·

I fully agree with the principle that experiments should be reproducible. In systems research conferences (SOSP, Eurosys, OSDI) very few of the systems publish

I fully agree with the principle that experiments should be reproducible. In systems research conferences (SOSP, Eurosys, OSDI) very few of the systems publish their algorithms. The worst culprits are the companies, such as Google, Facebook, and Microsoft, who don't even publish all of their algorithms or systems properties. Google have published significant papers that leave out the key algorithms on how parts of their systems work (e.g., In Borg how does the scheduler synchronize the cluster state with the replicas? In Spanner, what are the properties of the underlying storage system, Collosus?). I would love to see all distributed systems papers be able to reproduce everything by just downloading a file that allows the paper's results to be reproduced. We can do it with the help of either virtualization or containers and software configuration frameworks (to parameterize the experiments). We can specify the hardware and network programmatically, we can install the software automatically, we can parameterize system software (with Chef or Puppet attributes), and run the experiments reproducibly. Without huge investment and cloud computing or Docker, we could specify systems that have reproducible hardware/network/software. The first attempt I've seen at this is www.karamel.io that allows you to design reproducible experiments by using JClouds to spawn VMs and setup the virtualized hardware, then orhcestrating Chef to install software that can be parameterized. I hope to see more of this line of platform gain adoption in our community. It would be a boon for research.

dekhn 11y ago
Lots of people have proposed and implemented VM-based reproducibility environments. 1) it's actually harder to get reproducible computational experiments than they expected. For example, you can run same VM on a different processor and get different results, which makes bitwise reproduction hard, and statistical tests for nonbitwise equality are harder 2) developing and maintaining the VMs and the environment takes a fair amount of effort from skilled people 3) the resulting improvements to science don't seem to exceed the cost thresholds implied by #1 and #2, and nobody's volunteering their time. My conclusion: good idea, but probably not critically necessary.
- jamesblonde 11y ago
  That's good feedback. Although I'm considering mostly distributed systems, where full reproducability is not possible. For single-threaded, deterministic programs (discrete event simulations) where performance is not a measurement point, you can do it with VMs. The nicest thing would be to be able to parameterize experiments. Not just re-run them, but change the parameters for different runs.

dekhn 11y ago

Lots of people have proposed and implemented VM-based reproducibility environments. 1) it's actually harder to get reproducible computational experiments than they expected. For example, you can run same VM on a different processor and get different results, which makes bitwise reproduction hard, and statistical tests for nonbitwise equality are harder 2) developing and maintaining the VMs and the environment takes a fair amount of effort from skilled people 3) the resulting improvements to science don't seem to exceed the cost thresholds implied by #1 and #2, and nobody's volunteering their time. My conclusion: good idea, but probably not critically necessary.

jamesblonde 11y ago
That's good feedback. Although I'm considering mostly distributed systems, where full reproducability is not possible. For single-threaded, deterministic programs (discrete event simulations) where performance is not a measurement point, you can do it with VMs. The nicest thing would be to be able to parameterize experiments. Not just re-run them, but change the parameters for different runs.