6 ms·

You could make a container with the pipeline and data set up to go. Also, ideally, you'd publicly host the Dockerfile or w/e build time config, too.

ivan888 6y ago
Downvoters: explain? Maybe this suggestion is aimed too much at a specific piece of technology (Docker). But there are plenty of tools out there to create reproducible builds, which I think is the overall point of this response
- afarrell 6y ago
  Speculation: many downvoters are people who have repeatedly had a frustrating user experiences and are just exasperated at being repeatedly told how thier tools are good enough.
- TheGallopedHigh 6y ago
  There is so much on the plate of the academic that to learn yet another piece of software — like docker which can be trying at the best of times — is yet another straw that can break the back.
- mattkrause 6y ago
  I didn't downvote but... I'm skeptical that specific tools and "formalities" like requiring Docker containers will do a lot to improve the quality of scientific research. Obviously, being able to re-run an old analysis is better than not being able to do so but that wouldn't have helped (much) with this issue The root cause here was a conceptual problem, not a library version mismatch. They used the same data to a) group data points into bins and b) test for differences between those bins. Having a perfectly-engineered, smoothly-running pipeline might have made it easier to confirm the result, but I don't think it would help detect it. The things that would have things like rewarding collaboration (more eyes make bugs shallower) instead of rewarding mostly lead authorship, time and support to carefully mull over findings instead of a helter-skelter rush to publish, and a publication culture that doesn't avoid ambiguity. Those are much harder to achieve though....
- CJefferson 6y ago
  Because, as far as I can tell, docker doesn't make easy reproducible images. How do I make a docker file which will create the same image, bit for bit (at least version for version) in 2 years time? I find usually it's required to update packages, which is good for security, bad for not breaking things.
skissane 6y ago
> Also, ideally, you'd publicly host the Dockerfile Docker doesn't make it easy to produce a reproducible config. People commonly write Dockerfiles with stuff like this in them: RUN apt-get install -y my-favourite-package And then which version of my-favourite-package you get depends on what is the latest one on the update servers at a time. Maybe a new version is released tomorrow with a regression bug that breaks everything. There are ways to get around this problem, but the default ways of doing things in Docker doesn't encourage reproducibility.
- sfblah 6y ago
  The solution is to put the actual package into the Dockerfile instead of the reference? Asking because I don't actually know the answer.
  - GauntletWizard 6y ago
    The solution is to publish the created docker container for examination and reuse in recreating the results. It is possible to create reproducible docker containers, but not with docker.
  - yoz-y 6y ago
    In one company I worked we hosted our own repo for npm and mirrored all packages we used there. For our own stuff we stored all old versions in it too. The software itself always installed specific version rather than "latest" Another option is to store the whole docker images, granted this will not give you the ability to always rebuild it, but at least you have a "binary" you will be able to run later.
- dmart 6y ago
  This is true, although the built Docker image itself (rather than Dockerfile) wouldn’t have these problems. I think in situations where reproducibility is paramount, it would make sense to offer a tarball of the image (`docker save`) along with the Dockerfile itself.
- acd10j 6y ago
  After making Dockerfile you can additionally push your built docker image to docker hub or other public facing repo, People can then directly pull your docker image in case any issue with Dockerfile.
iujjkfjdkkdkf 6y ago
I look at this whole idea another way: if your results are so fragile that they need a docker container (with all the caveats others are mentioning about versions etc), it's not really "reproducible" just because someone can re-run your code. Real reproducibility would come from having discovered some objective fact that shows itself again when similar work is attempted. Reproducibility =/= generality which is a much more important aspect of research.
- one-punch 6y ago
  While I agree that science should aim for replicability (same result for similar methods on the same scientific question) instead of reproducibility (same methods) [1], it does not hurt to let others to reuse part of your research pipeline. I think we can in fact test and improve replicability if the whole pipeline is published. We should aim for allowing others to change certain parameters or data in the pipeline. That is, the result should be robust with respect to minor changes in the pipeline, which could be more easily tested by others with a published, fully specified pipeline. Referring to my other comment [2], Guix or Nix really shine in allowing others to change part of the pipelines, due to their functional approach to composing components. [1]: https://phys.org/news/2019-05-replicability-science.html https://phys.org/news/2019-05-replicability-science.html [2]: https://news.ycombinator.com/item?id=25866651 https://news.ycombinator.com/item?id=25866651

ivan888 6y ago

Downvoters: explain? Maybe this suggestion is aimed too much at a specific piece of technology (Docker). But there are plenty of tools out there to create reproducible builds, which I think is the overall point of this response

afarrell 6y ago
Speculation: many downvoters are people who have repeatedly had a frustrating user experiences and are just exasperated at being repeatedly told how thier tools are good enough.
TheGallopedHigh 6y ago
There is so much on the plate of the academic that to learn yet another piece of software — like docker which can be trying at the best of times — is yet another straw that can break the back.
mattkrause 6y ago
I didn't downvote but... I'm skeptical that specific tools and "formalities" like requiring Docker containers will do a lot to improve the quality of scientific research. Obviously, being able to re-run an old analysis is better than not being able to do so but that wouldn't have helped (much) with this issue The root cause here was a conceptual problem, not a library version mismatch. They used the same data to a) group data points into bins and b) test for differences between those bins. Having a perfectly-engineered, smoothly-running pipeline might have made it easier to confirm the result, but I don't think it would help detect it. The things that would have things like rewarding collaboration (more eyes make bugs shallower) instead of rewarding mostly lead authorship, time and support to carefully mull over findings instead of a helter-skelter rush to publish, and a publication culture that doesn't avoid ambiguity. Those are much harder to achieve though....
CJefferson 6y ago
Because, as far as I can tell, docker doesn't make easy reproducible images. How do I make a docker file which will create the same image, bit for bit (at least version for version) in 2 years time? I find usually it's required to update packages, which is good for security, bad for not breaking things.

skissane 6y ago

> Also, ideally, you'd publicly host the Dockerfile Docker doesn't make it easy to produce a reproducible config. People commonly write Dockerfiles with stuff like this in them: RUN apt-get install -y my-favourite-package And then which version of my-favourite-package you get depends on what is the latest one on the update servers at a time. Maybe a new version is released tomorrow with a regression bug that breaks everything. There are ways to get around this problem, but the default ways of doing things in Docker doesn't encourage reproducibility.

sfblah 6y ago
The solution is to put the actual package into the Dockerfile instead of the reference? Asking because I don't actually know the answer.
- GauntletWizard 6y ago
  The solution is to publish the created docker container for examination and reuse in recreating the results. It is possible to create reproducible docker containers, but not with docker.
- yoz-y 6y ago
  In one company I worked we hosted our own repo for npm and mirrored all packages we used there. For our own stuff we stored all old versions in it too. The software itself always installed specific version rather than "latest" Another option is to store the whole docker images, granted this will not give you the ability to always rebuild it, but at least you have a "binary" you will be able to run later.
dmart 6y ago
This is true, although the built Docker image itself (rather than Dockerfile) wouldn’t have these problems. I think in situations where reproducibility is paramount, it would make sense to offer a tarball of the image (`docker save`) along with the Dockerfile itself.
acd10j 6y ago
After making Dockerfile you can additionally push your built docker image to docker hub or other public facing repo, People can then directly pull your docker image in case any issue with Dockerfile.

iujjkfjdkkdkf 6y ago

I look at this whole idea another way: if your results are so fragile that they need a docker container (with all the caveats others are mentioning about versions etc), it's not really "reproducible" just because someone can re-run your code. Real reproducibility would come from having discovered some objective fact that shows itself again when similar work is attempted. Reproducibility =/= generality which is a much more important aspect of research.

one-punch 6y ago
While I agree that science should aim for replicability (same result for similar methods on the same scientific question) instead of reproducibility (same methods) [1], it does not hurt to let others to reuse part of your research pipeline. I think we can in fact test and improve replicability if the whole pipeline is published. We should aim for allowing others to change certain parameters or data in the pipeline. That is, the result should be robust with respect to minor changes in the pipeline, which could be more easily tested by others with a published, fully specified pipeline. Referring to my other comment [2], Guix or Nix really shine in allowing others to change part of the pipelines, due to their functional approach to composing components. [1]: https://phys.org/news/2019-05-replicability-science.html https://phys.org/news/2019-05-replicability-science.html [2]: https://news.ycombinator.com/item?id=25866651 https://news.ycombinator.com/item?id=25866651