3 ms·

>Reproducible Research: I think of Docker as akin to setting the random number seed in a report. The same dependencies and versions of libraries that was used i

>Reproducible Research: I think of Docker as akin to setting the random number seed in a report. The same dependencies and versions of libraries that was used in your machine are used on the other person’s machine. If you are lucky! Docker makes no guarantees about this. I'm betting most data science dockerfiles start with "RUN apt-get update && apt-get install -y -q \ && ... ". How much do you stick in numerous run commands following that? Docker images are not deterministic by default.

yjftsjthsd-h 9y ago
Follow up: "Nix for Data Science"?
jake-low 9y ago
IMO one thing the author could have improved about the article is to make a more clear distinction between a Dockerfile and a docker image. What you say is true about builds in docker: using the `docker build` command to create an image from a Dockerfile isn't necessarily deterministic, for the reason you stated. However, a docker image is deterministic (each one gets a SHA256 identifier based on its contents -- if the hashes match, then the images are identical), and if you care about the particular versions of dependencies, an image is what you should be sharing with your colleagues/readers/etc. in order to let them reproduce your results. I like Docker and I get a lot of value from using it, but personally I feel this is one place where the project made an error in design. Too many git repos these days have a Dockerfile in them that you can use to build the code, but that will only work if the stars align and the dependencies you install when you build your image are the same (or compatible with) the ones the author used. IMO a better design would be for docker images to be flat files e.g. "docker build Dockerfile > my-cool-image.img" and then "docker run my-cool-image.img ...". I think if this were the pattern, more people would be adding their `my-cool-image.img` files to their git repos instead of using the Dockerfile as a flawed source of truth.

yjftsjthsd-h 9y ago

Follow up: "Nix for Data Science"?

jake-low 9y ago

IMO one thing the author could have improved about the article is to make a more clear distinction between a Dockerfile and a docker image. What you say is true about builds in docker: using the `docker build` command to create an image from a Dockerfile isn't necessarily deterministic, for the reason you stated. However, a docker image is deterministic (each one gets a SHA256 identifier based on its contents -- if the hashes match, then the images are identical), and if you care about the particular versions of dependencies, an image is what you should be sharing with your colleagues/readers/etc. in order to let them reproduce your results. I like Docker and I get a lot of value from using it, but personally I feel this is one place where the project made an error in design. Too many git repos these days have a Dockerfile in them that you can use to build the code, but that will only work if the stars align and the dependencies you install when you build your image are the same (or compatible with) the ones the author used. IMO a better design would be for docker images to be flat files e.g. "docker build Dockerfile > my-cool-image.img" and then "docker run my-cool-image.img ...". I think if this were the pattern, more people would be adding their `my-cool-image.img` files to their git repos instead of using the Dockerfile as a flawed source of truth.