4 ms·

> If you run docker build twice with the same Dockerfile you might get 2 images that behave in different ways. For example, a third-party package could silently

> If you run docker build twice with the same Dockerfile you might get 2 images that behave in different ways. For example, a third-party package could silently be updated and cause breakage. Aggressively pinning dependency versions helps, but doesn't completely prevent this issue. If "aggressive" means fully, then why doesn't that fix the issue?

Volundr 5y ago
A lot of package repositories prevent this now, but theoretically your package repository could allow a new version of the package to be uploaded with the same version number. Docker tags have this issue in fact, so even your base image could change underneath you.
- throwaway9870 5y ago
  I can't imagine using a package system for building images that doesn't allow a full local cache of exactly the packages I want installed.
  - Volundr 5y ago
    I mean sure, you can absolutely retain your own docker repository, your own NPM repository, etc and then configure them in your Dockerfile before installing packages. Pretty much every technical problem has a technical solution, but it's more effort that "just" fully pinning your versions.
    - throwaway9870 5y ago
      If there are too many packages to pin, then I assume there are too many package to audit changes in for each image build you do? If that is true, how do you have any confidence that the image is going to function correct?
      - Volundr 5y ago
        I think we're talking about two different things. I'm thinking of pinning packages as say, my npm.lock specifies I will use version 2.2.19 of tailwind css. Or my Dockerfile has redis:6.2.6 as it's base image. Both of those are fully pinned, but nothing stops Tailwind or Redis from uploading new versions of the package and reusing the version number (I believe NPM actually disallows this, Docker hub does not). So my build is not technically reproducible. I can solve this by maintaining my own Docker registry, and my own NPM registry, but that's more work that just fully specifying versions in a configuration file. As for auditing changes, well most people don't. They update their dependencies and run their tests and if the tests pass, ship it! Most companies don't have a team auditing every single dependency that's pulled in, much less every update after a dependency has been approved. They simply trust the authors of Redis not to screw them. It's great if you do, I'm certainly not arguing against it, but it's far from the norm.
        etcet 5y ago
        You can reference docker images by their SHA digest which is immutable, e.g.: docker pull redis@sha256:619af14d3a95c30759a1978da1b2ce375504f1af70ff9eea2a8e35febc45d747
lilyball 5y ago
Probably because of non-deterministic builds. Especially if the dependency pulls info from the network at build time.
- throwaway9870 5y ago
  I guess that would not fit my definition of "fully". Do people really build images this way? It sounds completely insane to pull packages like that randomly from the Internet.
  - Volundr 5y ago
    Both my professional experience, and public examples would seem to suggest that's the norm. Example, here's the official postgres Dockerfile: https://github.com/docker-library/postgres/blob/3bb48045b4dc5df24bf2271c679f7a4e9efcbe6e/14/bullseye/Dockerfile https://github.com/docker-library/postgres/blob/3bb48045b4dc.... Do you work in an environment that maintains custom copies of every dependency in company managed repos? If so, my experience suggests your the outlier, not the people running apt, npm, etc inside their Dockerfiles.
    - throwaway9870 5y ago
      Not custom copies, but a locked down cache of packages. For Gentoo you can do this by locking the portage tree you use and keeping a copy of the distfiles from the first run, for Python it was a requirements.txt file with a cache of the tar files from PyPi, for go it was including 3rd party code in repo. I don't know what the team did for npm. It was really nice doing a full image rebuild and knowing the only thing that changed it was you explicitly changed.
      - Volundr 5y ago
        I'm genuinely curious about this. How are you distributing these caches so that if I do a build on my machine it'll produce the exact same image as on yours? If I'm understanding what you mean by "cache" (I'm thinking the node_modules folder for example for NPM) it'd certainly work, but it feels like a logistical nightmare to me.
cbrewster 5y ago
Author here. In our case, we had a large base Docker image called Polygott (https://github.com/replit/polygott https://github.com/replit/polygott) it pulls in dependencies for 50+ different languages from various repositories. We would pin things where possible, but its still very difficult to ensure reproducible builds. Additionally, docker builds have free access to the network to do anything it would like. Nix goes to great lengths to sandbox builds and limit network access. Anything accessed from a network requires a pinned sha 256 hash to ensure the remote data hasn't changed. (https://nixos.wiki/wiki/Nix#Sandboxing https://nixos.wiki/wiki/Nix#Sandboxing)
- throwaway9870 5y ago
  That makes sense. I think the real issue isn't Docker vs Nix, it is that some package managers are almost impossible to use to build reproducible images. I worked with debootstrap 10+ years ago trying to script reproducible builds and found it exceeding hard. Gentoo made it almost trivial (Google used it for ChromeOS so perhaps they felt similar). I will look into Nix. It appears that with the proper package manager support, Docker would be fine? I come from a hardware background and seem to be a lot more paranoid than most software folks. I would struggle to trust a build where so much is not pinned.
- treis 5y ago
  Why do you need reproducible builds for Docker? The whole point is that you build it once and then you use that container on as many servers as you want.
  - cbrewster 5y ago
    What happens when you need to update some dependency within that image? Now you have to do an image rebuild. If you're lucky only the top-most layers will be rebuilt and the base layers stay the same, if you're unlucky nearly the whole image is rebuilt. Usually we just want to update some subset of dependencies, but during the rebuild some other dependencies may get updated unintentionally (eg if they aren't pinned to a particular version). For most, this may not be an issue but at Replit, everyone's projects use this base Docker image. Unintended updates can cause breakage for our users.

Volundr 5y ago

A lot of package repositories prevent this now, but theoretically your package repository could allow a new version of the package to be uploaded with the same version number. Docker tags have this issue in fact, so even your base image could change underneath you.

throwaway9870 5y ago
I can't imagine using a package system for building images that doesn't allow a full local cache of exactly the packages I want installed.
- Volundr 5y ago
  I mean sure, you can absolutely retain your own docker repository, your own NPM repository, etc and then configure them in your Dockerfile before installing packages. Pretty much every technical problem has a technical solution, but it's more effort that "just" fully pinning your versions.
  - throwaway9870 5y ago
    If there are too many packages to pin, then I assume there are too many package to audit changes in for each image build you do? If that is true, how do you have any confidence that the image is going to function correct?
    - Volundr 5y ago
      I think we're talking about two different things. I'm thinking of pinning packages as say, my npm.lock specifies I will use version 2.2.19 of tailwind css. Or my Dockerfile has redis:6.2.6 as it's base image. Both of those are fully pinned, but nothing stops Tailwind or Redis from uploading new versions of the package and reusing the version number (I believe NPM actually disallows this, Docker hub does not). So my build is not technically reproducible. I can solve this by maintaining my own Docker registry, and my own NPM registry, but that's more work that just fully specifying versions in a configuration file. As for auditing changes, well most people don't. They update their dependencies and run their tests and if the tests pass, ship it! Most companies don't have a team auditing every single dependency that's pulled in, much less every update after a dependency has been approved. They simply trust the authors of Redis not to screw them. It's great if you do, I'm certainly not arguing against it, but it's far from the norm.
      - etcet 5y ago
        You can reference docker images by their SHA digest which is immutable, e.g.: docker pull redis@sha256:619af14d3a95c30759a1978da1b2ce375504f1af70ff9eea2a8e35febc45d747

lilyball 5y ago

Probably because of non-deterministic builds. Especially if the dependency pulls info from the network at build time.

throwaway9870 5y ago
I guess that would not fit my definition of "fully". Do people really build images this way? It sounds completely insane to pull packages like that randomly from the Internet.
- Volundr 5y ago
  Both my professional experience, and public examples would seem to suggest that's the norm. Example, here's the official postgres Dockerfile: https://github.com/docker-library/postgres/blob/3bb48045b4dc5df24bf2271c679f7a4e9efcbe6e/14/bullseye/Dockerfile https://github.com/docker-library/postgres/blob/3bb48045b4dc.... Do you work in an environment that maintains custom copies of every dependency in company managed repos? If so, my experience suggests your the outlier, not the people running apt, npm, etc inside their Dockerfiles.
  - throwaway9870 5y ago
    Not custom copies, but a locked down cache of packages. For Gentoo you can do this by locking the portage tree you use and keeping a copy of the distfiles from the first run, for Python it was a requirements.txt file with a cache of the tar files from PyPi, for go it was including 3rd party code in repo. I don't know what the team did for npm. It was really nice doing a full image rebuild and knowing the only thing that changed it was you explicitly changed.
    - Volundr 5y ago
      I'm genuinely curious about this. How are you distributing these caches so that if I do a build on my machine it'll produce the exact same image as on yours? If I'm understanding what you mean by "cache" (I'm thinking the node_modules folder for example for NPM) it'd certainly work, but it feels like a logistical nightmare to me.

cbrewster 5y ago

Author here. In our case, we had a large base Docker image called Polygott (https://github.com/replit/polygott https://github.com/replit/polygott) it pulls in dependencies for 50+ different languages from various repositories. We would pin things where possible, but its still very difficult to ensure reproducible builds. Additionally, docker builds have free access to the network to do anything it would like. Nix goes to great lengths to sandbox builds and limit network access. Anything accessed from a network requires a pinned sha 256 hash to ensure the remote data hasn't changed. (https://nixos.wiki/wiki/Nix#Sandboxing https://nixos.wiki/wiki/Nix#Sandboxing)

throwaway9870 5y ago
That makes sense. I think the real issue isn't Docker vs Nix, it is that some package managers are almost impossible to use to build reproducible images. I worked with debootstrap 10+ years ago trying to script reproducible builds and found it exceeding hard. Gentoo made it almost trivial (Google used it for ChromeOS so perhaps they felt similar). I will look into Nix. It appears that with the proper package manager support, Docker would be fine? I come from a hardware background and seem to be a lot more paranoid than most software folks. I would struggle to trust a build where so much is not pinned.
treis 5y ago
Why do you need reproducible builds for Docker? The whole point is that you build it once and then you use that container on as many servers as you want.
- cbrewster 5y ago
  What happens when you need to update some dependency within that image? Now you have to do an image rebuild. If you're lucky only the top-most layers will be rebuilt and the base layers stay the same, if you're unlucky nearly the whole image is rebuilt. Usually we just want to update some subset of dependencies, but during the rebuild some other dependencies may get updated unintentionally (eg if they aren't pinned to a particular version). For most, this may not be an issue but at Replit, everyone's projects use this base Docker image. Unintended updates can cause breakage for our users.