4 ms·

Not in general, no. Docker image layers are just snapshots of the filesystem changes that result from build commands. But there is nothing that guarantees that

Not in general, no. Docker image layers are just snapshots of the filesystem changes that result from build commands. But there is nothing that guarantees that the effects of those commands are reproducible. For example, it's incredibly common for Dockerfiles to contain commands like: RUN apt-get update && apt-get install -y foobar which means when you build the container, you get whatever version of foobar happens to currently be in the repository that your base image points to. So you can easily end up in a situation where rebuilding an image gives you different behavior, even though nothing in your Dockerfile or build context changed.

hinkley 3y ago
Even the Docker support team is confused by this. Most of why I stopped engaging is that they would reject a feature because it lead to dockerfiles they said were 'not reproducible', but were just as reproducible as many of the core features. Which is to say, not in the slightest. Every time you do something in a Dockerfile that involves pulling something from the network, you don't have reproducibility. Unless you pull something using a cryptographic hash (which I've never actually seen anyone do), the network can give you a different answer every time. The answer might not even be idempotent (calling it once changes the result of calling it again). My argument was the same as yours. Virtually every dockerfile has an 'update' call as the second or third layer, which means none of the repeatable layers afterward (like useradd or mkdir) are actually repeatable. You're building a castle on the sand. Docker images are reproducible. Docker builds are not. And that's okay, as long as everyone accepts that to be true. And if Docker contributors can't accept that then we are all truly fucked, until something else comes along that retreads the ideas.
- xvector 3y ago
  Sounds like Nix is what you're looking for.
  - evilduck 3y ago
    Which can conveniently produce Docker images.
    - Quekid5 3y ago
      Is there some documentation on exactly how to do this? I've been curious about this in the past, but my search-fu is failing me...
      - arianvanp 3y ago
        There's a whole chapter on images (VM, docker, appimage) in the manual. I can recommend starting there. https://nixos.org/manual/nixpkgs/stable/#chap-images https://nixos.org/manual/nixpkgs/stable/#chap-images The smallest example is: pkgs.dockerTools.buildLayeredImage { name = "hello"; contents = [ pkgs.hello ]; }
- MichaelZuo 3y ago
  Is there really a significant number of folks who are confused by the idea that networks can serve different files under the same name at different times?
coding123 3y ago
Yes but two things about this: 1) you can apt-get a specific version 2) even if you don't apt-get a specific version, if two developers build and then run the container (within some time frame), they will get the same version. It would be rare to get a different one on the same day. And 3) You can still apt-get a specific version And 4) That's still far and a way greater chance two developers will have the same version when using docker in this way than if they're both using two crazy different versions of Ubuntu locally on their desktop. Also most developers that are working on the same codebase will push to their branch and circleCI will make it so there's only one docker image, that's tested and moves along to prod eventually.
- imp0cat 3y ago
  You can apt-get a specific version, but that is not a 100% guarantee that the contents of the package will be identical to what was there yesterday. It is absolutely possible to delete a package from repo and replace it with another one with same version.
  - pxc 3y ago
    Distros don't do this and have communally enforced conventions for package revisions that don't involve updates/changes to the upstream source. Proprietary software vendors and in-house corporate repos might pull sloppy crap here, though. It is nice when your package manager reliably alerts you to changes like that, though! Binary package managers can't, really— although this feature could be added— but sourced-based ones often do. Nix and Guix enforce this categorically, including for upstream source tarballs for source builds. Even Homebrew includes hashes for its binaries, and provides options to enforce them (refusing to install packages whose recipes don't include hashes).
- ParetoOptimal 3y ago
  > even if you don't apt-get a specific version, if two developers build and then run the container (within some time frame) All of these "maybe it'll work" type ideas are what make modern software so brittle.
Izkata 3y ago
> which means when you build the container, you get whatever version of foobar happens to currently be in the repository that your base image points to. Well no, the layer won't be rebuilt if you already have it from yesterday or last week and that + previous lines in the Dockerfile didn't change. So yours would still be using the old version, but someone new to the project would get the most up-to-date version. Have to remember to skip the cache to get the most recent version.
- coldtea 3y ago
  >Well no, the layer won't be rebuilt if you already have it from yesterday or last week and that + previous lines in the Dockerfile didn't change. When people say it needs to be reproducible, they don't mean "reproducible in the same host machine, with the same cache lying around".
a_bonobo 3y ago
It's why I like conda's yml files, every dependency listed with its version number and from which conda-channel it came from
- smabie 3y ago
  you can omit version numbers in Conda files as well tho
  - a_bonobo 3y ago
    yes true! But by default, `conda env export > env.yml` will include the version numbers of the current environment, which is how I usually use it.

hinkley 3y ago

Even the Docker support team is confused by this. Most of why I stopped engaging is that they would reject a feature because it lead to dockerfiles they said were 'not reproducible', but were just as reproducible as many of the core features. Which is to say, not in the slightest. Every time you do something in a Dockerfile that involves pulling something from the network, you don't have reproducibility. Unless you pull something using a cryptographic hash (which I've never actually seen anyone do), the network can give you a different answer every time. The answer might not even be idempotent (calling it once changes the result of calling it again). My argument was the same as yours. Virtually every dockerfile has an 'update' call as the second or third layer, which means none of the repeatable layers afterward (like useradd or mkdir) are actually repeatable. You're building a castle on the sand. Docker images are reproducible. Docker builds are not. And that's okay, as long as everyone accepts that to be true. And if Docker contributors can't accept that then we are all truly fucked, until something else comes along that retreads the ideas.

xvector 3y ago
Sounds like Nix is what you're looking for.
- evilduck 3y ago
  Which can conveniently produce Docker images.
  - Quekid5 3y ago
    Is there some documentation on exactly how to do this? I've been curious about this in the past, but my search-fu is failing me...
    - arianvanp 3y ago
      There's a whole chapter on images (VM, docker, appimage) in the manual. I can recommend starting there. https://nixos.org/manual/nixpkgs/stable/#chap-images https://nixos.org/manual/nixpkgs/stable/#chap-images The smallest example is: pkgs.dockerTools.buildLayeredImage { name = "hello"; contents = [ pkgs.hello ]; }
MichaelZuo 3y ago
Is there really a significant number of folks who are confused by the idea that networks can serve different files under the same name at different times?

coding123 3y ago

Yes but two things about this: 1) you can apt-get a specific version 2) even if you don't apt-get a specific version, if two developers build and then run the container (within some time frame), they will get the same version. It would be rare to get a different one on the same day. And 3) You can still apt-get a specific version And 4) That's still far and a way greater chance two developers will have the same version when using docker in this way than if they're both using two crazy different versions of Ubuntu locally on their desktop. Also most developers that are working on the same codebase will push to their branch and circleCI will make it so there's only one docker image, that's tested and moves along to prod eventually.

imp0cat 3y ago
You can apt-get a specific version, but that is not a 100% guarantee that the contents of the package will be identical to what was there yesterday. It is absolutely possible to delete a package from repo and replace it with another one with same version.
- pxc 3y ago
  Distros don't do this and have communally enforced conventions for package revisions that don't involve updates/changes to the upstream source. Proprietary software vendors and in-house corporate repos might pull sloppy crap here, though. It is nice when your package manager reliably alerts you to changes like that, though! Binary package managers can't, really— although this feature could be added— but sourced-based ones often do. Nix and Guix enforce this categorically, including for upstream source tarballs for source builds. Even Homebrew includes hashes for its binaries, and provides options to enforce them (refusing to install packages whose recipes don't include hashes).
ParetoOptimal 3y ago
> even if you don't apt-get a specific version, if two developers build and then run the container (within some time frame) All of these "maybe it'll work" type ideas are what make modern software so brittle.

Izkata 3y ago

> which means when you build the container, you get whatever version of foobar happens to currently be in the repository that your base image points to. Well no, the layer won't be rebuilt if you already have it from yesterday or last week and that + previous lines in the Dockerfile didn't change. So yours would still be using the old version, but someone new to the project would get the most up-to-date version. Have to remember to skip the cache to get the most recent version.

coldtea 3y ago
>Well no, the layer won't be rebuilt if you already have it from yesterday or last week and that + previous lines in the Dockerfile didn't change. When people say it needs to be reproducible, they don't mean "reproducible in the same host machine, with the same cache lying around".

a_bonobo 3y ago

It's why I like conda's yml files, every dependency listed with its version number and from which conda-channel it came from

smabie 3y ago
you can omit version numbers in Conda files as well tho
- a_bonobo 3y ago
  yes true! But by default, `conda env export > env.yml` will include the version numbers of the current environment, which is how I usually use it.