9 ms·

Slightly off-topic, it really irks me when people take about reproducible builds/environments while using docker, while having something along the lines of apt-

Slightly off-topic, it really irks me when people take about reproducible builds/environments while using docker, while having something along the lines of apt-get update/pip install in their dockerfile. Like that completely destroys the reproducibility of your container!

outside1234 6y ago
And by that you mean that the version of those deps aren't pinned?
- beagle3 6y ago
  You depend on what the repository serves for "apt-get". You may pin it, but it still doesn't guarantee you're going to get the same version if it was replaced on the repository without changing the version.
- dhruvrrp 6y ago
  At least for python, just pinning the top level dependencies are not enough. if you pin tensorflow==2.4.0, it doesn’t pin its required packages, rather it just defines a range. An example would be tf will try to get wheel>=0.26 (which was released in 2015 and is currently on release 0.36.2)
kiallmacinnes 6y ago
Yea - I get where you're coming from - you can't rebuild the container, but each team member can reuse the same container - which is at least a step towards being fully reproducible.
mumblemumble 6y ago
I can't speak for others, but, while we don't pin versions in the Dockerfiles for our build container images, we do version the images themselves. Reproducibility is a continuum, not a binary. We've chosen a point on that continuum that we believe gives us the best trade-off between reliability and maintenance effort. 100% from-the-ground-up reproducibility would be ideal, of course, but there's a cost-benefit tradeoff, and we're not being paid to be perfectionists.
hctaw 6y ago
I mean in a sense, the fucked up behavior of system package managers relative to the actual concerns of building and distributing software to the masses is the reason we need Docker to sandbox environments in the first place. It's 2021, there is next to no reason why the default behavior is installation to /usr/lib with shared objects that are rarely shared, with global access when installed for one application used by one user, and linked dynamically when the object never changes over the lifetime of the running application. For all its warts this is something that npm gets right. Package management is a tool for software development, not software distribution to end users.
- deleted 6y ago
  [deleted]
- moistbar 6y ago
  Both apt and yum support installing to whatever directory you like as long as you have write permissions there.
  - hctaw 6y ago
    "support" and "default" are very different things. Especially with package managers like homebrew, which comes with a loud warning that non-default (root) paths may not be supported by dependencies. It also doesn't really, since this is a system level issue. Applications need to package their dependencies, not the other way around. Dependencies form graphs, not flat lists. The existence of a global cache of libraries shared by all programs is a total inversion of requirements.
  - AnIdiotOnTheNet 6y ago
    Not really. They support dumping the package contents to an arbitrary directory, but you can't actually run the software from there without either the software having been written to support arbitrary paths (almost none is) or using namespacing and chroot to build it a sandbox wherein its baked-in paths actually work. Contrast that with something like RiscOS AppDirs, classic Mac applications, or Next/Mac Application Bundles.
JMTQp8lwXL 6y ago
I never understood why commands like apt-get don't take a version with the dependency name.
- astrea 6y ago
  They do. You can do `apt-get install package=version`
- wongarsu 6y ago
  As far as I'm aware it's supported (something like `apt-get install virtualbox=5.0.18-dfsg-2build1 -V`). It's just not commonly used because you usually just choose a distro with the desired update granularity (whether you want to the newest version out there or a consistent version with backported security fixes, or something in between).
amscanne 6y ago
The industry seems to have adopted “hermetic” as a word that describes truly reproducible builds, while “reproducible” has a lower standard. In many cases, it seems to be used to mean “not dependent on local build environment”.
umvi 6y ago
We don't rebuild image every time. We store images in GitLab registry. Also, just because we use apt-get to install python3 doesn't mean build isn't reproducible. The actual toolchain and sysroot are highly version controlled and python is just used to kick off the build script so it almost doesn't really matter which version python is as long as it's backwards compatible with the build script.
- physicsguy 6y ago
  We tried this in our research group, and found that issues come in when it gets to things like... Oh no! CMake is too old a version to support a dependency we have to build in the image construction. So we better pull in a version of CMake from a PPA which is community maintained, and build it from source/etc.
chrisjc 6y ago
Depends on how you do it, right? For instance, we create base images configured with SDKs, libraries, frameworks, configurations, binaries, etc... Those base images are then built, versioned, tagged and then pushed to our container repos ready to be used by developers, CI/CD, etc... Images based on these base images never need an apt-get, pip install, etc... If there is a dependency missing, updated needed, etc... we'll create a new base image with it, following the steps above. I would love some constructive feedback.
- sp332 6y ago
  That solves the problem one way, but I don't know if I would call it reproducible. You're working around the fact that it's not reproducible by only doing it once. You don't get the benefits, e.g. everyone who uses your images has to trust that you built them the right way.
  - chrisjc 6y ago
    I guess what I mean is that everything built using this base image should be reproducible. There is no reason (hopefully) to reproduce the base image. Any changes to the base requirements (apt-get, pip, etc) requires a whole new build and results in an entirely different artifact. And just to be clear, I'm not building (no human) the base image. The base image is also created within it's own build pipeline that has all of the necessary things to track its materialization and lineage. Logs, manifests, etc... Once the image has been thoroughly tested and verified (both by humans and verification scripts) each time a change is merged, the git repo is tagged, docker image is built and tagged and then pushed to the container repo. Perhaps you could explain what you mean by the other way? Why would you ever need to recreate the base image? Perhaps if the container repo dropped off the face of the earth and had to be created from scratch?
- donmcronald 6y ago
  This is what I think is the most practical approach. Pinning down ALL your dependencies to the exact version is much harder than it sounds. What you’re describing sound like the way Jib [1] does it. The pictures in this [2] blog post help visualize it. The reason I like the approach you describe is because it keeps things simpler at the start of a project and consistent across most projects. I also think it makes sense to have those support containers build on a schedule. For example, you build your build/CI container weekly and that’s the CI container for the week. On demand project builds use that CI container which has all dependencies, etc. baked in. It would be nice if CI systems would let me explicitly tag builds as (non)reproducible. 1. https://github.com/GoogleContainerTools/jib https://github.com/GoogleContainerTools/jib 2. https://phauer.com/2019/no-fat-jar-in-docker-image/ https://phauer.com/2019/no-fat-jar-in-docker-image/
astrea 6y ago
I'm interested to know your alternate solution. How would you recommend getting the required libraries into your image?
- GordonS 6y ago
  I guess the OP meant such that apt/yum/dnf/whatever runs every time the image runs, rather than just once when it's built. Not that that's something I see very often, mind.
  - corytheboyd 6y ago
    Isn’t the standard approach here then to derive from base images, which have exact versions? Being honest, I’m not a Docker/VM legend but I’ve seen a few attempts at managing this, and base images was one of them. Personally, I don’t see the issue with it if you’re at least being a little careful— don’t make obvious mistakes like installing latest/nightly packages automatically, etc.
    - GordonS 6y ago
      Yes, that's the standard approach, but the base images are frequently updated. If you really want to pin at a specific image, you need to specify the image hash, rather than using the "latest", or even a version tag (e.g. "2.1"). In your image that extends from the base image, you'll typically update the package repo cache (it is typically cleared after building the base image, to reduce the size), then install whatever packages you want. Like you, I don't see a particular issue with updating system-level packages - especially from a security standpoint.
falcolas 6y ago
Keeping apt-get in our container builds is vital, since it helps keep CVEs out of our containers. We can do automated re-builds of all of our containers weekly and (typically) it lets us keep on top of the CVE game.
NashHallucinate 6y ago
Not if you specify the versions to install, at least
cgdub 6y ago
If I use tools that can give me reproducible builds, like Bazel or Nix, I won't need or want to use containers for development.
collyw 6y ago
Whats the correct way to get libraries into your docker container? We have plenty of apt-get / pip calls in the dockerfiles at my work (set up by someone else).

outside1234 6y ago

And by that you mean that the version of those deps aren't pinned?

beagle3 6y ago
You depend on what the repository serves for "apt-get". You may pin it, but it still doesn't guarantee you're going to get the same version if it was replaced on the repository without changing the version.
dhruvrrp 6y ago
At least for python, just pinning the top level dependencies are not enough. if you pin tensorflow==2.4.0, it doesn’t pin its required packages, rather it just defines a range. An example would be tf will try to get wheel>=0.26 (which was released in 2015 and is currently on release 0.36.2)

kiallmacinnes 6y ago

Yea - I get where you're coming from - you can't rebuild the container, but each team member can reuse the same container - which is at least a step towards being fully reproducible.

mumblemumble 6y ago

I can't speak for others, but, while we don't pin versions in the Dockerfiles for our build container images, we do version the images themselves. Reproducibility is a continuum, not a binary. We've chosen a point on that continuum that we believe gives us the best trade-off between reliability and maintenance effort. 100% from-the-ground-up reproducibility would be ideal, of course, but there's a cost-benefit tradeoff, and we're not being paid to be perfectionists.

hctaw 6y ago

I mean in a sense, the fucked up behavior of system package managers relative to the actual concerns of building and distributing software to the masses is the reason we need Docker to sandbox environments in the first place. It's 2021, there is next to no reason why the default behavior is installation to /usr/lib with shared objects that are rarely shared, with global access when installed for one application used by one user, and linked dynamically when the object never changes over the lifetime of the running application. For all its warts this is something that npm gets right. Package management is a tool for software development, not software distribution to end users.

deleted 6y ago
[deleted]
moistbar 6y ago
Both apt and yum support installing to whatever directory you like as long as you have write permissions there.
- hctaw 6y ago
  "support" and "default" are very different things. Especially with package managers like homebrew, which comes with a loud warning that non-default (root) paths may not be supported by dependencies. It also doesn't really, since this is a system level issue. Applications need to package their dependencies, not the other way around. Dependencies form graphs, not flat lists. The existence of a global cache of libraries shared by all programs is a total inversion of requirements.
- AnIdiotOnTheNet 6y ago
  Not really. They support dumping the package contents to an arbitrary directory, but you can't actually run the software from there without either the software having been written to support arbitrary paths (almost none is) or using namespacing and chroot to build it a sandbox wherein its baked-in paths actually work. Contrast that with something like RiscOS AppDirs, classic Mac applications, or Next/Mac Application Bundles.

JMTQp8lwXL 6y ago

I never understood why commands like apt-get don't take a version with the dependency name.

astrea 6y ago
They do. You can do `apt-get install package=version`
wongarsu 6y ago
As far as I'm aware it's supported (something like `apt-get install virtualbox=5.0.18-dfsg-2build1 -V`). It's just not commonly used because you usually just choose a distro with the desired update granularity (whether you want to the newest version out there or a consistent version with backported security fixes, or something in between).

amscanne 6y ago

The industry seems to have adopted “hermetic” as a word that describes truly reproducible builds, while “reproducible” has a lower standard. In many cases, it seems to be used to mean “not dependent on local build environment”.

umvi 6y ago

We don't rebuild image every time. We store images in GitLab registry. Also, just because we use apt-get to install python3 doesn't mean build isn't reproducible. The actual toolchain and sysroot are highly version controlled and python is just used to kick off the build script so it almost doesn't really matter which version python is as long as it's backwards compatible with the build script.

physicsguy 6y ago
We tried this in our research group, and found that issues come in when it gets to things like... Oh no! CMake is too old a version to support a dependency we have to build in the image construction. So we better pull in a version of CMake from a PPA which is community maintained, and build it from source/etc.

chrisjc 6y ago

Depends on how you do it, right? For instance, we create base images configured with SDKs, libraries, frameworks, configurations, binaries, etc... Those base images are then built, versioned, tagged and then pushed to our container repos ready to be used by developers, CI/CD, etc... Images based on these base images never need an apt-get, pip install, etc... If there is a dependency missing, updated needed, etc... we'll create a new base image with it, following the steps above. I would love some constructive feedback.

sp332 6y ago
That solves the problem one way, but I don't know if I would call it reproducible. You're working around the fact that it's not reproducible by only doing it once. You don't get the benefits, e.g. everyone who uses your images has to trust that you built them the right way.
- chrisjc 6y ago
  I guess what I mean is that everything built using this base image should be reproducible. There is no reason (hopefully) to reproduce the base image. Any changes to the base requirements (apt-get, pip, etc) requires a whole new build and results in an entirely different artifact. And just to be clear, I'm not building (no human) the base image. The base image is also created within it's own build pipeline that has all of the necessary things to track its materialization and lineage. Logs, manifests, etc... Once the image has been thoroughly tested and verified (both by humans and verification scripts) each time a change is merged, the git repo is tagged, docker image is built and tagged and then pushed to the container repo. Perhaps you could explain what you mean by the other way? Why would you ever need to recreate the base image? Perhaps if the container repo dropped off the face of the earth and had to be created from scratch?
donmcronald 6y ago
This is what I think is the most practical approach. Pinning down ALL your dependencies to the exact version is much harder than it sounds. What you’re describing sound like the way Jib [1] does it. The pictures in this [2] blog post help visualize it. The reason I like the approach you describe is because it keeps things simpler at the start of a project and consistent across most projects. I also think it makes sense to have those support containers build on a schedule. For example, you build your build/CI container weekly and that’s the CI container for the week. On demand project builds use that CI container which has all dependencies, etc. baked in. It would be nice if CI systems would let me explicitly tag builds as (non)reproducible. 1. https://github.com/GoogleContainerTools/jib https://github.com/GoogleContainerTools/jib 2. https://phauer.com/2019/no-fat-jar-in-docker-image/ https://phauer.com/2019/no-fat-jar-in-docker-image/

astrea 6y ago

I'm interested to know your alternate solution. How would you recommend getting the required libraries into your image?

GordonS 6y ago
I guess the OP meant such that apt/yum/dnf/whatever runs every time the image runs, rather than just once when it's built. Not that that's something I see very often, mind.
- corytheboyd 6y ago
  Isn’t the standard approach here then to derive from base images, which have exact versions? Being honest, I’m not a Docker/VM legend but I’ve seen a few attempts at managing this, and base images was one of them. Personally, I don’t see the issue with it if you’re at least being a little careful— don’t make obvious mistakes like installing latest/nightly packages automatically, etc.
  - GordonS 6y ago
    Yes, that's the standard approach, but the base images are frequently updated. If you really want to pin at a specific image, you need to specify the image hash, rather than using the "latest", or even a version tag (e.g. "2.1"). In your image that extends from the base image, you'll typically update the package repo cache (it is typically cleared after building the base image, to reduce the size), then install whatever packages you want. Like you, I don't see a particular issue with updating system-level packages - especially from a security standpoint.

falcolas 6y ago

Keeping apt-get in our container builds is vital, since it helps keep CVEs out of our containers. We can do automated re-builds of all of our containers weekly and (typically) it lets us keep on top of the CVE game.

NashHallucinate 6y ago

Not if you specify the versions to install, at least

cgdub 6y ago

If I use tools that can give me reproducible builds, like Bazel or Nix, I won't need or want to use containers for development.

collyw 6y ago

Whats the correct way to get libraries into your docker container? We have plenty of apt-get / pip calls in the dockerfiles at my work (set up by someone else).