3 ms·

> wrap your build environment with docker (or your favorite packager) so that your tooling and dependencies become both explicit and reproducable If you want e

> wrap your build environment with docker (or your favorite packager) so that your tooling and dependencies become both explicit and reproducable If you want explicitness and reproducibility please don't reach for Docker. Unless you take a lot of care, you will only get the most watered down version of reproducibility with Docker probably luring you into a false sense of security. E.g. pointing to mutable image tags without integrity hashes and invoking apt-get are things you'll find in most Dockerfiles out there and both leave open a huge surface area for things to go wrong and end up in slightly different states. And while they are not that easy to pick up, solutions like Bazel and Nix will give you a lot better foundation to stand on.

basicallybones 3y ago
A nice middle ground is using a tool like Google's Skaffold, which provides "Bazel-like" capabilities for composing Docker images and tagging them based on a number of strategies, including file manifests. In my case, I also use build args to explicitly set versions of external dependencies. I also pull external images and build base images with upgraded versions once, then re-tag them in my private repository, which is an easy-to-implement mechanism for reproducibility. While I am in a Typescript environment with this setup at the moment, my personal experience that Skaffold with Docker has a lighter implementation and maintenance overhead than Bazel. (You also get the added benefit of easy deployment and automatic rebuilds.) I quite liked using Bazel in a small Golang monorepo, but I ran into pain when trying to do things like include third-party pre-compiled binaries in the Docker builds, because of the unusual build rules convention. The advantage of Skaffold is it provides a thin build/tag/deploy/verify layer over Docker and other container types. Might be worth a look! Kudos to the Google team building it! https://skaffold.dev https://skaffold.dev
hawk_ 3y ago
> If you want explicitness and reproducibility please don't reach for Docker Is it common in C++ builds to rely on the current O/S libraries instead of say making most dependencies explicit, close to full cross-compile? Do dependencies need to be pulled in using apt-get and not something like maven?
- SJC_Hacker 3y ago
  If you care about security and bug fixes, then yes From what I've seen the "minor version" is fixed. e.g. FROM ubuntu:22.04 and not FROM ubuntu:laatest
bluGill 3y ago
I have docker as part of my reproducibility builds. We recently ran into a problem trying to rebuild some old code - turns out the ssl certificates in our older images has expired and so now the code isn't reproducible anyway. One more reason to agree with the idea that docker shouldn't be used. Though we use docker for a different reason: we are building on linux, targeting linux. It is really easy to use the host system headers or library instead of the target versions of the same - and 99% of the time you will get away with it as they are the same. Then several years later you upgrade your host linux machine and can't figure out why a build that used to work isn't (and since 99% of the stuff is the same this can be months before you test that one exception and then it crashes). Docker ensures we don't install any host headers, or libraries except what is needed for the compiler.
SJC_Hacker 3y ago
> If you want explicitness and reproducibility please don't reach for Docker. Unless you take a lot of care, you will only get the most watered down version of reproducibility with Docker probably luring you into a false sense of security. E.g. pointing to mutable image tags without integrity hashes and invoking apt-get are things you'll find in most Dockerfiles out there and both leave open a huge surface area for things to go wrong and end up in slightly different states. If this is frequently a problem you're doing something wrong, or using such a crappy external library/toolchain that breaks frequently on the same version. Docker is a way to ensure that the software builds with "the most recent minor version" of some OS/toolchain/libraries. The reason why you want the most recent version is because of security fixes and bugs. I agree that you should check integrtiy hashes where appropriate, if you really want to fix versions.
Tainnor 3y ago
Not everything is all or nothing. 80-90% reproducible builds are often good enough and learning Nix only to get that last 10-20% is not always worth it. And there are ways of pinning all dependencies with docker if you really want to. I've had issues with Dockerfiles not building anymore due to changes in the package registry, but it was like 1-2 times out of 1000 of times I used docker.

basicallybones 3y ago

A nice middle ground is using a tool like Google's Skaffold, which provides "Bazel-like" capabilities for composing Docker images and tagging them based on a number of strategies, including file manifests. In my case, I also use build args to explicitly set versions of external dependencies. I also pull external images and build base images with upgraded versions once, then re-tag them in my private repository, which is an easy-to-implement mechanism for reproducibility. While I am in a Typescript environment with this setup at the moment, my personal experience that Skaffold with Docker has a lighter implementation and maintenance overhead than Bazel. (You also get the added benefit of easy deployment and automatic rebuilds.) I quite liked using Bazel in a small Golang monorepo, but I ran into pain when trying to do things like include third-party pre-compiled binaries in the Docker builds, because of the unusual build rules convention. The advantage of Skaffold is it provides a thin build/tag/deploy/verify layer over Docker and other container types. Might be worth a look! Kudos to the Google team building it! https://skaffold.dev https://skaffold.dev

hawk_ 3y ago

> If you want explicitness and reproducibility please don't reach for Docker Is it common in C++ builds to rely on the current O/S libraries instead of say making most dependencies explicit, close to full cross-compile? Do dependencies need to be pulled in using apt-get and not something like maven?

SJC_Hacker 3y ago
If you care about security and bug fixes, then yes From what I've seen the "minor version" is fixed. e.g. FROM ubuntu:22.04 and not FROM ubuntu:laatest

bluGill 3y ago

I have docker as part of my reproducibility builds. We recently ran into a problem trying to rebuild some old code - turns out the ssl certificates in our older images has expired and so now the code isn't reproducible anyway. One more reason to agree with the idea that docker shouldn't be used. Though we use docker for a different reason: we are building on linux, targeting linux. It is really easy to use the host system headers or library instead of the target versions of the same - and 99% of the time you will get away with it as they are the same. Then several years later you upgrade your host linux machine and can't figure out why a build that used to work isn't (and since 99% of the stuff is the same this can be months before you test that one exception and then it crashes). Docker ensures we don't install any host headers, or libraries except what is needed for the compiler.

SJC_Hacker 3y ago

> If you want explicitness and reproducibility please don't reach for Docker. Unless you take a lot of care, you will only get the most watered down version of reproducibility with Docker probably luring you into a false sense of security. E.g. pointing to mutable image tags without integrity hashes and invoking apt-get are things you'll find in most Dockerfiles out there and both leave open a huge surface area for things to go wrong and end up in slightly different states. If this is frequently a problem you're doing something wrong, or using such a crappy external library/toolchain that breaks frequently on the same version. Docker is a way to ensure that the software builds with "the most recent minor version" of some OS/toolchain/libraries. The reason why you want the most recent version is because of security fixes and bugs. I agree that you should check integrtiy hashes where appropriate, if you really want to fix versions.

Tainnor 3y ago

Not everything is all or nothing. 80-90% reproducible builds are often good enough and learning Nix only to get that last 10-20% is not always worth it. And there are ways of pinning all dependencies with docker if you really want to. I've had issues with Dockerfiles not building anymore due to changes in the package registry, but it was like 1-2 times out of 1000 of times I used docker.