8 ms·

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible y

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the continuous build?

q_eng_anon 5y ago
https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless This is the container community’s response to this ambiguity I think.
monocasa 5y ago
I think that's what they're saying, that containers are orthogonal to reproducible builds.
jffry 5y ago
> Containers only give reproducible deployments, not builds I think that means containers alone are insufficient for creating reproducible builds, not that containers make reproducible builds impossible.
- lazide 5y ago
  I believe that is correct. It’s also hard to make container builds really reproducible (even with other parts of the build being so). Some sibling comments have talked about why. Containers are to ops like C is to programming languages. Man is it useful, but boy are there footguns everywhere. Usually still better than the ops equivalent of assembly, which is what came before though - all the custom manual processes. Kubernetes is maybe like early C++. A bit weird, still has footguns, and is it really helping? Mostly it is in most cases, but we’re also discovering many more new failure modes and having to learn some new complex things. As it matures, new classes of problems will be way easier to solve with less work, and most of the early footguns will be found and fixed. There will be new variants that hide the details and de-footgun things too, and that will have various trade offs. No idea what will be the equivalent of Rust or Java yet.
  - thinkharderdev 5y ago
    I love this analogy
thinkingkong 5y ago
The dependencies can change out from underneath you transparently unless you pin everything all the way down the stack. Upstream docker images for example are an easy to understand vector of change. The deb packages can all change minor versions between runs, the npm packages (for example) can change their contents without making a version bump. Theres tons of implicit trust with all these tools, build wise.
- Osiris 5y ago
  npm packages can change without a version change? Can you explain this? npm doesn't allow you to delete any published versions (you can only deprecate them). You aren't allowed to publish a version that's already been published. Even when there have been malicious packages published the solution has been to publish newer versions of the package with the old code. There's no way to delete the malicious package (maybe npm internally can do it?).
  - thinkingkong 5y ago
    Sorry, without a minor version change. You can easily publish a patch version and most people don't pin that part of their dependency.
    - LunaSea 5y ago
      They don't need to pin it directly. They only need to "npm ci" (based on package-lock.json) instead of "npm install" (based on package.json) within the Docker container to get a fully reproducible build.
- pojzon 5y ago
  Tbh I hope ppl do pin stuff to specific versions and have their own repositories of packages because you dont want to have external dependencies during builds that can fail. DevOps 101 more or less..
hinkley 5y ago
This is something that even the Docker core team is not entirely clear on (either they understand it and can't explain it, or they don't understand it). The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository". This is all before you get to application code which may be doing the same non-repeatable actions with gems or crates or maven or node modules, or calling a service and storing the result in your build. Getting repeatable docker images usually is a matter of first getting your build system to be repeatable, and then expanding it to include the docker images. And often times standing up a private repository so you have something like a reliable snapshot of the public repository, one you can go back to if someone plays games with the published versions of things.
- meatmanek 5y ago
  pedantic nit-pick that doesn't detract from your main point: Turing completeness doesn't imply that it can never be idempotent. In fact we'd expect that a particular Turing machine given a particular tape should reliably produce the same output, and similarly any Turing-complete program given the exact same input should produce the exact same output. Turing machines are single-threaded, non-networked machines with no concept of time. The main reason the RUN statement isn't idempotent, in my opinion, is that it can reach out to the network and e.g. fetch the latest packages from a repository, like you mention. (Other things like the clock, race conditions in multi-threaded programs, etc. can also cause RUN statements to do different things on multiple runs, but I'd argue those are relatively uncommon and unimportant.)
  - dilyevsky 5y ago
    Yup, the problem is not that RUN is Turing Complete the problem is that it’s non-hermetic.
    - hinkley 5y ago
      If it's Turing Complete you can't prove that it's hermetic. You can't even prove if it'll ever finish running (halting problem). Maybe someday someone will invent a sort of EBPF for containers, but usually the first batch of RUN commands and the last are calling package managers, and in between you're doing things like creating users and setting permissions using common unix shell commands, which have the same problems. With the possible exception of the unix package managers, we have no hope of those ever being rewritten inside of a proof system, because the package manager is often a form of dogfooding for the language community. The Node package manager is going to be written in Node and use common networking and archive libraries. Same for Ruby, Rust, you name it.
smilliken 5y ago
You already got a few good answers, but I'll echo them: you can do reproducible builds in containers, and nothing's stopping you from using nix inside containers. But you're at the mercy of all the different package managers that people will end up using (apt, npm, pip, make, curl, etc). So your system is only as good as the worst one. I inherited a dozen or so docker containers a while back that I tried to maintain. Literally none of them would build— they all required going down the rabbit hole of troubleshooting some build error of some transitive dependency. So most of them never got updated and the problem got worse over time until they were abandoned. The reason Nix is different is because it was a radically ambitious idea to penetrate deep into how all software is built, and fix issues however many layers down the stack it needed to. They set out to boil the ocean, and somehow succeeded. Containers give up, and paper over the problems by adding another layer of complexity on top. Nix is also complex, but it solves a much larger problem, and it goes much deeper to address root causes of issues.
- nunez 5y ago
  I don't know Nix and can't comment on that, but in my experience, when I've inherited containers that couldn't build, this was usually due to its image orphaned from their parent Dockerfiles (i.e. someone wrote a Dockerfile, pushed an image from said Dockerfile, but never committed the Dockerfile anywhere, so now the image is orphaned and unreproducable) or due to the container being mutated after being brought up with `docker exec` or similar. Assuming that the container's Dockerfile is persisted somewhere in source control, the base image used by that Dockerfile is tagged with a version whose upstream hasn't changed, and that the container isn't modified from the image that Dockerfile produced, you get extremely reproducable builds with extremely explicit dependencies therein. That said, I definitely see the faults in all of this (the base image version is mutable, and the Dockerfile schema doesn't allow you to verify that an image is what you'd expect with a checksum or something like that, containers can be mutated after startup, containers running as root is still a huge problem, etc), but this is definitely a step up from running apps in VMs. Now that I'm typing this out, I'm surprised that buildpacks or Chef's Habitat didn't take off; they solve a lot of these problems while providing similar reproducability and isolation guarantees.

q_eng_anon 5y ago

https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless This is the container community’s response to this ambiguity I think.

monocasa 5y ago

I think that's what they're saying, that containers are orthogonal to reproducible builds.

jffry 5y ago

> Containers only give reproducible deployments, not builds I think that means containers alone are insufficient for creating reproducible builds, not that containers make reproducible builds impossible.

lazide 5y ago
I believe that is correct. It’s also hard to make container builds really reproducible (even with other parts of the build being so). Some sibling comments have talked about why. Containers are to ops like C is to programming languages. Man is it useful, but boy are there footguns everywhere. Usually still better than the ops equivalent of assembly, which is what came before though - all the custom manual processes. Kubernetes is maybe like early C++. A bit weird, still has footguns, and is it really helping? Mostly it is in most cases, but we’re also discovering many more new failure modes and having to learn some new complex things. As it matures, new classes of problems will be way easier to solve with less work, and most of the early footguns will be found and fixed. There will be new variants that hide the details and de-footgun things too, and that will have various trade offs. No idea what will be the equivalent of Rust or Java yet.
- thinkharderdev 5y ago
  I love this analogy

thinkingkong 5y ago

The dependencies can change out from underneath you transparently unless you pin everything all the way down the stack. Upstream docker images for example are an easy to understand vector of change. The deb packages can all change minor versions between runs, the npm packages (for example) can change their contents without making a version bump. Theres tons of implicit trust with all these tools, build wise.

Osiris 5y ago
npm packages can change without a version change? Can you explain this? npm doesn't allow you to delete any published versions (you can only deprecate them). You aren't allowed to publish a version that's already been published. Even when there have been malicious packages published the solution has been to publish newer versions of the package with the old code. There's no way to delete the malicious package (maybe npm internally can do it?).
- thinkingkong 5y ago
  Sorry, without a minor version change. You can easily publish a patch version and most people don't pin that part of their dependency.
  - LunaSea 5y ago
    They don't need to pin it directly. They only need to "npm ci" (based on package-lock.json) instead of "npm install" (based on package.json) within the Docker container to get a fully reproducible build.
pojzon 5y ago
Tbh I hope ppl do pin stuff to specific versions and have their own repositories of packages because you dont want to have external dependencies during builds that can fail. DevOps 101 more or less..

hinkley 5y ago

This is something that even the Docker core team is not entirely clear on (either they understand it and can't explain it, or they don't understand it). The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository". This is all before you get to application code which may be doing the same non-repeatable actions with gems or crates or maven or node modules, or calling a service and storing the result in your build. Getting repeatable docker images usually is a matter of first getting your build system to be repeatable, and then expanding it to include the docker images. And often times standing up a private repository so you have something like a reliable snapshot of the public repository, one you can go back to if someone plays games with the published versions of things.

meatmanek 5y ago
pedantic nit-pick that doesn't detract from your main point: Turing completeness doesn't imply that it can never be idempotent. In fact we'd expect that a particular Turing machine given a particular tape should reliably produce the same output, and similarly any Turing-complete program given the exact same input should produce the exact same output. Turing machines are single-threaded, non-networked machines with no concept of time. The main reason the RUN statement isn't idempotent, in my opinion, is that it can reach out to the network and e.g. fetch the latest packages from a repository, like you mention. (Other things like the clock, race conditions in multi-threaded programs, etc. can also cause RUN statements to do different things on multiple runs, but I'd argue those are relatively uncommon and unimportant.)
- dilyevsky 5y ago
  Yup, the problem is not that RUN is Turing Complete the problem is that it’s non-hermetic.
  - hinkley 5y ago
    If it's Turing Complete you can't prove that it's hermetic. You can't even prove if it'll ever finish running (halting problem). Maybe someday someone will invent a sort of EBPF for containers, but usually the first batch of RUN commands and the last are calling package managers, and in between you're doing things like creating users and setting permissions using common unix shell commands, which have the same problems. With the possible exception of the unix package managers, we have no hope of those ever being rewritten inside of a proof system, because the package manager is often a form of dogfooding for the language community. The Node package manager is going to be written in Node and use common networking and archive libraries. Same for Ruby, Rust, you name it.

smilliken 5y ago

You already got a few good answers, but I'll echo them: you can do reproducible builds in containers, and nothing's stopping you from using nix inside containers. But you're at the mercy of all the different package managers that people will end up using (apt, npm, pip, make, curl, etc). So your system is only as good as the worst one. I inherited a dozen or so docker containers a while back that I tried to maintain. Literally none of them would build— they all required going down the rabbit hole of troubleshooting some build error of some transitive dependency. So most of them never got updated and the problem got worse over time until they were abandoned. The reason Nix is different is because it was a radically ambitious idea to penetrate deep into how all software is built, and fix issues however many layers down the stack it needed to. They set out to boil the ocean, and somehow succeeded. Containers give up, and paper over the problems by adding another layer of complexity on top. Nix is also complex, but it solves a much larger problem, and it goes much deeper to address root causes of issues.

nunez 5y ago
I don't know Nix and can't comment on that, but in my experience, when I've inherited containers that couldn't build, this was usually due to its image orphaned from their parent Dockerfiles (i.e. someone wrote a Dockerfile, pushed an image from said Dockerfile, but never committed the Dockerfile anywhere, so now the image is orphaned and unreproducable) or due to the container being mutated after being brought up with `docker exec` or similar. Assuming that the container's Dockerfile is persisted somewhere in source control, the base image used by that Dockerfile is tagged with a version whose upstream hasn't changed, and that the container isn't modified from the image that Dockerfile produced, you get extremely reproducable builds with extremely explicit dependencies therein. That said, I definitely see the faults in all of this (the base image version is mutable, and the Dockerfile schema doesn't allow you to verify that an image is what you'd expect with a checksum or something like that, containers can be mutated after startup, containers running as root is still a huge problem, etc), but this is definitely a step up from running apps in VMs. Now that I'm typing this out, I'm surprised that buildpacks or Chef's Habitat didn't take off; they solve a lot of these problems while providing similar reproducability and isolation guarantees.