3 ms·
I would rephrase this as: The Dockerfile format imposes a hierarchical relationship between layers. This quickly becomes very annoying, since dependencies usua
by hamandcheese 3y ago
I would rephrase this as:
The Dockerfile format imposes a hierarchical relationship between layers. This quickly becomes very annoying, since dependencies usually form dependency graphs, not dependency trees.
Alternative tools, like nix (probably bazel too), are not bound in the same way. They can achieve fine grained caching by mapping their dependency graph to docker layers, which is something that can not be expressed with a Dockerfile.
- cpuguy83 3y agoSteps in a stage are hierarchical. The final result need not be. You can build a bunch of things then merge the results in a final stage without any hierarchy (this is "COPY --link" in a Dockerfile).
- georgyo 3y agoThat requires some very explicit and non-obvious effort to do. It's quite painful to do this properly in Docker.
- cpuguy83 3y agoAnd the consensus seems to be nix is not straight forward?
- georgyo 3y agoAnd the snake eats it tail. With nix, re-usability is very high. It's a function that is very baked in at very low levels of it's design. This comes with up front complexity but getting to these reusable layers is basically forced. Docker is very simple and often touts reusable layers, but in practice is not. Unless you tackle that complexity. Making reproducible and reusable content takes effort. Other tools are not designed for that. As a result the getting to the same state requires a similar amount of complexity. Worse, with docker you can never be sure that you actually succeeded in your goal of reproducibility. An analogy could be rust. Rust has up front complexity, but tackling that complexity gives confidence that memory safety and concurrency primitives are done correctly. It's not that C _can't_ achieve the same runtime safety, it's just requires a lot more skill to do correctly; and even then memory exploits are reported on a near daily basses for very popular and widely used libraries. Complex problems are complex. And sooner or later you'll need to face that complexity.
- cpuguy83 3y agoThis is not how docker works. Docker, exactly like nix, is based on a graph of content addressable dependencies. What you are describing is chaining a bunch of commands together. Yes, this forms a dependency chain stored in separate layers and is part of the cache chain. Nix suffers the exact same problems with reproducibility. The thing it provides is the toolchain of dependencies that are reproducible. Docker does not provide your dependencies. If the inputs change then so does the output. If the output itself is not reproducible (like, say an artifact with a build-time embedded in it) then you have something that is inherently not reproducible and two people trying to build the same exact nix package will have different results. EDIT: Fixed a sentence I apparently got distracted while writing and didn't complete (about layer caching).
- Foxboron 3y agoNix is not content addressable though, the hashes is based off on the derivation files which are equal to the lock files you would find in other package managers. > The thing it provides is the toolchain of dependencies that are reproducible. [...] If the inputs change then so does the output. If the output itself is not reproducible (like, say an artifact with a build-time embedded in it) then you have something that is inherently not reproducible and two people trying to build the same exact nix package will have different results. There are no guarantees they are reproducible. The only guarantees Nix gives you is that the build environment is the same which allows you to make some claims about the system behaving the same way. But they are certainly no guarantees about artifacts being bit-by-bit identical.
- runeks 3y ago> The Dockerfile format imposes a hierarchical relationship between layers. This quickly becomes very annoying, since dependencies usually form dependency graphs, not dependency trees. Isn't a Dockerfile just a sequence of dependencies, rather than a tree?
- hamandcheese 3y agoYou're right, though it becomes hierarchical once you have multiple Dockerfiles inheriting from some base image (which I did not articulate in my original comment).