6 ms·

This blog post is missing the reasoning on why shared docker layers are useful. It is because of caching. The more images are sharing the same layers the better

by whazor 3y ago

This blog post is missing the reasoning on why shared docker layers are useful. It is because of caching. The more images are sharing the same layers the better, as it allows you to cache more stuff. Better caching means faster startup of containers. Why is docker bad at this? In order to enjoy the caching benefit, each time you build a docker image you want it to output as much existing layers as possible. So running apt-get install python3 today should result in the exact same layer as yesterday, if there are no new updates. But this requires the all the files to be exactly the same, including the metadata like creation time. As docker layers are cached by hashing the files. Now, Nix already does storing dependencies by hash. So the layers will always be the same with the same version and same configuration.

m1keil 3y ago
In Docker, if the layers are cached, the layer with apt-get won't be automatically invalidated (unless --no-cache or any changes to the upper layers).
- raffraffraff 3y ago
  But that's what I would expect to happen. I don't see a problem.
  - eichin 3y ago
    Won't get invalidated even if what "apt-get install python3" does changes - the cache is only based on the syntax of the RUN string plus the previous layer hash, IIRC. (COPY actually invalidates if the file being copied changes, so maybe there's a way to fetch a hash of the repo and stash it where copy will notice, or something, but then it seems you need external tooling to do that bit?)
  - m1keil 3y ago
    I didn't claim there is a problem. The original comment made it sound as if docker will expire a cached layer because the (potential) result of apt-get is different, which isn't the case.
- whazor 3y ago
  I am thinking more about pipelines that run daily. Also, this ‘docker cache’ effectively means not running the step. So you might miss important security updates. Via Nix you can ensure that your dependencies are updated. And no updates means same hash. When said caching, I meant on the nodes that run the containers. With Nix you can also update only one layer, while keeping the other layers the same.
hamandcheese 3y ago
I would rephrase this as: The Dockerfile format imposes a hierarchical relationship between layers. This quickly becomes very annoying, since dependencies usually form dependency graphs, not dependency trees. Alternative tools, like nix (probably bazel too), are not bound in the same way. They can achieve fine grained caching by mapping their dependency graph to docker layers, which is something that can not be expressed with a Dockerfile.
- cpuguy83 3y ago
  Steps in a stage are hierarchical. The final result need not be. You can build a bunch of things then merge the results in a final stage without any hierarchy (this is "COPY --link" in a Dockerfile).
  - georgyo 3y ago
    That requires some very explicit and non-obvious effort to do. It's quite painful to do this properly in Docker.
    - cpuguy83 3y ago
      And the consensus seems to be nix is not straight forward?
      - georgyo 3y ago
        And the snake eats it tail. With nix, re-usability is very high. It's a function that is very baked in at very low levels of it's design. This comes with up front complexity but getting to these reusable layers is basically forced. Docker is very simple and often touts reusable layers, but in practice is not. Unless you tackle that complexity. Making reproducible and reusable content takes effort. Other tools are not designed for that. As a result the getting to the same state requires a similar amount of complexity. Worse, with docker you can never be sure that you actually succeeded in your goal of reproducibility. An analogy could be rust. Rust has up front complexity, but tackling that complexity gives confidence that memory safety and concurrency primitives are done correctly. It's not that C _can't_ achieve the same runtime safety, it's just requires a lot more skill to do correctly; and even then memory exploits are reported on a near daily basses for very popular and widely used libraries. Complex problems are complex. And sooner or later you'll need to face that complexity.

by whazor 3y ago

m1keil 3y ago

In Docker, if the layers are cached, the layer with apt-get won't be automatically invalidated (unless --no-cache or any changes to the upper layers).

raffraffraff 3y ago
But that's what I would expect to happen. I don't see a problem.
- eichin 3y ago
  Won't get invalidated even if what "apt-get install python3" does changes - the cache is only based on the syntax of the RUN string plus the previous layer hash, IIRC. (COPY actually invalidates if the file being copied changes, so maybe there's a way to fetch a hash of the repo and stash it where copy will notice, or something, but then it seems you need external tooling to do that bit?)
- m1keil 3y ago
  I didn't claim there is a problem. The original comment made it sound as if docker will expire a cached layer because the (potential) result of apt-get is different, which isn't the case.
whazor 3y ago
I am thinking more about pipelines that run daily. Also, this ‘docker cache’ effectively means not running the step. So you might miss important security updates. Via Nix you can ensure that your dependencies are updated. And no updates means same hash. When said caching, I meant on the nodes that run the containers. With Nix you can also update only one layer, while keeping the other layers the same.

hamandcheese 3y ago

I would rephrase this as: The Dockerfile format imposes a hierarchical relationship between layers. This quickly becomes very annoying, since dependencies usually form dependency graphs, not dependency trees. Alternative tools, like nix (probably bazel too), are not bound in the same way. They can achieve fine grained caching by mapping their dependency graph to docker layers, which is something that can not be expressed with a Dockerfile.

cpuguy83 3y ago
Steps in a stage are hierarchical. The final result need not be. You can build a bunch of things then merge the results in a final stage without any hierarchy (this is "COPY --link" in a Dockerfile).
- georgyo 3y ago
  That requires some very explicit and non-obvious effort to do. It's quite painful to do this properly in Docker.
  - cpuguy83 3y ago
    And the consensus seems to be nix is not straight forward?
    - georgyo 3y ago
      And the snake eats it tail. With nix, re-usability is very high. It's a function that is very baked in at very low levels of it's design. This comes with up front complexity but getting to these reusable layers is basically forced. Docker is very simple and often touts reusable layers, but in practice is not. Unless you tackle that complexity. Making reproducible and reusable content takes effort. Other tools are not designed for that. As a result the getting to the same state requires a similar amount of complexity. Worse, with docker you can never be sure that you actually succeeded in your goal of reproducibility. An analogy could be rust. Rust has up front complexity, but tackling that complexity gives confidence that memory safety and concurrency primitives are done correctly. It's not that C _can't_ achieve the same runtime safety, it's just requires a lot more skill to do correctly; and even then memory exploits are reported on a near daily basses for very popular and widely used libraries. Complex problems are complex. And sooner or later you'll need to face that complexity.