3 ms·

Steps in a stage are hierarchical. The final result need not be. You can build a bunch of things then merge the results in a final stage without any hierarchy

Steps in a stage are hierarchical. The final result need not be. You can build a bunch of things then merge the results in a final stage without any hierarchy (this is "COPY --link" in a Dockerfile).

georgyo 3y ago
That requires some very explicit and non-obvious effort to do. It's quite painful to do this properly in Docker.
- cpuguy83 3y ago
  And the consensus seems to be nix is not straight forward?
  - georgyo 3y ago
    And the snake eats it tail. With nix, re-usability is very high. It's a function that is very baked in at very low levels of it's design. This comes with up front complexity but getting to these reusable layers is basically forced. Docker is very simple and often touts reusable layers, but in practice is not. Unless you tackle that complexity. Making reproducible and reusable content takes effort. Other tools are not designed for that. As a result the getting to the same state requires a similar amount of complexity. Worse, with docker you can never be sure that you actually succeeded in your goal of reproducibility. An analogy could be rust. Rust has up front complexity, but tackling that complexity gives confidence that memory safety and concurrency primitives are done correctly. It's not that C _can't_ achieve the same runtime safety, it's just requires a lot more skill to do correctly; and even then memory exploits are reported on a near daily basses for very popular and widely used libraries. Complex problems are complex. And sooner or later you'll need to face that complexity.
    - cpuguy83 3y ago
      This is not how docker works. Docker, exactly like nix, is based on a graph of content addressable dependencies. What you are describing is chaining a bunch of commands together. Yes, this forms a dependency chain stored in separate layers and is part of the cache chain. Nix suffers the exact same problems with reproducibility. The thing it provides is the toolchain of dependencies that are reproducible. Docker does not provide your dependencies. If the inputs change then so does the output. If the output itself is not reproducible (like, say an artifact with a build-time embedded in it) then you have something that is inherently not reproducible and two people trying to build the same exact nix package will have different results. EDIT: Fixed a sentence I apparently got distracted while writing and didn't complete (about layer caching).
      - Foxboron 3y ago
        Nix is not content addressable though, the hashes is based off on the derivation files which are equal to the lock files you would find in other package managers. > The thing it provides is the toolchain of dependencies that are reproducible. [...] If the inputs change then so does the output. If the output itself is not reproducible (like, say an artifact with a build-time embedded in it) then you have something that is inherently not reproducible and two people trying to build the same exact nix package will have different results. There are no guarantees they are reproducible. The only guarantees Nix gives you is that the build environment is the same which allows you to make some claims about the system behaving the same way. But they are certainly no guarantees about artifacts being bit-by-bit identical.
        cpuguy83 3y ago
        It's content addressable, just what are you addressing? The content address of a docker image is a json blob (referencing other objects). The content address of a Dockerfile "RUN" command is the content address of what came before it and the command being run.
        Foxboron 3y ago
        >It's content addressable, just what are you addressing? In the case of Nix it's addressed by the input. Not the content of the build. It's an important distinction and one Nix also makes. https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-store-make-content-addressed https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3... But doing this is going to give you a slight headace as most of the package repository in Nix is not checked for reproducible builds and there are no way to guarantee the hashes are actually static.

Steps in a stage are hierarchical. The final result need not be. You can build a bunch of things then merge the results in a final stage without any hierarchy (this is "COPY --link" in a Dockerfile).

georgyo 3y ago

That requires some very explicit and non-obvious effort to do. It's quite painful to do this properly in Docker.

cpuguy83 3y ago
And the consensus seems to be nix is not straight forward?
- georgyo 3y ago
  And the snake eats it tail. With nix, re-usability is very high. It's a function that is very baked in at very low levels of it's design. This comes with up front complexity but getting to these reusable layers is basically forced. Docker is very simple and often touts reusable layers, but in practice is not. Unless you tackle that complexity. Making reproducible and reusable content takes effort. Other tools are not designed for that. As a result the getting to the same state requires a similar amount of complexity. Worse, with docker you can never be sure that you actually succeeded in your goal of reproducibility. An analogy could be rust. Rust has up front complexity, but tackling that complexity gives confidence that memory safety and concurrency primitives are done correctly. It's not that C _can't_ achieve the same runtime safety, it's just requires a lot more skill to do correctly; and even then memory exploits are reported on a near daily basses for very popular and widely used libraries. Complex problems are complex. And sooner or later you'll need to face that complexity.
  - cpuguy83 3y ago
    This is not how docker works. Docker, exactly like nix, is based on a graph of content addressable dependencies. What you are describing is chaining a bunch of commands together. Yes, this forms a dependency chain stored in separate layers and is part of the cache chain. Nix suffers the exact same problems with reproducibility. The thing it provides is the toolchain of dependencies that are reproducible. Docker does not provide your dependencies. If the inputs change then so does the output. If the output itself is not reproducible (like, say an artifact with a build-time embedded in it) then you have something that is inherently not reproducible and two people trying to build the same exact nix package will have different results. EDIT: Fixed a sentence I apparently got distracted while writing and didn't complete (about layer caching).
    - Foxboron 3y ago
      Nix is not content addressable though, the hashes is based off on the derivation files which are equal to the lock files you would find in other package managers. > The thing it provides is the toolchain of dependencies that are reproducible. [...] If the inputs change then so does the output. If the output itself is not reproducible (like, say an artifact with a build-time embedded in it) then you have something that is inherently not reproducible and two people trying to build the same exact nix package will have different results. There are no guarantees they are reproducible. The only guarantees Nix gives you is that the build environment is the same which allows you to make some claims about the system behaving the same way. But they are certainly no guarantees about artifacts being bit-by-bit identical.
      - cpuguy83 3y ago
        It's content addressable, just what are you addressing? The content address of a docker image is a json blob (referencing other objects). The content address of a Dockerfile "RUN" command is the content address of what came before it and the command being run.
        Foxboron 3y ago
        >It's content addressable, just what are you addressing? In the case of Nix it's addressed by the input. Not the content of the build. It's an important distinction and one Nix also makes. https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-store-make-content-addressed https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3... But doing this is going to give you a slight headace as most of the package repository in Nix is not checked for reproducible builds and there are no way to guarantee the hashes are actually static.