4 ms·

Docker's build system has been given a huge overhaul and does create a dependency tree. It is highly efficient and even does things like support mounts for cach

Docker's build system has been given a huge overhaul and does create a dependency tree. It is highly efficient and even does things like support mounts for caching package downloads, build artifacts, etc. See https://github.com/moby/buildkit https://github.com/moby/buildkit. You can enable it today with `DOCKER_BUILDKIT=1 docker build ...` There is also buildx which is an experimental tool to replace `docker build` with a new CLI: https://github.com/docker/buildx https://github.com/docker/buildx

throwaway894345 7y ago
I don't see how buildkit could possibly build the correct dependency tree because the Dockerfile language doesn't let you express nonlinear dependencies. If you have a command `do_foo` that depends on do_bar and do_baz (but do_bar and do_baz are independent) and you do something like: RUN do_bar # line 1 RUN do_baz # line 2 RUN do_foo # line 3 I'm guessing the buildkit dep graph will look like `line_3 -> line_2 -> line_1` (linear). Unless there is some new way of expressing to Docker that do_foo depends on do_bar and do_baz but that the latter two are independent. EDIT: clarified example.
- cpuguy83 7y ago
  Dependcies are also for multiple stages, e.g.: So `COPY --from=<some stage>` and `FROM <other stage>` Also, a Dockerfile is just a frontend for buildkit. The heart of buildkit is "LLB" (sort of like LLVM IR in usage) which is what the Dockerfile compiles into. Buildkit just executes LLB, doesn't have to be from a Dockerfile. For that matter you can have a "Dockerfile" (in name only) that is not even really dockerfile since the format lets you specify frontend to use (which would be a container image reference) to process it. There's even a buildkit frontend to build buildpacks: https://github.com/tonistiigi/buildkit-pack https://github.com/tonistiigi/buildkit-pack Works with any buildkit enabled builder, even `docker build`.
  - throwaway894345 7y ago
    Yeah, there are a lot of workarounds when you leave the standard tooling. I like the idea of using something like Bazel to build Docker images (since modeling dependencies and caching build steps is its raison d'etre) and eschewing Dockerfiles altogether; however, I haven't tried it (and Bazel has its own costs). I'm not familiar with buildkit in particular, but it's cool that it has an internal representation. I'll have to dig around.
    - cpuguy83 7y ago
      I'm not sure I understand what you mean by workarounds and "leavings standard tooling"
      - weberc2 7y ago
        I’m guessing he meant roughly ”docker build”
    - takeda 7y ago
      Nix also can build efficient docker images and computes layers in a way to make them reusable among multiple projects.
      - cpuguy83 7y ago
        Yes! Check out https://nixery.dev/ https://nixery.dev/
        takeda 7y ago
        Yeah, it is cool, but it feels more like a demo for nix. If you want to create docker image of your own app, you probably would use: https://nixos.org/nixpkgs/manual/#sec-pkgs-dockerTools https://nixos.org/nixpkgs/manual/#sec-pkgs-dockerTools This will produce an exported docker image as tar file, which then you can import it using either docker, or tool like skopeo[1] (which is also included in nixpkgs). The nix-shell functionality is also quite nice, because it allows you to create common development environment, with all tooling available that one might need to work. [1] https://github.com/containers/skopeo https://github.com/containers/skopeo
- thenewnewguy 7y ago
  Correct (as far as I know), but: A) Changing a Dockerfile is rare B) Typically the lines that change (adding your code) are near the end of the Dockerfile, and the long part with installing libraries is at the beginning
  - throwaway894345 7y ago
    (A) is true, but it's not the only way to bust the cache. Changes to the filesystem are much more common. You can try to account for this by selectively copying in files in some sort of dependency order and this can work okay so long as the dependency order resembles your filesystem hierarchy, but if you want to (for example) install all of your third party dependencies before adding your source code, you'll need to add each requirements.txt or package.json file individually so as to avoid copying in your source code (which changes more frequently). Doing this also tightly-couples your Dockerfile to your in-tree dependency structure, and keeping the Dockerfile in sync with your dependency tree is an exercise in vanity. Further, because you're forcing a tree-structure (dependency tree) into a linear structure, you are going to be rebuilding a bunch of stuff that doesn't need to be built (this gets worse the wider your dependency tree). Maybe you can hack around this by making an independent build stage per in-tree package, which might allow you to model your dependency tree in your Dockerfile, but you're still left keeping this in sync manually. No good options.

throwaway894345 7y ago

I don't see how buildkit could possibly build the correct dependency tree because the Dockerfile language doesn't let you express nonlinear dependencies. If you have a command `do_foo` that depends on do_bar and do_baz (but do_bar and do_baz are independent) and you do something like: RUN do_bar # line 1 RUN do_baz # line 2 RUN do_foo # line 3 I'm guessing the buildkit dep graph will look like `line_3 -> line_2 -> line_1` (linear). Unless there is some new way of expressing to Docker that do_foo depends on do_bar and do_baz but that the latter two are independent. EDIT: clarified example.

cpuguy83 7y ago
Dependcies are also for multiple stages, e.g.: So `COPY --from=<some stage>` and `FROM <other stage>` Also, a Dockerfile is just a frontend for buildkit. The heart of buildkit is "LLB" (sort of like LLVM IR in usage) which is what the Dockerfile compiles into. Buildkit just executes LLB, doesn't have to be from a Dockerfile. For that matter you can have a "Dockerfile" (in name only) that is not even really dockerfile since the format lets you specify frontend to use (which would be a container image reference) to process it. There's even a buildkit frontend to build buildpacks: https://github.com/tonistiigi/buildkit-pack https://github.com/tonistiigi/buildkit-pack Works with any buildkit enabled builder, even `docker build`.
- throwaway894345 7y ago
  Yeah, there are a lot of workarounds when you leave the standard tooling. I like the idea of using something like Bazel to build Docker images (since modeling dependencies and caching build steps is its raison d'etre) and eschewing Dockerfiles altogether; however, I haven't tried it (and Bazel has its own costs). I'm not familiar with buildkit in particular, but it's cool that it has an internal representation. I'll have to dig around.
  - cpuguy83 7y ago
    I'm not sure I understand what you mean by workarounds and "leavings standard tooling"
    - weberc2 7y ago
      I’m guessing he meant roughly ”docker build”
  - takeda 7y ago
    Nix also can build efficient docker images and computes layers in a way to make them reusable among multiple projects.
    - cpuguy83 7y ago
      Yes! Check out https://nixery.dev/ https://nixery.dev/
      - takeda 7y ago
        Yeah, it is cool, but it feels more like a demo for nix. If you want to create docker image of your own app, you probably would use: https://nixos.org/nixpkgs/manual/#sec-pkgs-dockerTools https://nixos.org/nixpkgs/manual/#sec-pkgs-dockerTools This will produce an exported docker image as tar file, which then you can import it using either docker, or tool like skopeo[1] (which is also included in nixpkgs). The nix-shell functionality is also quite nice, because it allows you to create common development environment, with all tooling available that one might need to work. [1] https://github.com/containers/skopeo https://github.com/containers/skopeo
thenewnewguy 7y ago
Correct (as far as I know), but: A) Changing a Dockerfile is rare B) Typically the lines that change (adding your code) are near the end of the Dockerfile, and the long part with installing libraries is at the beginning
- throwaway894345 7y ago
  (A) is true, but it's not the only way to bust the cache. Changes to the filesystem are much more common. You can try to account for this by selectively copying in files in some sort of dependency order and this can work okay so long as the dependency order resembles your filesystem hierarchy, but if you want to (for example) install all of your third party dependencies before adding your source code, you'll need to add each requirements.txt or package.json file individually so as to avoid copying in your source code (which changes more frequently). Doing this also tightly-couples your Dockerfile to your in-tree dependency structure, and keeping the Dockerfile in sync with your dependency tree is an exercise in vanity. Further, because you're forcing a tree-structure (dependency tree) into a linear structure, you are going to be rebuilding a bunch of stuff that doesn't need to be built (this gets worse the wider your dependency tree). Maybe you can hack around this by making an independent build stage per in-tree package, which might allow you to model your dependency tree in your Dockerfile, but you're still left keeping this in sync manually. No good options.