8 ms·

I've been using Nix for this. It is great for building an image that contains exactly the dependencies you need. Often that is just my app, glibc and maybe a fe

I've been using Nix for this. It is great for building an image that contains exactly the dependencies you need. Often that is just my app, glibc and maybe a few data files. However if you need more it is trivial to bundle it in. For example I have a CI image that contains bash, Nix itself, some shell scripts with their dependencies and a few other commands and files that GitLab CI expects to be available. I absolutely love the declarative nature and not needing to worry about each `RUN` step creating a layer which may bloat the image. For example my build process validates my inline SQL queries against the database schema. It was refreshingly simple to spin up a Postgres instance inside of the build step, applying migrations using a different CLI tool then start the build without any of these deps ending up in the final image. The only real downside is that Nix doesn't have great support for incremental builds, so for my Rust app building the optimized build from scratch can be slow if you change a comment in a source file. But most Docker builds don't do this either (or if they do it is often buggy which I see as worse). Bazel does help this which is a notable advantage for trading off the ability to pull in other programs from nixpkgs.

pharmakom 4y ago
Someone is working on consuming nix packages inside Bazel.
- rockwotj 4y ago
  See: https://github.com/tweag/rules_nixpkgs https://github.com/tweag/rules_nixpkgs
webstrand 4y ago
Do you have an example or an article demonstrating this? I just recently had the desire to build systemd-nspawn images declaratively, but couldn't find much other than Dockerfiles.
- kevincox 4y ago
  Sure. Putting a simple binary in a container: https://gitlab.com/kevincox/tiles/-/blob/a2b907eab7a84989c9484973c8941aac95472e25/default.nix#L100-111 https://gitlab.com/kevincox/tiles/-/blob/a2b907eab7a84989c94.... This is the trivial case where you just stick the main executable in the command string. Nix will automatically include the dependencies. The GitLab CI example is a bit more complex. It requires some commands that are unused by the image and some config files: https://gitlab.com/kevincox/nix-ci/-/blob/efe6f4deedc50c2474cb99fef13242e70bf09e51/default.nix#L51-71 https://gitlab.com/kevincox/nix-ci/-/blob/efe6f4deedc50c2474...
- rgoulter 4y ago
  https://lethalman.blogspot.com/2016/04/cheap-docker-images-with-nix_15.html https://lethalman.blogspot.com/2016/04/cheap-docker-images-w...
tazjin 4y ago
I built a service for doing this ad-hoc via image names a few years ago and it enjoys some popularity with CI & debugging use-cases: https://nixery.dev/ https://nixery.dev/
- kevincox 4y ago
  I've definitely used this in CI before. It is useful for base images for docker-based CI.
foverzar 4y ago
> not needing to worry about each `RUN` step creating a layer which may bloat the image Could someone please explain to me, why exactly do people avoid layers and treat them as "bloat"? I always thought that layers are nice to have: you only need to rebuild those that had changed (typically the last ones that handle the application, while environment layers remain the same) and pulling image updates is handled much better due to only changed layers being pulled. How is this "bloat"? Isn't that the opposite? Pushing images containing 80% of the same stuff feels more like a bloat to me. Am I missing something here?
- kevincox 4y ago
  It depends. There is some implicit "bloat" because setting up 100 layers and accessing files through them isn't free (but caching works quite well). However the biggest problem with layers is that you can never delete data. So doing something like `RUN apt-get install foo` `RUN foo generate-things` `RUN apt-get uninstall foo` will effectively still have `foo` in the image. It definitely depends on the use case. In many cases `RUN foo`, `RUN bar`, `RUN baz` is fine. But if you are every creating temporary data in an image the layer system will keep that around. This is why you often see things like `RUN apt-get update && apt-get install foo && rm -r /var/lib/apt`. You squeeze it into a single layer so that the deletion of the temp files actually avoids image bloat.
  - throwaway290 4y ago
    It is possible with build stages and copy from, but yeah not super trivial.
    - kevincox 4y ago
      Exactly. You need to remember to do it and restructure your build at least slightly to do so. It isn't hard but non-default and annoying.
    - mcronce 4y ago
      Definitely not trivial, but staged builds are my go-to solution. Depending on the specifics of the tech you're including it can be a lot easier than figuring out how to clean up every little build artifact within a layer - just add a second FROM line and copy precisely the pieces you need in the final image, and nothing else. I also think it makes the build stage a lot easier to follow for people who aren't as familiar with Dockerfiles and all the quirks that come with optimizing a final image.
jvolkman 4y ago
I put together an example that mixes Nix and Bazel a couple of years ago: https://github.com/jvolkman/bazel-nix-example https://github.com/jvolkman/bazel-nix-example Nix is used to build a base Docker image, and Bazel builds layers on top.
lewo 4y ago
To get Rust incremental builds, did you consider using something such as crane https://github.com/ipetkov/crane https://github.com/ipetkov/crane ? And regarding OCI images, i built nix2container (https://github.com/nlewo/nix2container https://github.com/nlewo/nix2container) to speed up image build and push times.
herewulf 4y ago
I've been doing the same with Guix. However, more so lately with declarative, lightweight VMs. It's nice to be able to painlessly make throw away environments that I can easily log into via SSH.

pharmakom 4y ago

Someone is working on consuming nix packages inside Bazel.

rockwotj 4y ago
See: https://github.com/tweag/rules_nixpkgs https://github.com/tweag/rules_nixpkgs

webstrand 4y ago

Do you have an example or an article demonstrating this? I just recently had the desire to build systemd-nspawn images declaratively, but couldn't find much other than Dockerfiles.

kevincox 4y ago
Sure. Putting a simple binary in a container: https://gitlab.com/kevincox/tiles/-/blob/a2b907eab7a84989c9484973c8941aac95472e25/default.nix#L100-111 https://gitlab.com/kevincox/tiles/-/blob/a2b907eab7a84989c94.... This is the trivial case where you just stick the main executable in the command string. Nix will automatically include the dependencies. The GitLab CI example is a bit more complex. It requires some commands that are unused by the image and some config files: https://gitlab.com/kevincox/nix-ci/-/blob/efe6f4deedc50c2474cb99fef13242e70bf09e51/default.nix#L51-71 https://gitlab.com/kevincox/nix-ci/-/blob/efe6f4deedc50c2474...
rgoulter 4y ago
https://lethalman.blogspot.com/2016/04/cheap-docker-images-with-nix_15.html https://lethalman.blogspot.com/2016/04/cheap-docker-images-w...

tazjin 4y ago

I built a service for doing this ad-hoc via image names a few years ago and it enjoys some popularity with CI & debugging use-cases: https://nixery.dev/ https://nixery.dev/

kevincox 4y ago
I've definitely used this in CI before. It is useful for base images for docker-based CI.

foverzar 4y ago

> not needing to worry about each `RUN` step creating a layer which may bloat the image Could someone please explain to me, why exactly do people avoid layers and treat them as "bloat"? I always thought that layers are nice to have: you only need to rebuild those that had changed (typically the last ones that handle the application, while environment layers remain the same) and pulling image updates is handled much better due to only changed layers being pulled. How is this "bloat"? Isn't that the opposite? Pushing images containing 80% of the same stuff feels more like a bloat to me. Am I missing something here?

kevincox 4y ago
It depends. There is some implicit "bloat" because setting up 100 layers and accessing files through them isn't free (but caching works quite well). However the biggest problem with layers is that you can never delete data. So doing something like `RUN apt-get install foo` `RUN foo generate-things` `RUN apt-get uninstall foo` will effectively still have `foo` in the image. It definitely depends on the use case. In many cases `RUN foo`, `RUN bar`, `RUN baz` is fine. But if you are every creating temporary data in an image the layer system will keep that around. This is why you often see things like `RUN apt-get update && apt-get install foo && rm -r /var/lib/apt`. You squeeze it into a single layer so that the deletion of the temp files actually avoids image bloat.
- throwaway290 4y ago
  It is possible with build stages and copy from, but yeah not super trivial.
  - kevincox 4y ago
    Exactly. You need to remember to do it and restructure your build at least slightly to do so. It isn't hard but non-default and annoying.
  - mcronce 4y ago
    Definitely not trivial, but staged builds are my go-to solution. Depending on the specifics of the tech you're including it can be a lot easier than figuring out how to clean up every little build artifact within a layer - just add a second FROM line and copy precisely the pieces you need in the final image, and nothing else. I also think it makes the build stage a lot easier to follow for people who aren't as familiar with Dockerfiles and all the quirks that come with optimizing a final image.

jvolkman 4y ago

I put together an example that mixes Nix and Bazel a couple of years ago: https://github.com/jvolkman/bazel-nix-example https://github.com/jvolkman/bazel-nix-example Nix is used to build a base Docker image, and Bazel builds layers on top.

lewo 4y ago

To get Rust incremental builds, did you consider using something such as crane https://github.com/ipetkov/crane https://github.com/ipetkov/crane ? And regarding OCI images, i built nix2container (https://github.com/nlewo/nix2container https://github.com/nlewo/nix2container) to speed up image build and push times.

herewulf 4y ago

I've been doing the same with Guix. However, more so lately with declarative, lightweight VMs. It's nice to be able to painlessly make throw away environments that I can easily log into via SSH.