3 ms·

No discussion about Nix-built containers is complete without mentioning nix2container: https://github.com/nlewo/nix2container https://github.com/nlewo/nix2cont

No discussion about Nix-built containers is complete without mentioning nix2container: https://github.com/nlewo/nix2container https://github.com/nlewo/nix2container It is truly magical for handling large, multi-layered containers. Instead of building the container archives themselves and storing them in the nix store, it builds a JSON manifest that is consumed by a lightly patched version of skopeo that streams the layers directly to either your local container engine or the registry. This means you never rebuild or reupload a container layer that is unchanged. Disclosure: I contributed a change in nix2container that allows cheaply pulling non-Nix layers into the build, just using the content hashes from their registry manifests.

Nullabillity 3y ago
Nixpkgs' streamDockerImage does something similar, instead of storing a multi-layer tarball it produces a script that cats them all together on demand, ready to feed into `docker load` or whatever.
- takeda 3y ago
  In that case streamDockerImage produces script that you run then you pipe output to skopeo or docker. nix2container wraps all of that and it automatically runs it in behind the scenes when you call nix run. The whole image generation is much more efficient as well. In the standard streamDockerImage you get a script that generates a docker layers and the image. In nix2container all layers are stored in nix cache so subsequent run doesn't regenerate them. I believe that was the main goal behind this solution. Another benefit (and I think it is even better than the caching) is that it also allows you manually specify what dependencies go to each layer. The automatic way that the code offers, is nice in theory, but because docker has a limit of 128 layers, in practice it starts nicely then the last layer is a clump of remaining dependencies that didn't fit in previous layers. With nix2container I managed for example to create the first layer which contains just Python and all of its dependencies, then the next layer are my application dependencies (Python packages) the last layer is my application. With this approach a simple bugfix in the application only replaces the last layer that's the size of a few kb. Only a change of Python version will rebuild the whole thing.

Nullabillity 3y ago

Nixpkgs' streamDockerImage does something similar, instead of storing a multi-layer tarball it produces a script that cats them all together on demand, ready to feed into `docker load` or whatever.

takeda 3y ago
In that case streamDockerImage produces script that you run then you pipe output to skopeo or docker. nix2container wraps all of that and it automatically runs it in behind the scenes when you call nix run. The whole image generation is much more efficient as well. In the standard streamDockerImage you get a script that generates a docker layers and the image. In nix2container all layers are stored in nix cache so subsequent run doesn't regenerate them. I believe that was the main goal behind this solution. Another benefit (and I think it is even better than the caching) is that it also allows you manually specify what dependencies go to each layer. The automatic way that the code offers, is nice in theory, but because docker has a limit of 128 layers, in practice it starts nicely then the last layer is a clump of remaining dependencies that didn't fit in previous layers. With nix2container I managed for example to create the first layer which contains just Python and all of its dependencies, then the next layer are my application dependencies (Python packages) the last layer is my application. With this approach a simple bugfix in the application only replaces the last layer that's the size of a few kb. Only a change of Python version will rebuild the whole thing.