4 ms·
When I worked on an enterprise data analytics platform, a big problem was docker image growth. People were using different python versions, different cuda versi
by whazor 11mo ago
When I worked on an enterprise data analytics platform, a big problem was docker image growth. People were using different python versions, different cuda versions, all kinds of libraries. With Cuda being over a gigabyte, this all explodes.
The solution is to decompose the docker images and make sure that every layer is hash equivalent. So if people update their Cuda version, it result in a change within the Python layers.
But it looks like Flox now simplifies this via Nix. Every Nix package already has a hash and you can combine packages however you would like.
- justincormack 11mo agoYes, there were various attempts to do this in the container ecosystem, but there is a hard limit on layers on Docker images (because there are hard limits on overlay mounts; you don't really need to overlay all the Nix store mounts of course as they have different paths but the code is for teh geenral case). So then there were various ways of bundling sets of packages into layers, but just managing it directly through Nix store is much simpler.
- __MatrixMan__ 11mo agoI was an early and enthusiastic adopter of docker. I really liked how it would let me use layers to keep track of dependency between files. After spending a few years using nix, the docker image situation looks pretty bonkers. If two files end up in separate layers, the system assumes dependency so if the lower file changes you need to build a separate copy of the higher one just in case there's actual dependency there. Within nix you can be more precise about what depends on what, which is nice, but you do have to be thoughtful about it or you can summon the same footgun that got you with docker, just in smaller form. Because a nix derivation, while a box with nicely labeled inputs and output, is still a black box. If you insert a readme as an input to a derivation that does a build, nix will assume that the compiled binary depends on it and when you fix a typo in the readme and rebuild you'll end up with a duplicate binary build in the nix store despite the contents of the binary not actually depending on the text of the readme. > you can combine packages however you would like So this is true, more or less, but be aware that while nix lets you do this in ways that don't force needless duplication, it doesn't force you to avoid that duplication. Things carelessly packaged with nix can easily recreate the problem you mentioned with docker.
- justincormack 11mo agoThe problem is that whiteouts are not commutative. If the layers you build turn out to be bit for bit identical the layers will be shared anyway, but its much mroe complex than Nix where the composition operation is commutative.
- d3Xt3r 11mo ago> If you insert a readme as an input to a derivation that does a build, nix will assume that the compiled binary depends on it and when you fix a typo in the readme and rebuild you'll end up with a duplicate binary build in the nix store despite the contents of the binary not actually depending on the text of the readme. One my issues with Nix is the black box that is the store, and maybe it's just my system, but over time I find it full of redundant files / orphans and no obvious way to flatten it or clean it safely without breaking something. I wonder how flox solves this.
- jeremy_flox 11mo agoBoth fair points. The README rebuild issue is a Nix hiccup we don't solve; our quantized catalog reduces cascading rebuilds from upstream churn, but input over-specification is still there. On store bloat: Flox makes it clearer what's in use (explicit environments vs. implicit dependencies), but you still need nix-collect-garbage. The store accumulates cruft, that's Nix reality, we haven't changed it.
- jeremy_flox 11mo agoJust to follow up on this, Flox puts packages in one group by default so they share dependencies, plus our quantized catalog means way less version spread than raw Nix. So I do think we still improve on the Nix story, here. We're also adding "stabilities" (downsample from daily to weekly/monthly snapshots) to reduce churn even more. Still need GC, but a lot fewer bags on trash day. .
- __MatrixMan__ 11mo agoHave you found that nix-store --gc breaks things, or are you concerned that it's not an aggressive enough garbage collection? When I have space problems, I run that and they're gone, then later I do it again. It could be that I'm just avoiding functionality that it breaks though.
- ronef 11mo agoYes, this hits the nail on the head. We’ve seen the same explosion in image size and rebuild complexity, especially with AI/ML workloads where Python + CUDA + random pip wheels + system libs = image bloat and massive rebuilds. With the Kubernetes shim, you can run the hash-pinned environments without building or pulling an image at all. It starts the pod with a stub, then activates the exact runtime from a node-local store.