3 ms·

> develop a container runtime that would allow a Kubernetes node to run pods that specify Nix expressions directly in the image field https://github.com/google

by tazjin 7y ago

> develop a container runtime that would allow a Kubernetes node to run pods that specify Nix expressions directly in the image field https://github.com/google/nixery https://github.com/google/nixery I'm the author of this, and here is a talk about it: https://www.youtube.com/watch?v=pOI9H4oeXqA https://www.youtube.com/watch?v=pOI9H4oeXqA

solatic 7y ago
I'm familiar with Nixery and I think it's a really cool project. It's extremely close to what I'd like but it's not quite it - it requires either relying on nixery.dev to be online (unacceptable for production considering there are no availability guarantees) or running my own instance (which essentially means that I'm maintaining a type of registry). Why is there a need for an image registry? Part of the beauty of Nix is that Nix benefits from remote binary caches, but they are not required. Why not have a container runtime that, instead of downloading image layers, instead fetches from a Nix binary cache if possible and builds from source if not (with the caveat that production nodes should basically never be building from source)? (Also Nixery is GCE-only and we're on AWS but leave that aside).
- ecnahc515 7y ago
  Not OP, but a registry works anywhere, whereas a custom runtime doesn't (managed node groups in EKS, GKE, AKS etc). Additionally, once you use a custom runtime, now you have to deal with multiple runtimes in your cluster. You can no longer easily just run pods, you have to ensure they run on the nodes with the runtime for the images you want.
- tazjin 7y ago
  > which essentially means that I'm maintaining a type of registry Mhm, there's no state that can't be thrown away and recreated, so I'd argue the overhead of running it is much lower than a full-blown registry. > Why not have a container runtime that, instead of downloading image layers, instead fetches from a Nix binary cache It depends on where you want to do this - Kubernetes for example has lots of opinions about images and how they're downloaded, so just replacing the runtime wouldn't be enough. Nixery is an incremental step towards the end-goal, but there's a lot of mindset shifting that needs to happen first I think. > Also Nixery is GCE-only Nope, you can use a disk as the storage backend and then there's no dependency on GCS. S3 support would also be relatively easy to add by just implementing this interface: https://github.com/google/nixery/blob/master/storage/storage.go#L28 https://github.com/google/nixery/blob/master/storage/storage...

by tazjin 7y ago

solatic 7y ago

I'm familiar with Nixery and I think it's a really cool project. It's extremely close to what I'd like but it's not quite it - it requires either relying on nixery.dev to be online (unacceptable for production considering there are no availability guarantees) or running my own instance (which essentially means that I'm maintaining a type of registry). Why is there a need for an image registry? Part of the beauty of Nix is that Nix benefits from remote binary caches, but they are not required. Why not have a container runtime that, instead of downloading image layers, instead fetches from a Nix binary cache if possible and builds from source if not (with the caveat that production nodes should basically never be building from source)? (Also Nixery is GCE-only and we're on AWS but leave that aside).

ecnahc515 7y ago
Not OP, but a registry works anywhere, whereas a custom runtime doesn't (managed node groups in EKS, GKE, AKS etc). Additionally, once you use a custom runtime, now you have to deal with multiple runtimes in your cluster. You can no longer easily just run pods, you have to ensure they run on the nodes with the runtime for the images you want.
tazjin 7y ago
> which essentially means that I'm maintaining a type of registry Mhm, there's no state that can't be thrown away and recreated, so I'd argue the overhead of running it is much lower than a full-blown registry. > Why not have a container runtime that, instead of downloading image layers, instead fetches from a Nix binary cache It depends on where you want to do this - Kubernetes for example has lots of opinions about images and how they're downloaded, so just replacing the runtime wouldn't be enough. Nixery is an incremental step towards the end-goal, but there's a lot of mindset shifting that needs to happen first I think. > Also Nixery is GCE-only Nope, you can use a disk as the storage backend and then there's no dependency on GCS. S3 support would also be relatively easy to add by just implementing this interface: https://github.com/google/nixery/blob/master/storage/storage.go#L28 https://github.com/google/nixery/blob/master/storage/storage...