3 ms·

We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service. I have tested with EKS, and you should be a

We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service. I have tested with EKS, and you should be able to use nix-snapshotter with GKE too. May be able to put together some docs for that later. In the meantime, see this for running a different snapshotter on EKS: https://blog.realvarez.com/using-estargz-to-reduce-container-startup-time-on-amazon-eks/ https://blog.realvarez.com/using-estargz-to-reduce-container...

solatic 3y ago
I would definitely appreciate seeing some docs with quick pointers on setting that up for EKS at least, if not also GKE!
mikepurvis 3y ago
I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time, so for large closures you give up a lot of the benefit of the cluster being able to cache layers (as in, say, a nix2container- or nixery-style image) and achieve very fast subsequent startups. I'm a k8s novice, but would there be a way to run the nix daemon/cache in a semi-persistent pod on each node, and then "attach" it to the actual worker pods?
- rgoulter 3y ago
  > I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time I got the impression that it uses the node's nix daemon. From the project readme's FAQ https://github.com/pdtpartners/nix-snapshotter#faq https://github.com/pdtpartners/nix-snapshotter#faq > What's the difference between this and a nix-in-docker? > If you run nix inside a container (e.g. nixos/nix or nixpkgs/nix-flake) then you are indeed fetching packages using the Nix store. However, each container will have its own Nix store instead of de-duplicating at the host level. > nix-snapshotter is intended to live on the host system (sibling to containerd and/or kubelet) so that multiple containers running different images can share the underlying packages from the same Nix store.
  - mikepurvis 3y ago
    Yeah, I think that's the "main" idea of it, but it was also mentioned that this can work in GKE/EKS if you start a new daemon each time.
    - hinshun 3y ago
      The EKS/GKE integration involves modifying the host that the kubelet lives and adding nix-snapshotter as a sibling host service. I didn’t mean running nix-snapshotter as a Kubernetes resource because then there’s a chicken & egg problem. Kubernetes needs nix-snapshotter image service to resolve the nix-snapshotter image.

solatic 3y ago

I would definitely appreciate seeing some docs with quick pointers on setting that up for EKS at least, if not also GKE!

mikepurvis 3y ago

I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time, so for large closures you give up a lot of the benefit of the cluster being able to cache layers (as in, say, a nix2container- or nixery-style image) and achieve very fast subsequent startups. I'm a k8s novice, but would there be a way to run the nix daemon/cache in a semi-persistent pod on each node, and then "attach" it to the actual worker pods?

rgoulter 3y ago
> I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time I got the impression that it uses the node's nix daemon. From the project readme's FAQ https://github.com/pdtpartners/nix-snapshotter#faq https://github.com/pdtpartners/nix-snapshotter#faq > What's the difference between this and a nix-in-docker? > If you run nix inside a container (e.g. nixos/nix or nixpkgs/nix-flake) then you are indeed fetching packages using the Nix store. However, each container will have its own Nix store instead of de-duplicating at the host level. > nix-snapshotter is intended to live on the host system (sibling to containerd and/or kubelet) so that multiple containers running different images can share the underlying packages from the same Nix store.
- mikepurvis 3y ago
  Yeah, I think that's the "main" idea of it, but it was also mentioned that this can work in GKE/EKS if you start a new daemon each time.
  - hinshun 3y ago
    The EKS/GKE integration involves modifying the host that the kubelet lives and adding nix-snapshotter as a sibling host service. I didn’t mean running nix-snapshotter as a Kubernetes resource because then there’s a chicken & egg problem. Kubernetes needs nix-snapshotter image service to resolve the nix-snapshotter image.