11 ms·

Deploying Containers on NixOS: A Guide

86 points by bkiran 2y ago

xyst 2y ago
I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster. Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades). I had actually planned to setup another remote cluster in my parents home (800 miles away), but ended up 86’ing that because their residential internet is the absolute worst. Currently cluster only accessible when behind VPN or on local network. Haven’t setup proper authN/authZ controls yet.
- yjftsjthsd-h 2y ago
  > Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Multiple worker VMs per physical host? Why?
  - xyst 2y ago
    Mostly for easier segregation of workloads. Some of the IoT shit that runs on this cluster I segment it off through k8s and network policies. Also most workloads wouldn’t need to use all cores or memory on that machine (one machine has 128G)
    - yjftsjthsd-h 2y ago
      > Mostly for easier segregation of workloads. Some of the IoT shit that runs on this cluster I segment it off through k8s and network policies. Depending on your threat model that could make sense. > Also most workloads wouldn’t need to use all cores or memory on that machine (one machine has 128G) What? Limiting CPU or memory use of a pod is one of kubernetes' core competencies; just tell it what each thing needs and it handles all that for you.
- Havoc 2y ago
  There is probably room for both. Planning to use a nix server as storage & DBs for my k8s cluster
  - bkiran 2y ago
    Nice, would love to know how it goes. What kind of storage and DB's are you planning to work with?
    - Havoc 2y ago
      Mostly trying to get rid of longhorn. I've found it to be a continuous source of troubles w/ etcd sync & IO issues. Current iteration of cluster is all optane so might give longhorn another go, but still want to move storage off cluster. Especially source repo needs to live on some sort of striped zfs array. Ordered one of these quad nvme NAS things [0] so that's probably going to be storage. Either nix or proxmox...not decided. Also still a bit fuzzy on what best game plan on PVs is. Minio/s3 or nfs appear to be options. DB...just the usual suspects...mainly postgres for gitea I think. Mongo for dev stuff. Used to vanilla proxmox/docker/lxc so this is all unchartered territory for me. [0] https://vi.aliexpress.com/item/1005007188570776.html https://vi.aliexpress.com/item/1005007188570776.html
      - bkiran 2y ago
        I could never get around on longhorn or any of the other "lightweight" K8 distros. I guess that's why I spent time setting up NixOS and writing this post. Okay nice, seems like various storage services for a home lab setup. Seems like a cool project. Especially if you can distribute it across all those NAS's
- bkiran 2y ago
  To each their own. When K8's is managed it's awesome. I would like to do without the headache of dealing with K8's installation or some orchestrator layer. I can "schedule" my on application instances at the size I'm working with. Also, I run this setup on cloud vm(multiple actually). So it's not restricted to running on a single machine running in a closet.
Cyph0n 2y ago
NixOS OCI containers are a powerful way to run apps that are not packaged for Nix or NixOS. And because they’re ultimately systemd units, you can customize virtually everything without having to fiddle with the container runtime. If you want to take this a step further and migrate or run a Compose project on NixOS, I maintain a tool that makes this pretty easy to do :) https://github.com/aksiksi/compose2nix https://github.com/aksiksi/compose2nix
- schlarpc 2y ago
  Thanks for the link, I've got a bunch of hacky code for running Immich's compose under Nix that this might be able to replace.
  - Cyph0n 2y ago
    Coincidentally, I use Immich as an example for my demo in the overview video: https://youtu.be/hCAFyzJ81Pg?t=281 https://youtu.be/hCAFyzJ81Pg?t=281
lobochrome 2y ago
Could someone please help me understand the benefit of Nix versus just using docker-compose? Also please don't tell me k8s it. Way too heavy. I run a few applications on a souped-up Raspi (paperless-ngx, jellyfin, some postgres dbs). I can't see why this would improve things. :) Honest question.
- Havoc 2y ago
  Nix is more about package management (and a OS) while docker is more container focused. But because the individual software on nix is so well separated / encapsulated it carries similar benefits to containers. So they’re different but with overlap
- cloudripper 2y ago
  Nix is a language, package manager, and OS. This post discusses NixOS. While docker-compose allows you to compose your containers with a yaml/Dockerfiles, NixOS allows you to compose the system that all of your containers run on (from userspace down to kernel selection/configs, file system, etc), as well as your containers - all in a declarative .nix file. That .nix file can be used to spin up any number of identitically configured systems. It's also reproducible, in that you can specify the sources (refined to a specific commit if you prefer) for any and all packages on the system - and build them with Nix within a sandboxed environment protecting dependencies and env configurations (Nix is also a powerful build system).
  - bkiran 2y ago
    Agreed, reproducibility is a huge benefit. Being able to spin up a new machine quickly with a config you already know works is an awesome feeling.
- colordrops 2y ago
  I run several services on my home server. For those that are well packaged for Nix, I just use those. For ones that are poorly packaged or not available, I use oci-containers. They all run as systemd services and operate the same way, so it's a consistent interface.
- 2y ago
Dedime 2y ago
I've used docker-compose, k8s, and NixOS myself, being from a similar technical background as the author, but I find myself disagreeing with some of the author's opinions on the technologies. They're not wrong of course, but I've had different experiences. k8s: Installing and using k8s can indeed be a nightmare. In my job, we use Azure, so it's not so bad since launching a cluster is mostly handled by Azure. Setting it up for personal use is less fun. The mountains of YAML you can end up using to deploy even semi-complex services is even less fun. That being said, I've been wanting to use it for a personal project (distributed cluster using cloud VPSs and bare metal at home connected using WireGuard). I just wish it was smaller and faster. Most guides recommend 2Gb of RAM and 2 CPU for the smallest of small deployments. docker-compose: I actually love docker-compose for my personal stuff. I have an intel NUC hosting homeassistant, pihole, caddy, deluge, jellyfin and a handful of other stuff. Everything lives in a series of folders for each service. Backups (both data and code via git), disaster recovery, and just general reasoning about of it is so easy. The docker-compose files are small and easy to read. I also find docker-compose to be about as immutable as you'd like it - version control your docker-compose directories, pin your image SHAs, and you're in a good place. Or don't, and it will still work pretty well. NixOS: I've done it. I installed it on my Framework Laptop since it was all the rage at the t ime. I lived with it for about a year, and it was okay - for day-to-day use - AFTER I had spent weeks learning how to use NixOS. I will freely admit it's an awesome technology in some respects. But the documentation just was not there. It was way too hard to learn how to do even basic tasks. I thought nix flakes might be the "aha" moment I was looking for, but I gave up trying to get that to work after a couple of days of troubleshooting. Don't even get me started on trying to package up something from scratch. As a random example, I googled "packaging python for nix" and the top result [1] is just way too complex for something that should be pretty simple. The example includes some abomination of a .nix file with inline bash and python scripts. I don't really know where I'm going with this. I really do like the idea of NixOS. I just wish it was much, much easier to reason about. Curious to hear what others make of this. [1](https://nixos.wiki/wiki/Packaging/Python https://nixos.wiki/wiki/Packaging/Python)
- LorenzoGood 2y ago
  > As a random example, I googled "packaging python for nix" and the top result [1] is just way too complex for something that should be pretty simple. Aware that this is more of a critique about the documentation situation, as opposed to the python packaging situation. However, there is poetry2nix[1]. Which makes packaging look something like this: myPythonApp = mkPoetryApplication { projectDir = ./.; }; [1](https://github.com/nix-community/poetry2nix)(although https://github.com/nix-community/poetry2nix)(although it has been merged into nixpkgs master)
aliasxneo 2y ago
> Private Registry, No Problem Minor nit: configuration is not the hard part. The hard part is getting "/root/registry-password.txt" onto the NixOS machine in the first place. I mean, you could just scp it I guess, but why spend hours tuning a NixOS config that requires you to manually do stuff in the end? I'm aware of all of the NixOS "secret management" methods out there but I found none of them satisfying back when I was still using NixOS.
- Macha 2y ago
  If you're on e.g. AWS or GCP, you can pull them from the cloud's IAM service. If you're on kubernetes, you can use k8s secrets. If you have e.g. vault you can use that. It's really only deploying on unmanaged servers where this comes around, but it's also somewhat of a hard problem. Like you don't (or shouldn't) bake secrets into disk/VM/container images, so once you're no longer building on some managed layer then you do have to figure out bootstrapping yourself.
- bkiran 2y ago
  Yeah I agree it's manual but it takes about 5 minutes to SCP the password onto the machine. I have some playbooks I setup to creating a new machine. All in all it takes about 10 min to get it up and running. Maybe not instant but at the moment I don't need anything else.
- LorenzoGood 2y ago
  You can also use something like agenix or sops-nix to deploy the secrets encrypted to the machine in the system closure.
  - Macha 2y ago
    These are a little chicken and egg as you need the system's host key for that. If you want to use a signed host key, you need to deploy that, otherwise if you just let it generate a host key you're in TOFU territory
    - LorenzoGood 2y ago
rebeccaskinner 2y ago
I think it's great to document this, and some people are going to prefer working with containers no matter what. That said, personally I've moved away from it and these days I just use nixos modules and run all of the services on my home server directly on the host. You don't get the same isolation that you might get with proper containers, and that might be an issue for production machines, but I find the simplicity is a win for a home server.
- lolinder 2y ago
  If you're after simplicity, it's hard to beat docker compose for self hosting stuff. I run NixOS on my laptop and routinely run into things that aren't yet packaged for NixOS, but I've yet to come across a project where I had to write my own dockerfile.
  - rebeccaskinner 2y ago
    Personally I’ve always found working with docker to be pretty frustrating, especially dealing with docker compose. Most of what I run on my server is at least in nixpkgs already if not already a NixOS module, and I just find it less frustrating to write nix than to deal with Docker. That said, I know nix pretty well at this point and I would probably have a different opinion if I hadn’t spent so much time learning nix.
    - lolinder 2y ago
      > That said, I know nix pretty well at this point and I would probably have a different opinion if I hadn’t spent so much time learning nix. Yeah, I'm in exactly the opposite boat—I've been using docker professionally and at home for 5+ years now and know it very well, while nix is still very new to me! There's probably no way to objectively tell which one we'd have preferred if we started in the opposite order.
deleted 2y ago
[deleted]
aitchnyu 2y ago
Tangential, has anybody replaced private container registries with a place to upload and download container tarballs?

86 points by bkiran 2y ago

xyst 2y ago

I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster. Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades). I had actually planned to setup another remote cluster in my parents home (800 miles away), but ended up 86’ing that because their residential internet is the absolute worst. Currently cluster only accessible when behind VPN or on local network. Haven’t setup proper authN/authZ controls yet.

yjftsjthsd-h 2y ago
> Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Multiple worker VMs per physical host? Why?
- xyst 2y ago
  Mostly for easier segregation of workloads. Some of the IoT shit that runs on this cluster I segment it off through k8s and network policies. Also most workloads wouldn’t need to use all cores or memory on that machine (one machine has 128G)
  - yjftsjthsd-h 2y ago
    > Mostly for easier segregation of workloads. Some of the IoT shit that runs on this cluster I segment it off through k8s and network policies. Depending on your threat model that could make sense. > Also most workloads wouldn’t need to use all cores or memory on that machine (one machine has 128G) What? Limiting CPU or memory use of a pod is one of kubernetes' core competencies; just tell it what each thing needs and it handles all that for you.
Havoc 2y ago
There is probably room for both. Planning to use a nix server as storage & DBs for my k8s cluster
- bkiran 2y ago
  Nice, would love to know how it goes. What kind of storage and DB's are you planning to work with?
  - Havoc 2y ago
    Mostly trying to get rid of longhorn. I've found it to be a continuous source of troubles w/ etcd sync & IO issues. Current iteration of cluster is all optane so might give longhorn another go, but still want to move storage off cluster. Especially source repo needs to live on some sort of striped zfs array. Ordered one of these quad nvme NAS things [0] so that's probably going to be storage. Either nix or proxmox...not decided. Also still a bit fuzzy on what best game plan on PVs is. Minio/s3 or nfs appear to be options. DB...just the usual suspects...mainly postgres for gitea I think. Mongo for dev stuff. Used to vanilla proxmox/docker/lxc so this is all unchartered territory for me. [0] https://vi.aliexpress.com/item/1005007188570776.html https://vi.aliexpress.com/item/1005007188570776.html
    - bkiran 2y ago
      I could never get around on longhorn or any of the other "lightweight" K8 distros. I guess that's why I spent time setting up NixOS and writing this post. Okay nice, seems like various storage services for a home lab setup. Seems like a cool project. Especially if you can distribute it across all those NAS's
bkiran 2y ago
To each their own. When K8's is managed it's awesome. I would like to do without the headache of dealing with K8's installation or some orchestrator layer. I can "schedule" my on application instances at the size I'm working with. Also, I run this setup on cloud vm(multiple actually). So it's not restricted to running on a single machine running in a closet.

Cyph0n 2y ago

NixOS OCI containers are a powerful way to run apps that are not packaged for Nix or NixOS. And because they’re ultimately systemd units, you can customize virtually everything without having to fiddle with the container runtime. If you want to take this a step further and migrate or run a Compose project on NixOS, I maintain a tool that makes this pretty easy to do :) https://github.com/aksiksi/compose2nix https://github.com/aksiksi/compose2nix

schlarpc 2y ago
Thanks for the link, I've got a bunch of hacky code for running Immich's compose under Nix that this might be able to replace.
- Cyph0n 2y ago
  Coincidentally, I use Immich as an example for my demo in the overview video: https://youtu.be/hCAFyzJ81Pg?t=281 https://youtu.be/hCAFyzJ81Pg?t=281

lobochrome 2y ago

Could someone please help me understand the benefit of Nix versus just using docker-compose? Also please don't tell me k8s it. Way too heavy. I run a few applications on a souped-up Raspi (paperless-ngx, jellyfin, some postgres dbs). I can't see why this would improve things. :) Honest question.

Havoc 2y ago
Nix is more about package management (and a OS) while docker is more container focused. But because the individual software on nix is so well separated / encapsulated it carries similar benefits to containers. So they’re different but with overlap
cloudripper 2y ago
Nix is a language, package manager, and OS. This post discusses NixOS. While docker-compose allows you to compose your containers with a yaml/Dockerfiles, NixOS allows you to compose the system that all of your containers run on (from userspace down to kernel selection/configs, file system, etc), as well as your containers - all in a declarative .nix file. That .nix file can be used to spin up any number of identitically configured systems. It's also reproducible, in that you can specify the sources (refined to a specific commit if you prefer) for any and all packages on the system - and build them with Nix within a sandboxed environment protecting dependencies and env configurations (Nix is also a powerful build system).
- bkiran 2y ago
  Agreed, reproducibility is a huge benefit. Being able to spin up a new machine quickly with a config you already know works is an awesome feeling.
colordrops 2y ago
I run several services on my home server. For those that are well packaged for Nix, I just use those. For ones that are poorly packaged or not available, I use oci-containers. They all run as systemd services and operate the same way, so it's a consistent interface.
2y ago

Dedime 2y ago

I've used docker-compose, k8s, and NixOS myself, being from a similar technical background as the author, but I find myself disagreeing with some of the author's opinions on the technologies. They're not wrong of course, but I've had different experiences. k8s: Installing and using k8s can indeed be a nightmare. In my job, we use Azure, so it's not so bad since launching a cluster is mostly handled by Azure. Setting it up for personal use is less fun. The mountains of YAML you can end up using to deploy even semi-complex services is even less fun. That being said, I've been wanting to use it for a personal project (distributed cluster using cloud VPSs and bare metal at home connected using WireGuard). I just wish it was smaller and faster. Most guides recommend 2Gb of RAM and 2 CPU for the smallest of small deployments. docker-compose: I actually love docker-compose for my personal stuff. I have an intel NUC hosting homeassistant, pihole, caddy, deluge, jellyfin and a handful of other stuff. Everything lives in a series of folders for each service. Backups (both data and code via git), disaster recovery, and just general reasoning about of it is so easy. The docker-compose files are small and easy to read. I also find docker-compose to be about as immutable as you'd like it - version control your docker-compose directories, pin your image SHAs, and you're in a good place. Or don't, and it will still work pretty well. NixOS: I've done it. I installed it on my Framework Laptop since it was all the rage at the t ime. I lived with it for about a year, and it was okay - for day-to-day use - AFTER I had spent weeks learning how to use NixOS. I will freely admit it's an awesome technology in some respects. But the documentation just was not there. It was way too hard to learn how to do even basic tasks. I thought nix flakes might be the "aha" moment I was looking for, but I gave up trying to get that to work after a couple of days of troubleshooting. Don't even get me started on trying to package up something from scratch. As a random example, I googled "packaging python for nix" and the top result [1] is just way too complex for something that should be pretty simple. The example includes some abomination of a .nix file with inline bash and python scripts. I don't really know where I'm going with this. I really do like the idea of NixOS. I just wish it was much, much easier to reason about. Curious to hear what others make of this. [1](https://nixos.wiki/wiki/Packaging/Python https://nixos.wiki/wiki/Packaging/Python)

LorenzoGood 2y ago
> As a random example, I googled "packaging python for nix" and the top result [1] is just way too complex for something that should be pretty simple. Aware that this is more of a critique about the documentation situation, as opposed to the python packaging situation. However, there is poetry2nix[1]. Which makes packaging look something like this: myPythonApp = mkPoetryApplication { projectDir = ./.; }; [1](https://github.com/nix-community/poetry2nix)(although https://github.com/nix-community/poetry2nix)(although it has been merged into nixpkgs master)

aliasxneo 2y ago

> Private Registry, No Problem Minor nit: configuration is not the hard part. The hard part is getting "/root/registry-password.txt" onto the NixOS machine in the first place. I mean, you could just scp it I guess, but why spend hours tuning a NixOS config that requires you to manually do stuff in the end? I'm aware of all of the NixOS "secret management" methods out there but I found none of them satisfying back when I was still using NixOS.

Macha 2y ago
If you're on e.g. AWS or GCP, you can pull them from the cloud's IAM service. If you're on kubernetes, you can use k8s secrets. If you have e.g. vault you can use that. It's really only deploying on unmanaged servers where this comes around, but it's also somewhat of a hard problem. Like you don't (or shouldn't) bake secrets into disk/VM/container images, so once you're no longer building on some managed layer then you do have to figure out bootstrapping yourself.
bkiran 2y ago
Yeah I agree it's manual but it takes about 5 minutes to SCP the password onto the machine. I have some playbooks I setup to creating a new machine. All in all it takes about 10 min to get it up and running. Maybe not instant but at the moment I don't need anything else.
LorenzoGood 2y ago
You can also use something like agenix or sops-nix to deploy the secrets encrypted to the machine in the system closure.
- Macha 2y ago
  These are a little chicken and egg as you need the system's host key for that. If you want to use a signed host key, you need to deploy that, otherwise if you just let it generate a host key you're in TOFU territory
  - LorenzoGood 2y ago

rebeccaskinner 2y ago

I think it's great to document this, and some people are going to prefer working with containers no matter what. That said, personally I've moved away from it and these days I just use nixos modules and run all of the services on my home server directly on the host. You don't get the same isolation that you might get with proper containers, and that might be an issue for production machines, but I find the simplicity is a win for a home server.

lolinder 2y ago
If you're after simplicity, it's hard to beat docker compose for self hosting stuff. I run NixOS on my laptop and routinely run into things that aren't yet packaged for NixOS, but I've yet to come across a project where I had to write my own dockerfile.
- rebeccaskinner 2y ago
  Personally I’ve always found working with docker to be pretty frustrating, especially dealing with docker compose. Most of what I run on my server is at least in nixpkgs already if not already a NixOS module, and I just find it less frustrating to write nix than to deal with Docker. That said, I know nix pretty well at this point and I would probably have a different opinion if I hadn’t spent so much time learning nix.
  - lolinder 2y ago
    > That said, I know nix pretty well at this point and I would probably have a different opinion if I hadn’t spent so much time learning nix. Yeah, I'm in exactly the opposite boat—I've been using docker professionally and at home for 5+ years now and know it very well, while nix is still very new to me! There's probably no way to objectively tell which one we'd have preferred if we started in the opposite order.

deleted 2y ago

[deleted]

aitchnyu 2y ago

Tangential, has anybody replaced private container registries with a place to upload and download container tarballs?