4 ms·

I just finished moving all of my servers off NixOS and I'm finally breathing a sigh of relief. Deterministic systems are a cool idea, but we're just not there

• judofyr 2y ago
  What did you replace it with? I've been been using NixOS for a (non-critical) server which replaces another server that was "managed" by Ansible. Now that's it running nicely on NixOS, and while it's far from perfect, I'm really struggling to see how I would manage it in any other way. The experience is so much nicer now.
  • hamandcheese 2y ago
    I'll echo this. I can't imaging going back. And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.
  • kstenerud 2y ago
    I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible). I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb). I've basically gone full circle at this point: - Docker - LXD with Bash scripts - LXD with a ton of Python - NixOS - Proxmox with Ansible - Proxmox with Bash scripts (albeit much simpler and flatter than last time) Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch. One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real. Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things. Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.
    • koyote 2y ago
      As someone who uses Proxmox without bash scripts and is scared of the day I have to re-work out all the config files I (often randomly through trial and error) changed: Do you have any tips on how to get started? Do you simply make the change and then paste the commands needed into a script? Also I assume you have a script to set up the (Proxmox) host machine? I also have quite a few Proxmox specific things that I had to change (e.g. GPU pass-through) which seems to break your "Now it doesn't matter what technology (container or VM) I use" advantage.
      • kstenerud 2y ago
        Edit: I've made the repo public: https://github.com/kstenerud/proxmox-containers https://github.com/kstenerud/proxmox-containers Here's one of my host setup files (run this immediately after installing Proxmox from the iso): DATA_SIZE=600GiB # Switch to free mode rm /etc/apt/sources.list.d/ceph.list rm /etc/apt/sources.list.d/pve-enterprise.list echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" >/etc/apt/sources.list.d/pve-no-subscription.list # Setup locale sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen locale-gen update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8 apt update apt dist-upgrade -y apt install -y software-properties-common add-apt-repository -y non-free apt install -y vainfo intel-media-va-driver-non-free lm-sensors apt install -y ansible pip python3-proxmoxer htop strace git # Setup user data area lvcreate -V ${DATA_SIZE} -T pve/data -n home_data mke2fs -t ext4 -O ^has_journal /dev/pve/home_data mkdir /home/data echo "/dev/pve/home_data /home/data ext4 rw,discard,relatime 0 2" >>/etc/fstab mkdir -p /mnt/containers/media echo "//media/media /mnt/containers/media cifs uid=100000,gid=100000,ro,guest,x-systemd.automount 0 0" >>/etc/fstab mkdir -p /mnt/containers/files echo "//files/files /mnt/containers/files cifs uid=100000,gid=100000,ro,credentials=/etc/samba/credentials,x-systemd.automount 0 0" >>/etc/fstab systemctl daemon-reload mount -a The host is kept VERY simple. Here's my basic Ubuntu container template: # ============ # Local config # ============ TEMPLATE_IMAGE="ubuntu-24.04-standard_24.04-2_amd64.tar.zst" INSTANCE_CT=200 INSTANCE_NAME=template-ubuntu INSTANCE_MEMORY=2048 # ====== # Script # ====== pveam download local ${TEMPLATE_IMAGE} || true pct create $INSTANCE_CT local:vztmpl/${TEMPLATE_IMAGE} \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --rootfs local-lvm:1 \ --net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 \ --ostype ubuntu \ --start 1 \ --timezone host \ --features nesting=1 \ --unprivileged 1 # Use a fast mirror pct exec $INSTANCE_CT -- sed -i 's/archive.ubuntu.com/ftp.uni-stuttgart.de/g' /etc/apt/sources.list # Set up locale pct exec $INSTANCE_CT -- sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen pct exec $INSTANCE_CT -- locale-gen pct exec $INSTANCE_CT -- update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8 # Bring everything up to date pct exec $INSTANCE_CT -- apt clean pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y pct exec $INSTANCE_CT -- apt install -y software-properties-common curl # Turn this into a template pct stop $INSTANCE_CT pct template $INSTANCE_CT Here's my Plex template (with GPU passthrough): # ============ # Local config # ============ TEMPLATE_CT=200 INSTANCE_CT=201 INSTANCE_NAME=template-plex INSTANCE_MEMORY=2048 # ====== # Script # ====== passthrough_gpu() { local instance_id="$1" echo 'lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file lxc.hook.pre-start: sh -c "chown 0:108 /dev/dri/renderD128" ' >> /etc/pve/lxc/${instance_id}.conf } pct clone $TEMPLATE_CT $INSTANCE_CT --full 1 pct resize $INSTANCE_CT rootfs 2G pct set $INSTANCE_CT \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 passthrough_gpu $INSTANCE_CT pct start $INSTANCE_CT pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y # Install Plex apt_add_key $INSTANCE_CT plexmediaserver https://downloads.plex.tv/plex-keys/PlexSign.key CD665CBA0E2F88B7373F7CB997203C7B3ADCA79D apt_add_repo $INSTANCE_CT plexmediaserver "https://downloads.plex.tv/repo/deb public main" pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt install -y plexmediaserver # Turn this into a template pct stop $INSTANCE_CT pct template $INSTANCE_CT Here's the Plex instance script (starts a Plex instance going from the Plex template): # ============ # Local config # ============ TEMPLATE_CT=201 INSTANCE_CT=20001 INSTANCE_NAME=plex INSTANCE_ADDRESS=99 INSTANCE_MEMORY=2048 # =========== # Host config # =========== HOST_DATA=/home/data HOST_BASE_UID=100000 HOST_BASE_GID=100000 # ====== # Script # ====== pct clone $TEMPLATE_CT $INSTANCE_CT --full 1 pct set $INSTANCE_CT \ --onboot 1 \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --net0 name=eth0,hwaddr=12:4B:53:00:00:${INSTANCE_ADDRESS},ip=dhcp,ip6=dhcp,bridge=vmbr0 # Mount the media dir pct set $INSTANCE_CT -mp0 /mnt/containers/media,mp=/media # Get the UID and GID of the plex user pct start $INSTANCE_CT PLEX_UID=$(pct exec $INSTANCE_CT -- id -u plex) PLEX_GID=$(pct exec $INSTANCE_CT -- id -g plex) PLEX_HOST_UID=$(($HOST_BASE_UID+$PLEX_UID)) PLEX_HOST_GID=$(($HOST_BASE_GID+$PLEX_GID)) pct stop $INSTANCE_CT # Mount the Plex config dir mkdir -p /home/data/${INSTANCE_NAME}/var-lib-plexmediaserver chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME} chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver pct set $INSTANCE_CT -mp1 ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver,mp=/var/lib/plexmediaserver pct start $INSTANCE_CT pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y IP_ADDR=$(pct exec $INSTANCE_CT -- hostname -I | xargs) set +x echo "===========================================================" echo "FOR FIRST TIME SETUP (no existing config):" echo echo "Go to https://www.plex.tv/claim/" echo "pct exec $INSTANCE_CT -- curl -X POST 'http://127.0.0.1:32400/myplex/claim?token=PASTE_TOKEN_HERE'" echo "Then, go to ${IP_ADDR}:32400" echo "==========================================================="
• nextaccountic 2y ago
  What didn't work in your NixOS servers?
• SuperSandro2000 2y ago
  Same experience with Debian. Preaching religious, unquestionable stability but a soon as you install an out of tree software, everything falls apart in your hands. Any Debian system I used fell apart on upgrades in some unrecoverable way and it was a giant pain in the ass. The last NixOS stable release I deployed in ~3 hours onto ~30 VMs with only minor issues and everything continued to just work.
  • kstenerud 2y ago
    I don't actually care about uncommon software stability on Debian. The hypervisor is Debian, so it runs very basic happy path software that always gets tested by the Debian team (and/or Proxmox team) for upgrade problems, and then all of my crazy-ass software runs in containers/VMs which are simple to destroy/rebuild if necessary. The Proxmox team ensures that the hypervisor stuff works, and now I don't have to worry about a basic change to a nix file resulting in "Unknown entity flibblefrazzle" coming from some random place 18 levels deep in the bowels of the package system/os. It really got to the point where I was afraid to touch anything anymore. Now I can run an up-to-date Plex! And Chrome Remote Desktop! Without spending 3 whole weekends knee deep in nix guts!
• Cyph0n 2y ago
  There is a middle ground imo: run your application/services as containers on top of NixOS. Granted, this doesn’t work for all cases.
  • kstenerud 2y ago
    That's basically what I was doing. But it still became too stressful to manage them. Proxmox has its own issues (they all do), but it's a much more inviting experience and nicer community.
    • Cyph0n 2y ago
      Fair enough. I actually run Proxmox too, but have most services running on NixOS as a VM. Best of both worlds haha :)
      • kstenerud 2y ago
        For me it came down to an epiphany: I'm doing all this careful setup on a tricky, esoteric platform to get reproducible builds, except actually, some of my containers have to run Debian because I can't build the software they run deterministically. So now I only sort-of have reproducible, immutable builds. So I'm jumping through all these hoops, stepping so far out of the norm that I become the effective maintainer and debugger of all of this, and ... Why was I doing this again? Oh yeah, deterministic builds and immutability. And I needed these ... why? Turns out I can get a similar effect on a mainstream platform with some scripts and a little bit of discipline.
      • JamesSwift 2y ago
        I do the same. Raw proxmox as the host, running mostly NixOS in the VMs
• anon291 2y ago
  As an opposing viewpoint, I migrated all my self hosted systems (physical machines in a data center) to use nixos and have truly reached self-hosting nirvana. Nothing breaks because I build everything before deployment, and when things do break during system build, it's fairly trivial to fix. I've successfully run all my own infrastructure on NixOS for about 5 years now. No downtime. Very easy to replace computers
• tonygiorgio 2y ago
  The OP’s article mostly seemed to be about disk space size, specifically when creating a bunch of micro vms. As a NixOS personal dev user, I’m curious about your insights here. Do you have other issues besides the disk space one? I’ve always wanted to do a NixOS server but I only know enough nix to get by on a dev machine. I’m worried about the complexity in deploying a live server.
  • kstenerud 2y ago
    NixOS is EXTREMELY opaque and obtuse. The documentation also, since everything assumes that you already know everything. I thought that I could just power through it, but there's just no end to the edge cases, bizarre magic, lack of useful error feedback (if any at all - the most common result is: nothing happens), and situations where you simply cannot do what you want to do. So you either conform to their model and live with the limitations (and spend countless weekends debugging your builds), or you give up and move on. I just got tired it all. I want to spend my time USING the computer, not setting it up. So now I use Debian, because everything includes a build for Debian and drivers for Debian. And the best part is: SOMEONE ELSE is maintaining it and keeping it current with the latest security fixes.