6 ms·

What did you replace it with? I've been been using NixOS for a (non-critical) server which replaces another server that was "managed" by Ansible. Now that's it

What did you replace it with? I've been been using NixOS for a (non-critical) server which replaces another server that was "managed" by Ansible. Now that's it running nicely on NixOS, and while it's far from perfect, I'm really struggling to see how I would manage it in any other way. The experience is so much nicer now.

hamandcheese 2y ago
I'll echo this. I can't imaging going back. And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.
kstenerud 2y ago
I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible). I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb). I've basically gone full circle at this point: - Docker - LXD with Bash scripts - LXD with a ton of Python - NixOS - Proxmox with Ansible - Proxmox with Bash scripts (albeit much simpler and flatter than last time) Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch. One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real. Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things. Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.
- koyote 2y ago
  As someone who uses Proxmox without bash scripts and is scared of the day I have to re-work out all the config files I (often randomly through trial and error) changed: Do you have any tips on how to get started? Do you simply make the change and then paste the commands needed into a script? Also I assume you have a script to set up the (Proxmox) host machine? I also have quite a few Proxmox specific things that I had to change (e.g. GPU pass-through) which seems to break your "Now it doesn't matter what technology (container or VM) I use" advantage.
  - kstenerud 2y ago
    Edit: I've made the repo public: https://github.com/kstenerud/proxmox-containers https://github.com/kstenerud/proxmox-containers Here's one of my host setup files (run this immediately after installing Proxmox from the iso): DATA_SIZE=600GiB # Switch to free mode rm /etc/apt/sources.list.d/ceph.list rm /etc/apt/sources.list.d/pve-enterprise.list echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" >/etc/apt/sources.list.d/pve-no-subscription.list # Setup locale sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen locale-gen update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8 apt update apt dist-upgrade -y apt install -y software-properties-common add-apt-repository -y non-free apt install -y vainfo intel-media-va-driver-non-free lm-sensors apt install -y ansible pip python3-proxmoxer htop strace git # Setup user data area lvcreate -V ${DATA_SIZE} -T pve/data -n home_data mke2fs -t ext4 -O ^has_journal /dev/pve/home_data mkdir /home/data echo "/dev/pve/home_data /home/data ext4 rw,discard,relatime 0 2" >>/etc/fstab mkdir -p /mnt/containers/media echo "//media/media /mnt/containers/media cifs uid=100000,gid=100000,ro,guest,x-systemd.automount 0 0" >>/etc/fstab mkdir -p /mnt/containers/files echo "//files/files /mnt/containers/files cifs uid=100000,gid=100000,ro,credentials=/etc/samba/credentials,x-systemd.automount 0 0" >>/etc/fstab systemctl daemon-reload mount -a The host is kept VERY simple. Here's my basic Ubuntu container template: # ============ # Local config # ============ TEMPLATE_IMAGE="ubuntu-24.04-standard_24.04-2_amd64.tar.zst" INSTANCE_CT=200 INSTANCE_NAME=template-ubuntu INSTANCE_MEMORY=2048 # ====== # Script # ====== pveam download local ${TEMPLATE_IMAGE} || true pct create $INSTANCE_CT local:vztmpl/${TEMPLATE_IMAGE} \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --rootfs local-lvm:1 \ --net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 \ --ostype ubuntu \ --start 1 \ --timezone host \ --features nesting=1 \ --unprivileged 1 # Use a fast mirror pct exec $INSTANCE_CT -- sed -i 's/archive.ubuntu.com/ftp.uni-stuttgart.de/g' /etc/apt/sources.list # Set up locale pct exec $INSTANCE_CT -- sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen pct exec $INSTANCE_CT -- locale-gen pct exec $INSTANCE_CT -- update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8 # Bring everything up to date pct exec $INSTANCE_CT -- apt clean pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y pct exec $INSTANCE_CT -- apt install -y software-properties-common curl # Turn this into a template pct stop $INSTANCE_CT pct template $INSTANCE_CT Here's my Plex template (with GPU passthrough): # ============ # Local config # ============ TEMPLATE_CT=200 INSTANCE_CT=201 INSTANCE_NAME=template-plex INSTANCE_MEMORY=2048 # ====== # Script # ====== passthrough_gpu() { local instance_id="$1" echo 'lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file lxc.hook.pre-start: sh -c "chown 0:108 /dev/dri/renderD128" ' >> /etc/pve/lxc/${instance_id}.conf } pct clone $TEMPLATE_CT $INSTANCE_CT --full 1 pct resize $INSTANCE_CT rootfs 2G pct set $INSTANCE_CT \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 passthrough_gpu $INSTANCE_CT pct start $INSTANCE_CT pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y # Install Plex apt_add_key $INSTANCE_CT plexmediaserver https://downloads.plex.tv/plex-keys/PlexSign.key CD665CBA0E2F88B7373F7CB997203C7B3ADCA79D apt_add_repo $INSTANCE_CT plexmediaserver "https://downloads.plex.tv/repo/deb public main" pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt install -y plexmediaserver # Turn this into a template pct stop $INSTANCE_CT pct template $INSTANCE_CT Here's the Plex instance script (starts a Plex instance going from the Plex template): # ============ # Local config # ============ TEMPLATE_CT=201 INSTANCE_CT=20001 INSTANCE_NAME=plex INSTANCE_ADDRESS=99 INSTANCE_MEMORY=2048 # =========== # Host config # =========== HOST_DATA=/home/data HOST_BASE_UID=100000 HOST_BASE_GID=100000 # ====== # Script # ====== pct clone $TEMPLATE_CT $INSTANCE_CT --full 1 pct set $INSTANCE_CT \ --onboot 1 \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --net0 name=eth0,hwaddr=12:4B:53:00:00:${INSTANCE_ADDRESS},ip=dhcp,ip6=dhcp,bridge=vmbr0 # Mount the media dir pct set $INSTANCE_CT -mp0 /mnt/containers/media,mp=/media # Get the UID and GID of the plex user pct start $INSTANCE_CT PLEX_UID=$(pct exec $INSTANCE_CT -- id -u plex) PLEX_GID=$(pct exec $INSTANCE_CT -- id -g plex) PLEX_HOST_UID=$(($HOST_BASE_UID+$PLEX_UID)) PLEX_HOST_GID=$(($HOST_BASE_GID+$PLEX_GID)) pct stop $INSTANCE_CT # Mount the Plex config dir mkdir -p /home/data/${INSTANCE_NAME}/var-lib-plexmediaserver chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME} chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver pct set $INSTANCE_CT -mp1 ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver,mp=/var/lib/plexmediaserver pct start $INSTANCE_CT pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y IP_ADDR=$(pct exec $INSTANCE_CT -- hostname -I | xargs) set +x echo "===========================================================" echo "FOR FIRST TIME SETUP (no existing config):" echo echo "Go to https://www.plex.tv/claim/" echo "pct exec $INSTANCE_CT -- curl -X POST 'http://127.0.0.1:32400/myplex/claim?token=PASTE_TOKEN_HERE'" echo "Then, go to ${IP_ADDR}:32400" echo "==========================================================="

hamandcheese 2y ago

I'll echo this. I can't imaging going back. And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.

kstenerud 2y ago

I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible). I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb). I've basically gone full circle at this point: - Docker - LXD with Bash scripts - LXD with a ton of Python - NixOS - Proxmox with Ansible - Proxmox with Bash scripts (albeit much simpler and flatter than last time) Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch. One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real. Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things. Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.

koyote 2y ago
As someone who uses Proxmox without bash scripts and is scared of the day I have to re-work out all the config files I (often randomly through trial and error) changed: Do you have any tips on how to get started? Do you simply make the change and then paste the commands needed into a script? Also I assume you have a script to set up the (Proxmox) host machine? I also have quite a few Proxmox specific things that I had to change (e.g. GPU pass-through) which seems to break your "Now it doesn't matter what technology (container or VM) I use" advantage.
- kstenerud 2y ago
  Edit: I've made the repo public: https://github.com/kstenerud/proxmox-containers https://github.com/kstenerud/proxmox-containers Here's one of my host setup files (run this immediately after installing Proxmox from the iso): DATA_SIZE=600GiB # Switch to free mode rm /etc/apt/sources.list.d/ceph.list rm /etc/apt/sources.list.d/pve-enterprise.list echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" >/etc/apt/sources.list.d/pve-no-subscription.list # Setup locale sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen locale-gen update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8 apt update apt dist-upgrade -y apt install -y software-properties-common add-apt-repository -y non-free apt install -y vainfo intel-media-va-driver-non-free lm-sensors apt install -y ansible pip python3-proxmoxer htop strace git # Setup user data area lvcreate -V ${DATA_SIZE} -T pve/data -n home_data mke2fs -t ext4 -O ^has_journal /dev/pve/home_data mkdir /home/data echo "/dev/pve/home_data /home/data ext4 rw,discard,relatime 0 2" >>/etc/fstab mkdir -p /mnt/containers/media echo "//media/media /mnt/containers/media cifs uid=100000,gid=100000,ro,guest,x-systemd.automount 0 0" >>/etc/fstab mkdir -p /mnt/containers/files echo "//files/files /mnt/containers/files cifs uid=100000,gid=100000,ro,credentials=/etc/samba/credentials,x-systemd.automount 0 0" >>/etc/fstab systemctl daemon-reload mount -a The host is kept VERY simple. Here's my basic Ubuntu container template: # ============ # Local config # ============ TEMPLATE_IMAGE="ubuntu-24.04-standard_24.04-2_amd64.tar.zst" INSTANCE_CT=200 INSTANCE_NAME=template-ubuntu INSTANCE_MEMORY=2048 # ====== # Script # ====== pveam download local ${TEMPLATE_IMAGE} || true pct create $INSTANCE_CT local:vztmpl/${TEMPLATE_IMAGE} \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --rootfs local-lvm:1 \ --net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 \ --ostype ubuntu \ --start 1 \ --timezone host \ --features nesting=1 \ --unprivileged 1 # Use a fast mirror pct exec $INSTANCE_CT -- sed -i 's/archive.ubuntu.com/ftp.uni-stuttgart.de/g' /etc/apt/sources.list # Set up locale pct exec $INSTANCE_CT -- sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen pct exec $INSTANCE_CT -- locale-gen pct exec $INSTANCE_CT -- update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8 # Bring everything up to date pct exec $INSTANCE_CT -- apt clean pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y pct exec $INSTANCE_CT -- apt install -y software-properties-common curl # Turn this into a template pct stop $INSTANCE_CT pct template $INSTANCE_CT Here's my Plex template (with GPU passthrough): # ============ # Local config # ============ TEMPLATE_CT=200 INSTANCE_CT=201 INSTANCE_NAME=template-plex INSTANCE_MEMORY=2048 # ====== # Script # ====== passthrough_gpu() { local instance_id="$1" echo 'lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file lxc.hook.pre-start: sh -c "chown 0:108 /dev/dri/renderD128" ' >> /etc/pve/lxc/${instance_id}.conf } pct clone $TEMPLATE_CT $INSTANCE_CT --full 1 pct resize $INSTANCE_CT rootfs 2G pct set $INSTANCE_CT \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 passthrough_gpu $INSTANCE_CT pct start $INSTANCE_CT pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y # Install Plex apt_add_key $INSTANCE_CT plexmediaserver https://downloads.plex.tv/plex-keys/PlexSign.key CD665CBA0E2F88B7373F7CB997203C7B3ADCA79D apt_add_repo $INSTANCE_CT plexmediaserver "https://downloads.plex.tv/repo/deb public main" pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt install -y plexmediaserver # Turn this into a template pct stop $INSTANCE_CT pct template $INSTANCE_CT Here's the Plex instance script (starts a Plex instance going from the Plex template): # ============ # Local config # ============ TEMPLATE_CT=201 INSTANCE_CT=20001 INSTANCE_NAME=plex INSTANCE_ADDRESS=99 INSTANCE_MEMORY=2048 # =========== # Host config # =========== HOST_DATA=/home/data HOST_BASE_UID=100000 HOST_BASE_GID=100000 # ====== # Script # ====== pct clone $TEMPLATE_CT $INSTANCE_CT --full 1 pct set $INSTANCE_CT \ --onboot 1 \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --net0 name=eth0,hwaddr=12:4B:53:00:00:${INSTANCE_ADDRESS},ip=dhcp,ip6=dhcp,bridge=vmbr0 # Mount the media dir pct set $INSTANCE_CT -mp0 /mnt/containers/media,mp=/media # Get the UID and GID of the plex user pct start $INSTANCE_CT PLEX_UID=$(pct exec $INSTANCE_CT -- id -u plex) PLEX_GID=$(pct exec $INSTANCE_CT -- id -g plex) PLEX_HOST_UID=$(($HOST_BASE_UID+$PLEX_UID)) PLEX_HOST_GID=$(($HOST_BASE_GID+$PLEX_GID)) pct stop $INSTANCE_CT # Mount the Plex config dir mkdir -p /home/data/${INSTANCE_NAME}/var-lib-plexmediaserver chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME} chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver pct set $INSTANCE_CT -mp1 ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver,mp=/var/lib/plexmediaserver pct start $INSTANCE_CT pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y IP_ADDR=$(pct exec $INSTANCE_CT -- hostname -I | xargs) set +x echo "===========================================================" echo "FOR FIRST TIME SETUP (no existing config):" echo echo "Go to https://www.plex.tv/claim/" echo "pct exec $INSTANCE_CT -- curl -X POST 'http://127.0.0.1:32400/myplex/claim?token=PASTE_TOKEN_HERE'" echo "Then, go to ${IP_ADDR}:32400" echo "==========================================================="