4 ms·
We do it via ansible, we create a boot environment (bectl create something) mount it (bectl mount something), then ansible upgrades the os and|or pkgs in the ne
by cyberpunk 4y ago
We do it via ansible, we create a boot environment (bectl create something) mount it (bectl mount something), then ansible upgrades the os and|or pkgs in the new be, and reboots into it. BE choice at boot is provided by some pxe/DHCP magic we rigged up.
If a host fails to come back up, we boot back into the old environment, and someone from ops gets a ticket to try it again manually (rare, usually the OS upgrade works, but then fails the app healthcheck).
We manage 1000s of instances this way, and it's saved our skin lots of times.
We can't do that on linux where we have to depend on hypervisor snapshots (we run proxmox hypervisors mostly) which stop the world for memory state and cause some issues for us.
YMMV, of course.