8 ms·

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubun

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed at boot, since the containers share the kernel with the host. At that point, you can't SSH in and have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. And then of course if you revert the kernel upgrade, you were once again vulnerable to whatever problem the security update was fixing...

Spivak 4y ago
I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.
- yjftsjthsd-h 4y ago
  > don’t use unattended upgrades > Build your images in CI job I know container images should generally be immutable, but I would expect unattended upgrades to be mostly used on the host, not in a container, in which that management system doesn't really work (unless you're doing VMs where you can deploy immutable root images to the VMs as well, or some fun bare metal + PXE combination).
  - ec109685 4y ago
    Idea is that your deploy step should handle both deploying code as well as upgrading OS, so all changes go through same pipeline.
  - CuriousCosmic 4y ago
    alternatively I suppose depending on the size of your operation, you should consider having a dummy prod using at least one of each of the servers in your environment and using that to validate host upgrades. after that you can push an unattended upgrade via a self-hosted package+upgrade server. Let things be automatic to the maximum degree possible but give yourself a single hard human checkpoint and some minimum level of validation in a dummy environment first.
  - Spivak 4y ago
    > or some fun bare metal + PXE combination This is actually what I implemented for our hypervisor tier, it’s not as scary as it sounds. I could legit completely rebuild our entire stack down to the metal in about 3 hours. Kick off a new hypervisor version, the inactive side PXE boots all the nodes, installs and configures a Proxmox cluster, slaves itself to our Ceph cluster, and then either does a hot migration of all the VMs or kicks off a full deploy which rebuilds all the infra (Consul, Rabbit, Redis, LDAP, Elastic, PowerDNS, etc) along with the app servers. The hardest part (which really isn’t) is maintaining the clusters across the blue/green sides. With this setup our only mutable infrastructure was our Ceph cluster (because replacing OSDs takes unacceptably long) and our DB (for performance the writers lived on dedicated servers, the read replicas lived on the VMs.).
- nix23 4y ago
  >I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs Some years ago everyone said the same about windows-servers ;)
- markstos 4y ago
  Sorry, not my experience. My experience has been that by the time I notice some serious vulnerability is in the news, my servers have already patched themselves. I have never "hated life" or had a "hard to find and undo bug" due to automatic security patching. I pretty quickly found what caused this and had a clear path to resolution. This is the first security update that caused a boot failure in about a decade. It was bad, but it didn't change my mind about unattended-upgrades. My takeaway that if that maybe I should have upgraded my 20.04 servers to 22.04 server sooner.
  - Spivak 4y ago
    You’re conflating unattended-upgrades (server mutability, hard to roll back) with automated patching in general. Do automated patching but also run the changes though your CI so you can catch breaking changes and roll them out in a way that’s easy to debug (you can diff images) and revert. I bet when you update your software dependencies you run those changes through your tests but your OS is a giant pile of code that usually gets updated differently and independently because mostly historical reasons.
    - markstos 4y ago
      > I bet when you update your software dependencies you run those changes through your tests but your OS is a giant pile of code that usually gets updated differently and independently because mostly historical reasons. Close. We are moving towards defining our server states through Ansible, but the project is not close to completion. Perhaps once that's further along, we could use Ansible Molecule + CI to test a new server state when there's a new patch available, but that's not an option on the table today. The system we had in place for /today/ worked: Lower priority or redundant servers were set to auto-reboot after applying security updates, while other critical servers require manual reboot at low-risk times. By then, the patch has already been tested on lower-risk servers. As a result, this issue caused no user-visible downtime for us, and due to the staggered runs of unattended-upgrades affected a minimal number of servers. And this was the first time in 10+ years that something like this happened and we have to choose to write to prioritize spending our process-improvement time based on likelihood and impact.
QuentinM 4y ago
Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calculation overflows after the window went to zero due to a temporary slow consumer - freezing the window size to a few bytes only instead of getting a prompt full window recovery. Not to mention we’ve also had our fair share of production triple faults from bugs in the Intel firmware patches for Spectre, which took weeks to investigate & fix between ourselves struggling to keep our exchange up & running, Intel, and AWS. And that is why there’s value in the CoreOS/ContainerLinux-like solutions we designed & implemented nearly a decade ago now. Being able to promptly rollback any kernel/system/package upgrades at once - either manually or either after it’s detected a few panics in quick successions is actually quite awesome. Not to mention the slow update rollout strategy baked into the Omaha controller. But the reality is that the what-ifs are always the hardest to market, nearly always after-thoughts and with fast-spiking/fast-decaying traction after major events.
- stingraycharles 4y ago
  It really seems like there’s no good non-redhat (but still “production capable”) alternative to CoreOS nowadays, right? It’s pretty much Fedora / Redhat CoreOS or go directly to things such as k3os?
  - georgyo 4y ago
    The rancher stack is pretty amazing. Elemental is pretty close to coreos: https://github.com/rancher/elemental/ https://github.com/rancher/elemental/ They even have a way to build arbitrary os images: https://github.com/rancher/elemental-toolkit https://github.com/rancher/elemental-toolkit It's pretty great
  - Already__Taken 4y ago
    k3os is in a dieing limbo, now is the time to get some interest in using stuff like it
ConstantVigil 4y ago
And while I haven't had this happen to me yet; the fear of something like this or even worse is why I try to stay one step behind the update paths on linux distros. Security patches matter, but I'm no one important, so I should be fine to wait a week or month... Anyone else who is important though... servers for example...
_y5hn 4y ago
Wouldn't rollback of kernel be a choice in grub menu? It's pretty standard for all distros to have that choice.
- withinboredom 4y ago
  That usually requires physical access to the server to select it during boot.
  - taspeotis 4y ago
    Or a real server with Lights Out Management.
    - withinboredom 4y ago
      That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.
      - phillu 4y ago
        That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server. Oh well, I guess experiences differ.
        withinboredom 4y ago
        My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?
akx 4y ago
> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.
- capableweb 4y ago
  Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
  - jacquesm 4y ago
    That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
  - dspillett 4y ago
    If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu? Failing that you could try the “single” directive and poke other configurations once booted in that mode. A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).
    - bravetraveler 4y ago
      Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem. There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.
  - gtirloni 4y ago
rawoke083600 4y ago
That sounds downright horrible !
sofixa 4y ago
> If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. Isn't the common wisdom that you should have them enabled, but staggered across hours/days?
- gtirloni 4y ago
  Not a huge Debian/Ubuntu user but I think the systemd timer that triggers the unattended updates has a random delay added to it. I don't know of it's hours or just seconds.
  - markstos 4y ago
    I believe it's staggered across hours by default and it seems that Canonical might have been able to at least stop pushing out the bad update even before they had a fix
- AtlasBarfed 4y ago
  Probably better you have rolling A/B replacements that stop the replacement run if the replacement doesn't come up. This is mostly an in-place upgrade issue?

Spivak 4y ago

I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.

yjftsjthsd-h 4y ago
> don’t use unattended upgrades > Build your images in CI job I know container images should generally be immutable, but I would expect unattended upgrades to be mostly used on the host, not in a container, in which that management system doesn't really work (unless you're doing VMs where you can deploy immutable root images to the VMs as well, or some fun bare metal + PXE combination).
- ec109685 4y ago
  Idea is that your deploy step should handle both deploying code as well as upgrading OS, so all changes go through same pipeline.
- CuriousCosmic 4y ago
  alternatively I suppose depending on the size of your operation, you should consider having a dummy prod using at least one of each of the servers in your environment and using that to validate host upgrades. after that you can push an unattended upgrade via a self-hosted package+upgrade server. Let things be automatic to the maximum degree possible but give yourself a single hard human checkpoint and some minimum level of validation in a dummy environment first.
- Spivak 4y ago
  > or some fun bare metal + PXE combination This is actually what I implemented for our hypervisor tier, it’s not as scary as it sounds. I could legit completely rebuild our entire stack down to the metal in about 3 hours. Kick off a new hypervisor version, the inactive side PXE boots all the nodes, installs and configures a Proxmox cluster, slaves itself to our Ceph cluster, and then either does a hot migration of all the VMs or kicks off a full deploy which rebuilds all the infra (Consul, Rabbit, Redis, LDAP, Elastic, PowerDNS, etc) along with the app servers. The hardest part (which really isn’t) is maintaining the clusters across the blue/green sides. With this setup our only mutable infrastructure was our Ceph cluster (because replacing OSDs takes unacceptably long) and our DB (for performance the writers lived on dedicated servers, the read replicas lived on the VMs.).
nix23 4y ago
>I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs Some years ago everyone said the same about windows-servers ;)
markstos 4y ago
Sorry, not my experience. My experience has been that by the time I notice some serious vulnerability is in the news, my servers have already patched themselves. I have never "hated life" or had a "hard to find and undo bug" due to automatic security patching. I pretty quickly found what caused this and had a clear path to resolution. This is the first security update that caused a boot failure in about a decade. It was bad, but it didn't change my mind about unattended-upgrades. My takeaway that if that maybe I should have upgraded my 20.04 servers to 22.04 server sooner.
- Spivak 4y ago
  You’re conflating unattended-upgrades (server mutability, hard to roll back) with automated patching in general. Do automated patching but also run the changes though your CI so you can catch breaking changes and roll them out in a way that’s easy to debug (you can diff images) and revert. I bet when you update your software dependencies you run those changes through your tests but your OS is a giant pile of code that usually gets updated differently and independently because mostly historical reasons.
  - markstos 4y ago
    > I bet when you update your software dependencies you run those changes through your tests but your OS is a giant pile of code that usually gets updated differently and independently because mostly historical reasons. Close. We are moving towards defining our server states through Ansible, but the project is not close to completion. Perhaps once that's further along, we could use Ansible Molecule + CI to test a new server state when there's a new patch available, but that's not an option on the table today. The system we had in place for /today/ worked: Lower priority or redundant servers were set to auto-reboot after applying security updates, while other critical servers require manual reboot at low-risk times. By then, the patch has already been tested on lower-risk servers. As a result, this issue caused no user-visible downtime for us, and due to the staggered runs of unattended-upgrades affected a minimal number of servers. And this was the first time in 10+ years that something like this happened and we have to choose to write to prioritize spending our process-improvement time based on likelihood and impact.

QuentinM 4y ago

Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calculation overflows after the window went to zero due to a temporary slow consumer - freezing the window size to a few bytes only instead of getting a prompt full window recovery. Not to mention we’ve also had our fair share of production triple faults from bugs in the Intel firmware patches for Spectre, which took weeks to investigate & fix between ourselves struggling to keep our exchange up & running, Intel, and AWS. And that is why there’s value in the CoreOS/ContainerLinux-like solutions we designed & implemented nearly a decade ago now. Being able to promptly rollback any kernel/system/package upgrades at once - either manually or either after it’s detected a few panics in quick successions is actually quite awesome. Not to mention the slow update rollout strategy baked into the Omaha controller. But the reality is that the what-ifs are always the hardest to market, nearly always after-thoughts and with fast-spiking/fast-decaying traction after major events.

stingraycharles 4y ago
It really seems like there’s no good non-redhat (but still “production capable”) alternative to CoreOS nowadays, right? It’s pretty much Fedora / Redhat CoreOS or go directly to things such as k3os?
- georgyo 4y ago
  The rancher stack is pretty amazing. Elemental is pretty close to coreos: https://github.com/rancher/elemental/ https://github.com/rancher/elemental/ They even have a way to build arbitrary os images: https://github.com/rancher/elemental-toolkit https://github.com/rancher/elemental-toolkit It's pretty great
- Already__Taken 4y ago
  k3os is in a dieing limbo, now is the time to get some interest in using stuff like it

ConstantVigil 4y ago

And while I haven't had this happen to me yet; the fear of something like this or even worse is why I try to stay one step behind the update paths on linux distros. Security patches matter, but I'm no one important, so I should be fine to wait a week or month... Anyone else who is important though... servers for example...

_y5hn 4y ago

Wouldn't rollback of kernel be a choice in grub menu? It's pretty standard for all distros to have that choice.

withinboredom 4y ago
That usually requires physical access to the server to select it during boot.
- taspeotis 4y ago
  Or a real server with Lights Out Management.
  - withinboredom 4y ago
    That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.
    - phillu 4y ago
      That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server. Oh well, I guess experiences differ.
      - withinboredom 4y ago
        My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?

akx 4y ago

> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.

capableweb 4y ago
Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
- jacquesm 4y ago
  That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
- dspillett 4y ago
  If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu? Failing that you could try the “single” directive and poke other configurations once booted in that mode. A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).
  - bravetraveler 4y ago
    Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem. There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.
- gtirloni 4y ago

rawoke083600 4y ago

That sounds downright horrible !

sofixa 4y ago

> If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. Isn't the common wisdom that you should have them enabled, but staggered across hours/days?

gtirloni 4y ago
Not a huge Debian/Ubuntu user but I think the systemd timer that triggers the unattended updates has a random delay added to it. I don't know of it's hours or just seconds.
- markstos 4y ago
  I believe it's staggered across hours by default and it seems that Canonical might have been able to at least stop pushing out the bad update even before they had a fix
AtlasBarfed 4y ago
Probably better you have rolling A/B replacements that stop the replacement run if the replacement doesn't come up. This is mostly an in-place upgrade issue?