11 ms·

An Ubuntu kernel bug causes container crashes

232 points by kennethko 4y ago

mobilio 4y ago
This isn't only bug. I've recently migrated to Ubuntu 22.04 and got this: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453 on HP ProLiant servers.
- iam-TJ 4y ago
  I've just added some info to those bugs on a possible upstream stable fix.
markstos 4y ago
This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed at boot, since the containers share the kernel with the host. At that point, you can't SSH in and have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. And then of course if you revert the kernel upgrade, you were once again vulnerable to whatever problem the security update was fixing...
- Spivak 4y ago
  I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.
  - yjftsjthsd-h 4y ago
    > don’t use unattended upgrades > Build your images in CI job I know container images should generally be immutable, but I would expect unattended upgrades to be mostly used on the host, not in a container, in which that management system doesn't really work (unless you're doing VMs where you can deploy immutable root images to the VMs as well, or some fun bare metal + PXE combination).
    - ec109685 4y ago
      Idea is that your deploy step should handle both deploying code as well as upgrading OS, so all changes go through same pipeline.
    - CuriousCosmic 4y ago
      alternatively I suppose depending on the size of your operation, you should consider having a dummy prod using at least one of each of the servers in your environment and using that to validate host upgrades. after that you can push an unattended upgrade via a self-hosted package+upgrade server. Let things be automatic to the maximum degree possible but give yourself a single hard human checkpoint and some minimum level of validation in a dummy environment first.
    - Spivak 4y ago
      > or some fun bare metal + PXE combination This is actually what I implemented for our hypervisor tier, it’s not as scary as it sounds. I could legit completely rebuild our entire stack down to the metal in about 3 hours. Kick off a new hypervisor version, the inactive side PXE boots all the nodes, installs and configures a Proxmox cluster, slaves itself to our Ceph cluster, and then either does a hot migration of all the VMs or kicks off a full deploy which rebuilds all the infra (Consul, Rabbit, Redis, LDAP, Elastic, PowerDNS, etc) along with the app servers. The hardest part (which really isn’t) is maintaining the clusters across the blue/green sides. With this setup our only mutable infrastructure was our Ceph cluster (because replacing OSDs takes unacceptably long) and our DB (for performance the writers lived on dedicated servers, the read replicas lived on the VMs.).
  - 4y ago
- QuentinM 4y ago
  Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calculation overflows after the window went to zero due to a temporary slow consumer - freezing the window size to a few bytes only instead of getting a prompt full window recovery. Not to mention we’ve also had our fair share of production triple faults from bugs in the Intel firmware patches for Spectre, which took weeks to investigate & fix between ourselves struggling to keep our exchange up & running, Intel, and AWS. And that is why there’s value in the CoreOS/ContainerLinux-like solutions we designed & implemented nearly a decade ago now. Being able to promptly rollback any kernel/system/package upgrades at once - either manually or either after it’s detected a few panics in quick successions is actually quite awesome. Not to mention the slow update rollout strategy baked into the Omaha controller. But the reality is that the what-ifs are always the hardest to market, nearly always after-thoughts and with fast-spiking/fast-decaying traction after major events.
  - stingraycharles 4y ago
    It really seems like there’s no good non-redhat (but still “production capable”) alternative to CoreOS nowadays, right? It’s pretty much Fedora / Redhat CoreOS or go directly to things such as k3os?
    - georgyo 4y ago
      The rancher stack is pretty amazing. Elemental is pretty close to coreos: https://github.com/rancher/elemental/ https://github.com/rancher/elemental/ They even have a way to build arbitrary os images: https://github.com/rancher/elemental-toolkit https://github.com/rancher/elemental-toolkit It's pretty great
    - Already__Taken 4y ago
      k3os is in a dieing limbo, now is the time to get some interest in using stuff like it
- ConstantVigil 4y ago
  And while I haven't had this happen to me yet; the fear of something like this or even worse is why I try to stay one step behind the update paths on linux distros. Security patches matter, but I'm no one important, so I should be fine to wait a week or month... Anyone else who is important though... servers for example...
- _y5hn 4y ago
  Wouldn't rollback of kernel be a choice in grub menu? It's pretty standard for all distros to have that choice.
  - withinboredom 4y ago
    That usually requires physical access to the server to select it during boot.
    - taspeotis 4y ago
      Or a real server with Lights Out Management.
      - withinboredom 4y ago
        That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.
        phillu 4y ago
        That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server. Oh well, I guess experiences differ.
        withinboredom 4y ago
        My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?
- akx 4y ago
  > have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.
  - capableweb 4y ago
    Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
    - jacquesm 4y ago
      That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
    - dspillett 4y ago
      If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu? Failing that you could try the “single” directive and poke other configurations once booted in that mode. A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).
      - bravetraveler 4y ago
        Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem. There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.
    - gtirloni 4y ago
- rawoke083600 4y ago
  That sounds downright horrible !
- sofixa 4y ago
  > If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. Isn't the common wisdom that you should have them enabled, but staggered across hours/days?
  - gtirloni 4y ago
    Not a huge Debian/Ubuntu user but I think the systemd timer that triggers the unattended updates has a random delay added to it. I don't know of it's hours or just seconds.
    - markstos 4y ago
      I believe it's staggered across hours by default and it seems that Canonical might have been able to at least stop pushing out the bad update even before they had a fix
  - AtlasBarfed 4y ago
    Probably better you have rolling A/B replacements that stop the replacement run if the replacement doesn't come up. This is mostly an in-place upgrade issue?
deleted 4y ago
[deleted]
mroche 4y ago
Copy-pasta of Jonathan Corbet: It's nice to see LWN on HN ... but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe. https://news.ycombinator.com/item?id=31852477 https://news.ycombinator.com/item?id=31852477 If you're interested in detailed commentary on and investigations of the FOSS space, I can't recommend a subscription to LWN enough!
- stefantalpalaru 4y ago
up6w6 4y ago
I'm using Oracle's ARM servers and I thought it was some weird patch they did to the kernel, the bug only disappeared when I force upgraded it to 22.04. Ubuntu/Canonical itself would be the last place I would have thought to be the source of a problem like that.
MBCook 4y ago
This seems like the kind of thing that automated testing should have been able to catch. It’s not like running Docker is a small use-case these days.
- jillesvangurp 4y ago
  My thoughts exactly, the details of what this bug is about technically are interesting and fascinating but the key take away is that something went terribly wrong with Ubuntu's testing processes. This should not have shipped without more scrutiny. Somebody presumably cut some corners there and it's worrying that that is possible at all. I actually rolled out Ubuntu 22.04 to a few servers a few weeks ago. Pretty uneventful update, all my Ansible scripts for 20.04 worked without modification against these new servers. So, I guess I dodge this bug for now. One reason I've always preferred Ubuntu over Red Hat for servers is that with Red Hat/Centos essentially everything I care about is perpetually and hopelessly out of date and obsolete. So, it just creates a lot of hassle to work around that and get reasonably current versions of things I actually need my servers to run. With Ubuntu that was always a lot more straightforward. I currently write this on a laptop with Manjaro and Linux 5.18. I'm glad I don't have to deal with about a year of long fixed issues with hardware, bluetooth, GPUs, performance, etc. IMHO there's very little value in sticking with older kernels on desktop machines. Especially when that involves a convoluted process of back-porting and integrating lots of complicated patches. I recently put Ubuntu on an old imac (secure boot prevents booting Manjaro) and I promptly ran into hardware issues that I recall having with Manjaro a few months ago that were fixed by simply upgrading the kernel. Bluetooth especially seems way more flaky. And that's not exactly flawless on 5.18 either. I get the if it ain't broke don't fix it thing; my point is that with modern Desktop Linux things being broken is a constant. The least broken version of Linux is usually the kernel that was just released that has all the cumulative fixes for all the issues addressed in previous kernel releases. Opting out of a few years of those fixes seems misguided. Even on servers, I suspect simply updating the kernel more regularly would not be the end of the world for most users. With an incubation period to catch bugs/blocking issues of course, the more people use a kernel version, the more stable it gets. I doubt many users would experience any regressions. And it's a lot cheaper to support. If I had the option, I don't think I would opt to run 2-3 year old kernels on any of my servers if I had a different choice. I don't see the value of opting out of 2-3 years worth of known & fixed stability, performance, and other issues.
symlinkk 4y ago
Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?
- fomine3 4y ago
  According to the article, looks reasonable modification (though it's hard) but should be tested.
- baggy_trough 4y ago
  Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.
  - kramerger 4y ago
    Seems like every department at canonical needs to learn this on their own. After all, they reinvented everything from DE to init system at least once in past. (They also have their own containers, LXD. I actually really like that one, please keep working on that canonical)
    - leaflets2 4y ago
      What are some other bad mistakes that's been made?
  - heurisko 4y ago
    22.04 uses the 5.15 LTS kernel. https://www.kernel.org/category/releases.html https://www.kernel.org/category/releases.html But I assume as Ubuntu follows an April release schedule, it doesn't always match with an appropriate LTS kernel.
    - lproven 4y ago
      Ubuntu was originally designed as a desktop OS and its release cycle was synched with the GNOME 2.x release cycle.
zerop 4y ago
Any good documentation that talks about how big open source software manage code changes, releases cadences, given contributors from across the world.
- wronglyprepaid 4y ago
  I'm fairly sure this differs for different project/organization, not sure there is a rule, and not sure there are really any considerations that are specific to open source, good practices are good practices regardless. That being said, I rate Canonical's practices as rather poor.
  - zerop 4y ago
    Taking an example, how are Linux Kernel releases are planned and managed
atoav 4y ago
The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.
- lixtra 4y ago
  This is not a docker bug, it is a kernel bug. It could be triggered by other complex applications that use kernel container features.
  - jacquesm 4y ago
    That pretty much makes the GPs point: emergent behavior arising from complexity.
  - atoav 4y ago
    Or it could not be triggered if you don't use containers.
    - Fnoord 4y ago
      Would it have occurred on Podman as well?
      - markstos 4y ago
        Some of my servers were running Ubuntu with systemd+podman to manage services, and none of them had a problem.
        remram 4y ago
        You would have to be running this specific kernel version and one of your containers has to memory-map a file (not from a volume?), from what I understand.
      - remram 4y ago
        Depending on your configuration. You would need to be using overlayfs with Podman.
oynqr 4y ago
They are just trying to convert docker users to snap enthusiasts.
- kramerger 4y ago
  Well, lxd is a snap now so you are maybe into something
oskenso 4y ago
This affected my team and I directly. I had to recover a server from backup, we then postponed kernel updates when I noticed something was wrong after reading this https://www.mail-archive.com/kernel-packages@lists.launchpad.net/msg482001.html https://www.mail-archive.com/kernel-packages@lists.launchpad...
mrintegrity 4y ago
This was exceptionally annoying for me, some ec2 instances are used only during the day and we stop/start them with an in house scheduling application outside office hours. Also automatic security upgrades are enabled. Came in to work one day last week and all of our UAT environment was down. It is possible to ssh in for about 2 seconds before the kernel panic so I solved it by doing this: while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done On the next reboot i was able to ssh in and change to the (then just released within the past hour) kernel that doesn't have this stupid bug. After another reboot you can move containderd back and it should be working again affected: linux-image-5.13.0-1028 not affected: >linux-image-5.13.0-1029
- kubanczyk 4y ago
  I love the approach. Nowadays you can even sleep 0.1 on most systems. (And on Busybox, you should have usleep.)
latte2021 4y ago
Does this apply for desktop also?
- lproven 4y ago
  Probably not, unless you're working with containers. Most desktop users are not, I suspect... unless they are developers building containers for later deployment on servers.
jacquesm 4y ago
Auto update strikes again. Really, we need to re-think this.
- akvadrako 4y ago
  There is no need to rethink it; it's never been a good idea to leave it turned on.
  - jacquesm 4y ago
    Well, you're between a rock and a hard place. No auto-update = security risk exposure, auto-update = stability risk exposure (and sometimes security risk exposure thrown in for free as well).
    - amelius 4y ago
      If the only externally visible service you run is sshd then how important is it to auto-update for security reasons? (Also considering that security risks in sshd are almost guaranteed to end up on the front page of HN, so you won't miss it).
      - leaflets2 4y ago
        > the front page of HN What if you're in bed with the flu But if you're a team, then maybe. Still, could delay the response with a whole day (checking HN once a day)
dspillett 4y ago
Perhaps add “June 2022” to the title to reduce panic? Updated packages that resolve the issue were released on 2022-06-10, so this article is a post-mortem not an alert of a new problem that could affect people now.
heurisko 4y ago
I like Ubuntu, but in the last few months I have been following how things are packaged more closely. For example, looking at the package for postgresql-14, an update still hasn't been released for the unscheduled mid-June release version 14.4, which fixed possible index corruption. http://changelogs.ubuntu.com/changelogs/pool/main/p/postgresql-14/postgresql-14_14.3-0ubuntu0.22.04.1/changelog http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres... I would have thought this would have been packaged earlier, as I would expect the Ubuntu + postgresql would be a common combination. It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.
- cpach 4y ago
  “It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.” I don’t really have any sources to back this up, but my impression is that Canonical is kinda trying to punch above their weight.
- rascul 4y ago
  It's in jammy-updates https://packages.ubuntu.com/jammy-updates/postgresql-14 https://packages.ubuntu.com/jammy-updates/postgresql-14 http://changelogs.ubuntu.com/changelogs/pool/main/p/postgresql-14/postgresql-14_14.4-0ubuntu0.22.04.1/changelog http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres...
nyc_pizzadev 4y ago
Interesting, around the same time both of my Ubuntu 20 laptops (Dell and Lenovo) started having major problems connecting to my home wifi. It only affects these 2 laptops, all my other devices have no problems. Before reading this I did think this was a result of a bad Ubuntu update. Given they switched kernel versions, my guess is that this is infact the culprit. Very annoying, it takes me anywhere from 5 to 45 minutes to establish a wifi session now.
compsciphd 4y ago
ubuntu 22.04 also broke many IBM laptops. Took them 2 months to fix it, without any acknowledgement that the bug existed. number of tickets in launchpad such as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957 It makes me Q the value in my org looking into an ubuntu advantage subscription. When there are tickets that have lots of "me too" that result in unusable laptops, one should at least triage them / consolidate them into a single ticket and then be able to mark when fixed.
lwswl 4y ago
The more container bugs the better. I hope they can't fix it.

232 points by kennethko 4y ago

This isn't only bug. I've recently migrated to Ubuntu 22.04 and got this: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971505 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970453 on HP ProLiant servers.

iam-TJ 4y ago
I've just added some info to those bugs on a possible upstream stable fix.

markstos 4y ago

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed at boot, since the containers share the kernel with the host. At that point, you can't SSH in and have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. And then of course if you revert the kernel upgrade, you were once again vulnerable to whatever problem the security update was fixing...

Spivak 4y ago
I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.
- yjftsjthsd-h 4y ago
  > don’t use unattended upgrades > Build your images in CI job I know container images should generally be immutable, but I would expect unattended upgrades to be mostly used on the host, not in a container, in which that management system doesn't really work (unless you're doing VMs where you can deploy immutable root images to the VMs as well, or some fun bare metal + PXE combination).
  - ec109685 4y ago
    Idea is that your deploy step should handle both deploying code as well as upgrading OS, so all changes go through same pipeline.
  - CuriousCosmic 4y ago
    alternatively I suppose depending on the size of your operation, you should consider having a dummy prod using at least one of each of the servers in your environment and using that to validate host upgrades. after that you can push an unattended upgrade via a self-hosted package+upgrade server. Let things be automatic to the maximum degree possible but give yourself a single hard human checkpoint and some minimum level of validation in a dummy environment first.
  - Spivak 4y ago
    > or some fun bare metal + PXE combination This is actually what I implemented for our hypervisor tier, it’s not as scary as it sounds. I could legit completely rebuild our entire stack down to the metal in about 3 hours. Kick off a new hypervisor version, the inactive side PXE boots all the nodes, installs and configures a Proxmox cluster, slaves itself to our Ceph cluster, and then either does a hot migration of all the VMs or kicks off a full deploy which rebuilds all the infra (Consul, Rabbit, Redis, LDAP, Elastic, PowerDNS, etc) along with the app servers. The hardest part (which really isn’t) is maintaining the clusters across the blue/green sides. With this setup our only mutable infrastructure was our Ceph cluster (because replacing OSDs takes unacceptably long) and our DB (for performance the writers lived on dedicated servers, the read replicas lived on the VMs.).
- 4y ago
QuentinM 4y ago
Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calculation overflows after the window went to zero due to a temporary slow consumer - freezing the window size to a few bytes only instead of getting a prompt full window recovery. Not to mention we’ve also had our fair share of production triple faults from bugs in the Intel firmware patches for Spectre, which took weeks to investigate & fix between ourselves struggling to keep our exchange up & running, Intel, and AWS. And that is why there’s value in the CoreOS/ContainerLinux-like solutions we designed & implemented nearly a decade ago now. Being able to promptly rollback any kernel/system/package upgrades at once - either manually or either after it’s detected a few panics in quick successions is actually quite awesome. Not to mention the slow update rollout strategy baked into the Omaha controller. But the reality is that the what-ifs are always the hardest to market, nearly always after-thoughts and with fast-spiking/fast-decaying traction after major events.
- stingraycharles 4y ago
  It really seems like there’s no good non-redhat (but still “production capable”) alternative to CoreOS nowadays, right? It’s pretty much Fedora / Redhat CoreOS or go directly to things such as k3os?
  - georgyo 4y ago
    The rancher stack is pretty amazing. Elemental is pretty close to coreos: https://github.com/rancher/elemental/ https://github.com/rancher/elemental/ They even have a way to build arbitrary os images: https://github.com/rancher/elemental-toolkit https://github.com/rancher/elemental-toolkit It's pretty great
  - Already__Taken 4y ago
    k3os is in a dieing limbo, now is the time to get some interest in using stuff like it
ConstantVigil 4y ago
And while I haven't had this happen to me yet; the fear of something like this or even worse is why I try to stay one step behind the update paths on linux distros. Security patches matter, but I'm no one important, so I should be fine to wait a week or month... Anyone else who is important though... servers for example...
_y5hn 4y ago
Wouldn't rollback of kernel be a choice in grub menu? It's pretty standard for all distros to have that choice.
- withinboredom 4y ago
  That usually requires physical access to the server to select it during boot.
  - taspeotis 4y ago
    Or a real server with Lights Out Management.
    - withinboredom 4y ago
      That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.
      - phillu 4y ago
        That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server. Oh well, I guess experiences differ.
        withinboredom 4y ago
        My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?
akx 4y ago
> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.
- capableweb 4y ago
  Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
  - jacquesm 4y ago
    That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
  - dspillett 4y ago
    If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu? Failing that you could try the “single” directive and poke other configurations once booted in that mode. A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).
    - bravetraveler 4y ago
      Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem. There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.
  - gtirloni 4y ago
rawoke083600 4y ago
That sounds downright horrible !
sofixa 4y ago
> If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. Isn't the common wisdom that you should have them enabled, but staggered across hours/days?
- gtirloni 4y ago
  Not a huge Debian/Ubuntu user but I think the systemd timer that triggers the unattended updates has a random delay added to it. I don't know of it's hours or just seconds.
  - markstos 4y ago
    I believe it's staggered across hours by default and it seems that Canonical might have been able to at least stop pushing out the bad update even before they had a fix
- AtlasBarfed 4y ago
  Probably better you have rolling A/B replacements that stop the replacement run if the replacement doesn't come up. This is mostly an in-place upgrade issue?

deleted 4y ago

[deleted]

mroche 4y ago

Copy-pasta of Jonathan Corbet: It's nice to see LWN on HN ... but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe. https://news.ycombinator.com/item?id=31852477 https://news.ycombinator.com/item?id=31852477 If you're interested in detailed commentary on and investigations of the FOSS space, I can't recommend a subscription to LWN enough!

stefantalpalaru 4y ago

up6w6 4y ago

I'm using Oracle's ARM servers and I thought it was some weird patch they did to the kernel, the bug only disappeared when I force upgraded it to 22.04. Ubuntu/Canonical itself would be the last place I would have thought to be the source of a problem like that.

MBCook 4y ago

This seems like the kind of thing that automated testing should have been able to catch. It’s not like running Docker is a small use-case these days.

jillesvangurp 4y ago
My thoughts exactly, the details of what this bug is about technically are interesting and fascinating but the key take away is that something went terribly wrong with Ubuntu's testing processes. This should not have shipped without more scrutiny. Somebody presumably cut some corners there and it's worrying that that is possible at all. I actually rolled out Ubuntu 22.04 to a few servers a few weeks ago. Pretty uneventful update, all my Ansible scripts for 20.04 worked without modification against these new servers. So, I guess I dodge this bug for now. One reason I've always preferred Ubuntu over Red Hat for servers is that with Red Hat/Centos essentially everything I care about is perpetually and hopelessly out of date and obsolete. So, it just creates a lot of hassle to work around that and get reasonably current versions of things I actually need my servers to run. With Ubuntu that was always a lot more straightforward. I currently write this on a laptop with Manjaro and Linux 5.18. I'm glad I don't have to deal with about a year of long fixed issues with hardware, bluetooth, GPUs, performance, etc. IMHO there's very little value in sticking with older kernels on desktop machines. Especially when that involves a convoluted process of back-porting and integrating lots of complicated patches. I recently put Ubuntu on an old imac (secure boot prevents booting Manjaro) and I promptly ran into hardware issues that I recall having with Manjaro a few months ago that were fixed by simply upgrading the kernel. Bluetooth especially seems way more flaky. And that's not exactly flawless on 5.18 either. I get the if it ain't broke don't fix it thing; my point is that with modern Desktop Linux things being broken is a constant. The least broken version of Linux is usually the kernel that was just released that has all the cumulative fixes for all the issues addressed in previous kernel releases. Opting out of a few years of those fixes seems misguided. Even on servers, I suspect simply updating the kernel more regularly would not be the end of the world for most users. With an incubation period to catch bugs/blocking issues of course, the more people use a kernel version, the more stable it gets. I doubt many users would experience any regressions. And it's a lot cheaper to support. If I had the option, I don't think I would opt to run 2-3 year old kernels on any of my servers if I had a different choice. I don't see the value of opting out of 2-3 years worth of known & fixed stability, performance, and other issues.

symlinkk 4y ago

Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?

fomine3 4y ago
According to the article, looks reasonable modification (though it's hard) but should be tested.
baggy_trough 4y ago
Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.
- kramerger 4y ago
  Seems like every department at canonical needs to learn this on their own. After all, they reinvented everything from DE to init system at least once in past. (They also have their own containers, LXD. I actually really like that one, please keep working on that canonical)
  - leaflets2 4y ago
    What are some other bad mistakes that's been made?
- heurisko 4y ago
  22.04 uses the 5.15 LTS kernel. https://www.kernel.org/category/releases.html https://www.kernel.org/category/releases.html But I assume as Ubuntu follows an April release schedule, it doesn't always match with an appropriate LTS kernel.
  - lproven 4y ago
    Ubuntu was originally designed as a desktop OS and its release cycle was synched with the GNOME 2.x release cycle.

zerop 4y ago

Any good documentation that talks about how big open source software manage code changes, releases cadences, given contributors from across the world.

wronglyprepaid 4y ago
I'm fairly sure this differs for different project/organization, not sure there is a rule, and not sure there are really any considerations that are specific to open source, good practices are good practices regardless. That being said, I rate Canonical's practices as rather poor.
- zerop 4y ago
  Taking an example, how are Linux Kernel releases are planned and managed

atoav 4y ago

The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

lixtra 4y ago
This is not a docker bug, it is a kernel bug. It could be triggered by other complex applications that use kernel container features.
- jacquesm 4y ago
  That pretty much makes the GPs point: emergent behavior arising from complexity.
- atoav 4y ago
  Or it could not be triggered if you don't use containers.
  - Fnoord 4y ago
    Would it have occurred on Podman as well?
    - markstos 4y ago
      Some of my servers were running Ubuntu with systemd+podman to manage services, and none of them had a problem.
      - remram 4y ago
        You would have to be running this specific kernel version and one of your containers has to memory-map a file (not from a volume?), from what I understand.
    - remram 4y ago
      Depending on your configuration. You would need to be using overlayfs with Podman.

oynqr 4y ago

They are just trying to convert docker users to snap enthusiasts.

kramerger 4y ago
Well, lxd is a snap now so you are maybe into something

oskenso 4y ago

This affected my team and I directly. I had to recover a server from backup, we then postponed kernel updates when I noticed something was wrong after reading this https://www.mail-archive.com/kernel-packages@lists.launchpad.net/msg482001.html https://www.mail-archive.com/kernel-packages@lists.launchpad...

mrintegrity 4y ago

This was exceptionally annoying for me, some ec2 instances are used only during the day and we stop/start them with an in house scheduling application outside office hours. Also automatic security upgrades are enabled. Came in to work one day last week and all of our UAT environment was down. It is possible to ssh in for about 2 seconds before the kernel panic so I solved it by doing this: while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done On the next reboot i was able to ssh in and change to the (then just released within the past hour) kernel that doesn't have this stupid bug. After another reboot you can move containderd back and it should be working again affected: linux-image-5.13.0-1028 not affected: >linux-image-5.13.0-1029

kubanczyk 4y ago
I love the approach. Nowadays you can even sleep 0.1 on most systems. (And on Busybox, you should have usleep.)

latte2021 4y ago

Does this apply for desktop also?

lproven 4y ago
Probably not, unless you're working with containers. Most desktop users are not, I suspect... unless they are developers building containers for later deployment on servers.

jacquesm 4y ago

Auto update strikes again. Really, we need to re-think this.

akvadrako 4y ago
There is no need to rethink it; it's never been a good idea to leave it turned on.
- jacquesm 4y ago
  Well, you're between a rock and a hard place. No auto-update = security risk exposure, auto-update = stability risk exposure (and sometimes security risk exposure thrown in for free as well).
  - amelius 4y ago
    If the only externally visible service you run is sshd then how important is it to auto-update for security reasons? (Also considering that security risks in sshd are almost guaranteed to end up on the front page of HN, so you won't miss it).
    - leaflets2 4y ago
      > the front page of HN What if you're in bed with the flu But if you're a team, then maybe. Still, could delay the response with a whole day (checking HN once a day)

dspillett 4y ago

Perhaps add “June 2022” to the title to reduce panic? Updated packages that resolve the issue were released on 2022-06-10, so this article is a post-mortem not an alert of a new problem that could affect people now.

heurisko 4y ago

I like Ubuntu, but in the last few months I have been following how things are packaged more closely. For example, looking at the package for postgresql-14, an update still hasn't been released for the unscheduled mid-June release version 14.4, which fixed possible index corruption. http://changelogs.ubuntu.com/changelogs/pool/main/p/postgresql-14/postgresql-14_14.3-0ubuntu0.22.04.1/changelog http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres... I would have thought this would have been packaged earlier, as I would expect the Ubuntu + postgresql would be a common combination. It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.

cpach 4y ago
“It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.” I don’t really have any sources to back this up, but my impression is that Canonical is kinda trying to punch above their weight.
rascul 4y ago
It's in jammy-updates https://packages.ubuntu.com/jammy-updates/postgresql-14 https://packages.ubuntu.com/jammy-updates/postgresql-14 http://changelogs.ubuntu.com/changelogs/pool/main/p/postgresql-14/postgresql-14_14.4-0ubuntu0.22.04.1/changelog http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres...

nyc_pizzadev 4y ago

Interesting, around the same time both of my Ubuntu 20 laptops (Dell and Lenovo) started having major problems connecting to my home wifi. It only affects these 2 laptops, all my other devices have no problems. Before reading this I did think this was a result of a bad Ubuntu update. Given they switched kernel versions, my guess is that this is infact the culprit. Very annoying, it takes me anywhere from 5 to 45 minutes to establish a wifi session now.

compsciphd 4y ago

ubuntu 22.04 also broke many IBM laptops. Took them 2 months to fix it, without any acknowledgement that the bug existed. number of tickets in launchpad such as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957 It makes me Q the value in my org looking into an ubuntu advantage subscription. When there are tickets that have lots of "me too" that result in unusable laptops, one should at least triage them / consolidate them into a single ticket and then be able to mark when fixed.

lwswl 4y ago

The more container bugs the better. I hope they can't fix it.