4 ms·

Expanding on gentoo's recommendations: I wonder if we should just universally accept that live patching should become part of the linux kernel? An automatic jo

Expanding on gentoo's recommendations: I wonder if we should just universally accept that live patching should become part of the linux kernel? An automatic job that updates (much like some system packages in some distros) that installs (signed) live patches from upstream? Of course we would run into a problem where a malicious patch can now be distributed reliably to hundreds of thousands of machines, but we already have that at a lower level with normal application updates. Canonical has thus far proved that it can be safe, but they're also a massive organization that is locking this feature for $200/yr for any commercial use. It would be neat if such patches could retroactively replace tagged functions that have identical sematics so that means it would automatically get backported without extra effort from the maintainers.

nine_k 5mo ago
Why would the source of the patches be less trusted than the source of (updated) kernels? I expect it to be the same, your distro. $200/year is peanuts for any commercial use worth the name. The problem, of course, is the whole non-free infrastructure it has to introduce. I wonder when large and critical OSS projects will start to be seen as a public good they are, with large corporations willingly financing them because not doing so is bad PR.
- graemep 5mo ago
  Public goods are not generally funded by large corporations.
  - siwatanejo 5mo ago
    Errr.. are you saying that Linux contributors are only affiliated to small companies or volunteers? Well, last time I checked, some contributors were RedHat(IBM) and Microsoft employees ;)
    - graemep 5mo ago
      I am saying that big company funding of some FOSS is exceptional compared to the economy at large. Most public goods are not funded by bug companies. They do not do the same for roads or public health services, or free education or public libraries etc. Even with FOSS few projects get the funding or contributions the Linux kernel does.
c7b 5mo ago
After the npm supply chain attacks people suggested automating delays before installing updates, now we're talking about automating update delivery... I'm afraid there won't be any easy or quick fix after decades of treating security as an afterthought.
- ordu 5mo ago
  Linux distros are not npm. It doesn't mean they are infallible to malicious actors, but I believe it is possible to make them infallible for some small set of packages at least. Attacks are still possible, but if we look at xz backdoor attack[1] it was insanely complicated attack and it still failed. Its fail doesn't look promising, attack could succeed just the attacker was unlucky. Still it shows that the success is not guaranteed. Theoretically npm can be improved in this way, if there were a separate "distro" for packaged, with dedicated maintainers for packages, who don't write code, just pull it from a mainstream and review it. It is not being done because of tragedy of commons, not because it is impossible. [1] https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor
  - ahartmetz 5mo ago
    Whenever you read about an incredibly unlucky criminal, there's a chance that the unlucky event is a parallel construction to the classified real reason why they were caught. Not sure how exactly that would have worked in this case.
    - ordu 5mo ago
      Yes, it could be. But it is a hypothetical that smells like a conspiracy theory. I wonder why you think it is a good idea to go for these hypotheticals? Are you arguing that the system may be more resilient than it seems? Like, maybe there is a conspiracy working on security. And they keep themselves secret so attackers would be susceptible to under-appreciate the real level of security and make mistakes that inevitable would caught? It seems like a over-stretched explanation, doesn't it. Care to explain yourself?
TacticalCoder 5mo ago
> I wonder if we should just universally accept that live patching should become part of the linux kernel? I think we can learn many lessons from the recent SNAFUs before going all wild on auto-patching. One lesson for example is that you shouldn't compile into the kernel modules that only about 0.00001% of all Linux installations out there are ever going to use. Another lesson is that even if the modules are compiled, but not into the kernel, they should probably be blacklisted (preventing them from loading) by default and only removed from the blacklist by people who really know they'll need these rarely used modules. We're way past the "but it needs to work on all cases": we're now into the "users installing our distro are getting hacked left and right" territory. In any case I think many things can be done before Linux distros reproduce the "security" practices of the NPM ecosystem.
- yjftsjthsd-h 5mo ago
  > we're now into the "users installing our distro are getting hacked left and right" territory Are we? Are users actually getting hacked, or have they theoretically been exposed to problems that could allow local privileged escalation if exploited but that nobody's seen used in the wild? (Edit: To be clear, I'm skeptical but this isn't a completely rhetorical question. If there are actual reports of these vulns causing problems, that would strongly incentivize a stronger response.)
- bombcar 5mo ago
  It used to be relatively standard even on the "big" distros to compile your own kernel if you needed something outside of the bog-standard. Modularization and all the related auto-detect auto-mod tools have resulted in most distros shipping a "works for almost everyone" kernel that has everything available as a module. Perhaps we should tend toward the first.
  - yjftsjthsd-h 5mo ago
    It seems like a reasonable middle ground for most distros is to put things in kernel modules, but then package those modules into separate packages. If you don't need somedriver.ko, then you don't `apt install linux-driver-somedriver`; if you do need it, just install the package and it just works without needing to compile anything and you get automatic updates and everything. For Gentoo, of course, "just recompile the kernel as desired" is more reasonable, though they have binary packages including for the kernel and I don't see why the same idea shouldn't work there.
throwa356262 5mo ago
I dont belive in live patching unless you are AWS. But I absolutely belive we should have a method for changing kernel configuration (e.g. kernel module blacklists) and syscall firewalls and alike.
- edelbitter 5mo ago
  Easier: Do not start with a "allow all" configuration in the first place. Maybe all of those userspace-work-done-in-kernel-because-muh-performance features should be restricted to (the "real") CAP_NET_ADMIN, unless positively enumerated as free-for-all-containers. And then subtract from that free-for-all list every time you learn that some kernel module in its currently available version cannot be trusted to do its own memory shuffling.
elevation 5mo ago
> live patching should become part of the linux kernel Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.
- PunchyHamster 5mo ago
  We've used ksplice for good amount of years (till bought by Oracle and they stopped publishing patches for other Linux distros) and all in all it was very stable technology 10 years ago > I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong. Because you haven't worked at that level in organization. Doing restart in some case might involve paperwork with your client and maintenance window outside of working hours even if service is redundant. And some customers are fine with a little bit of downtime and don't want active/active level of redundancy but still insist of maintenance windows for any work like that. Live patching makes that a whole lot easier
- toast0 5mo ago
  > Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong. I've run systems with live code updates for userland, and would have considered live kernel updates if it was reasonable on our systems. The thing is you typically build your system to tolerate reboot or unscheduled stop of a single node. Scheduled stop is nicer, but systems sometimes lock up even when you're not doing risky behaviors, so you know. But just because the system can tolerate a reboot or restart doesn't mean it's not disruptive. A lock up / etc during hot load is also disruptive, of course. But when you can push code without having to stop anything, with limited impact on users, it makes it easier and faster to do updates. You can use whatever rollout pattern you like to contain risk too; same as you would for an upgrade with restarts. For us, we have servers with hundreds of thousands or millions of tcp connections from mobile clients. Restarting a server would make all those clients have to reconnect and connecting is expensive. Restarting all the servers would result in many clients reconnecting several times. It was better to avoid it when possible.
kafrofrite 5mo ago
I guess there's merit for that, especially if you are in a cloud environment. In a previous company, I decided to set up dnf/kpatch for VMs that we considered critical. At the time I had a healthy disregard for reliability, mostly because we had enough trust in our terraform process so I decided to automate the whole thing through AWS System Manager across the fleet and guard the feature to work only for security patches targeting the kernel. Briefly, every VM that came up would have the necessary packages installed and from there live patching would execute periodically (I believe once a week) or manually. At some point after a quarter or something we had to devise a way to tag VMs to be excluded from this but this was relatively easy to do and most of the exclusions were testing infra anyway.

nine_k 5mo ago

Why would the source of the patches be less trusted than the source of (updated) kernels? I expect it to be the same, your distro. $200/year is peanuts for any commercial use worth the name. The problem, of course, is the whole non-free infrastructure it has to introduce. I wonder when large and critical OSS projects will start to be seen as a public good they are, with large corporations willingly financing them because not doing so is bad PR.

graemep 5mo ago
Public goods are not generally funded by large corporations.
- siwatanejo 5mo ago
  Errr.. are you saying that Linux contributors are only affiliated to small companies or volunteers? Well, last time I checked, some contributors were RedHat(IBM) and Microsoft employees ;)
  - graemep 5mo ago
    I am saying that big company funding of some FOSS is exceptional compared to the economy at large. Most public goods are not funded by bug companies. They do not do the same for roads or public health services, or free education or public libraries etc. Even with FOSS few projects get the funding or contributions the Linux kernel does.

c7b 5mo ago

After the npm supply chain attacks people suggested automating delays before installing updates, now we're talking about automating update delivery... I'm afraid there won't be any easy or quick fix after decades of treating security as an afterthought.

ordu 5mo ago
Linux distros are not npm. It doesn't mean they are infallible to malicious actors, but I believe it is possible to make them infallible for some small set of packages at least. Attacks are still possible, but if we look at xz backdoor attack[1] it was insanely complicated attack and it still failed. Its fail doesn't look promising, attack could succeed just the attacker was unlucky. Still it shows that the success is not guaranteed. Theoretically npm can be improved in this way, if there were a separate "distro" for packaged, with dedicated maintainers for packages, who don't write code, just pull it from a mainstream and review it. It is not being done because of tragedy of commons, not because it is impossible. [1] https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor
- ahartmetz 5mo ago
  Whenever you read about an incredibly unlucky criminal, there's a chance that the unlucky event is a parallel construction to the classified real reason why they were caught. Not sure how exactly that would have worked in this case.
  - ordu 5mo ago
    Yes, it could be. But it is a hypothetical that smells like a conspiracy theory. I wonder why you think it is a good idea to go for these hypotheticals? Are you arguing that the system may be more resilient than it seems? Like, maybe there is a conspiracy working on security. And they keep themselves secret so attackers would be susceptible to under-appreciate the real level of security and make mistakes that inevitable would caught? It seems like a over-stretched explanation, doesn't it. Care to explain yourself?

TacticalCoder 5mo ago

> I wonder if we should just universally accept that live patching should become part of the linux kernel? I think we can learn many lessons from the recent SNAFUs before going all wild on auto-patching. One lesson for example is that you shouldn't compile into the kernel modules that only about 0.00001% of all Linux installations out there are ever going to use. Another lesson is that even if the modules are compiled, but not into the kernel, they should probably be blacklisted (preventing them from loading) by default and only removed from the blacklist by people who really know they'll need these rarely used modules. We're way past the "but it needs to work on all cases": we're now into the "users installing our distro are getting hacked left and right" territory. In any case I think many things can be done before Linux distros reproduce the "security" practices of the NPM ecosystem.

yjftsjthsd-h 5mo ago
> we're now into the "users installing our distro are getting hacked left and right" territory Are we? Are users actually getting hacked, or have they theoretically been exposed to problems that could allow local privileged escalation if exploited but that nobody's seen used in the wild? (Edit: To be clear, I'm skeptical but this isn't a completely rhetorical question. If there are actual reports of these vulns causing problems, that would strongly incentivize a stronger response.)
bombcar 5mo ago
It used to be relatively standard even on the "big" distros to compile your own kernel if you needed something outside of the bog-standard. Modularization and all the related auto-detect auto-mod tools have resulted in most distros shipping a "works for almost everyone" kernel that has everything available as a module. Perhaps we should tend toward the first.
- yjftsjthsd-h 5mo ago
  It seems like a reasonable middle ground for most distros is to put things in kernel modules, but then package those modules into separate packages. If you don't need somedriver.ko, then you don't `apt install linux-driver-somedriver`; if you do need it, just install the package and it just works without needing to compile anything and you get automatic updates and everything. For Gentoo, of course, "just recompile the kernel as desired" is more reasonable, though they have binary packages including for the kernel and I don't see why the same idea shouldn't work there.

throwa356262 5mo ago

I dont belive in live patching unless you are AWS. But I absolutely belive we should have a method for changing kernel configuration (e.g. kernel module blacklists) and syscall firewalls and alike.

edelbitter 5mo ago
Easier: Do not start with a "allow all" configuration in the first place. Maybe all of those userspace-work-done-in-kernel-because-muh-performance features should be restricted to (the "real") CAP_NET_ADMIN, unless positively enumerated as free-for-all-containers. And then subtract from that free-for-all list every time you learn that some kernel module in its currently available version cannot be trusted to do its own memory shuffling.

elevation 5mo ago

> live patching should become part of the linux kernel Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.

PunchyHamster 5mo ago
We've used ksplice for good amount of years (till bought by Oracle and they stopped publishing patches for other Linux distros) and all in all it was very stable technology 10 years ago > I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong. Because you haven't worked at that level in organization. Doing restart in some case might involve paperwork with your client and maintenance window outside of working hours even if service is redundant. And some customers are fine with a little bit of downtime and don't want active/active level of redundancy but still insist of maintenance windows for any work like that. Live patching makes that a whole lot easier
toast0 5mo ago
> Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong. I've run systems with live code updates for userland, and would have considered live kernel updates if it was reasonable on our systems. The thing is you typically build your system to tolerate reboot or unscheduled stop of a single node. Scheduled stop is nicer, but systems sometimes lock up even when you're not doing risky behaviors, so you know. But just because the system can tolerate a reboot or restart doesn't mean it's not disruptive. A lock up / etc during hot load is also disruptive, of course. But when you can push code without having to stop anything, with limited impact on users, it makes it easier and faster to do updates. You can use whatever rollout pattern you like to contain risk too; same as you would for an upgrade with restarts. For us, we have servers with hundreds of thousands or millions of tcp connections from mobile clients. Restarting a server would make all those clients have to reconnect and connecting is expensive. Restarting all the servers would result in many clients reconnecting several times. It was better to avoid it when possible.

kafrofrite 5mo ago

I guess there's merit for that, especially if you are in a cloud environment. In a previous company, I decided to set up dnf/kpatch for VMs that we considered critical. At the time I had a healthy disregard for reliability, mostly because we had enough trust in our terraform process so I decided to automate the whole thing through AWS System Manager across the fleet and guard the feature to work only for security patches targeting the kernel. Briefly, every VM that came up would have the necessary packages installed and from there live patching would execute periodically (I believe once a week) or manually. At some point after a quarter or something we had to devise a way to tag VMs to be excluded from this but this was relatively easy to do and most of the exclusions were testing infra anyway.