8 ms·

Learnings from kCTF VRP's 42 Linux kernel exploits submissions

206 points by tim_sw 3y ago

klooney 3y ago
Oof. I hope this doesn't get wedge io_uring into a place where people can't use it because it's disabled, so they don't, and the whole improvement gets stuck and wasted.
- localplume 3y ago
  [dead]
rektide 3y ago
[flagged]
- cratermoon 3y ago
  "io_uring vulnerabilities were used in all the submissions which bypassed our mitigations."
  - wahern 3y ago
    Everything about io_uring has screamed security nightmare from day 1--the amount of kernel attack surface it opens up, the fact it put little thought into how it interacted with process and privilege boundaries and related overarching models (see, e.g., interactions with setuid and fork/clone), increased opportunity to exploit race conditions, etc. From a pure performance standpoint it's amazing, but it's circumventing a slew of interfaces that were, at least in part, designed to restrict process privilege--more importantly than merely designed, but refined and bug-fixed over decades. The use of lockless buffers for high throughput IPC works better in microkernel designs. Linux being a monolithic kernel, io_uring is like one, big trap door for exploits.
  - chrismorgan 3y ago
    Rephrasing the sentiments of the article: • io_uring is quite new, and currently still having some severe security problems. • Most kernel security bugs, they can mitigate effectively. But their mitigations aren’t fully effective on io_uring. • Therefore, until the security problems die down or until they can effectively mitigate io_uring security bugs, they’re disabling it, because the security concerns outweigh the performance advantages.
    - johncolanduoni 3y ago
      > • Most kernel security bugs, they can sandbox effectively. But their sandboxing isn’t fully effective on io_uring. This is definitely not true in general for their platform: I don’t believe they do any shared-kernel sandboxing of untrusted code at this point and their primary sandbox gVisor (used for stuff like App Engine, Cloud Functions, and Cloud Run) just doesn’t simulate io_uring at all.
      - chrismorgan 3y ago
        “Sandboxing” was the wrong word to use; I was referring to what they call “mitigations”. Following through a few links, you can end up at https://github.com/thejh/linux/blob/slub-virtual/MITIGATION_README https://github.com/thejh/linux/blob/slub-virtual/MITIGATION_... which describes what it all actually is. I was figuring that sandboxing was probably close enough as a description, and maybe more easily understood, but that wasn’t wise. I’ve changed it to use their wording.
  - hedora 3y ago
    The example in the article was fixed by backporting a patch from 5.15 to 5.10. 5.10 was released in October 2020. 5.15 was from November 2021. They also changed the program so they only pay out for bugs in a recent kernel. It seems more prudent to disable io_uring only on kernels that are say, over 2 years old. Alternatively, they could upgrade their known-insecure kernels. I doubt the io_uring patch is the only missing one that was produced in the last three years.
    - 7e 3y ago
      That's not my read. I read that they've found a bunch of exploits in io_uring but are only willing to talk publicly about previously disclosed ones, like the one fixed by backporting to 5.10. Just because they used that as an example doesn't mean they're refusing to backport security fixes. It just means they don't want to talk about exploits they found in newer kernels, which is understandable.
- throwawaaarrgh 3y ago
  [flagged]
jpgvm 3y ago
Shame but understandable. io_uring (like netlink) is essentially a completely new interface to kernel-land things so until security boundary stuff is sorted out this is probably the best move.
- hedora 3y ago
  No. It was completely new a few years ago, and they’re running kernels from three years ago that are missing security patches from two years ago.
  - sirdarckcat 3y ago
    [citation needed] * ChromeOS, COS and Android are opensource, and they are all up to date. Mind elaborating?
    - bewaretheirs 3y ago
      With respect to Android, I was under the impression that (in part because key members of the Linux community are allergic to the notion of a stable device driver ABI) the specific release of linux used on a particular model of phone was typically frozen at some point before the time the phone shipped, and subsequently only patched as needed. The mere fact that the source is available doesn't mean that someone will take the (potentially unbounded) time to port it to a future release and test it sufficiently.
  - viraptor 3y ago
    I'd like to see a source for this. Even if they do run older kennels, it doesn't mean they're missing patches. Google has significant involvement in security and enough engineers to roll their own systems. I don't expect they'd be missing any relevant patches, much less for years. (I know this was an older patch in this case, but they did mention other mitigations, so it may have been a considered choice not to backport) Given how large they are as a target, we'd also hear about them being exploited all the time if that was true.
  - greesil 3y ago
tptacek 3y ago
This is an editorialized title, and should instead be "Learnings from kCTF VRP's 42 Linux kernel exploits submissions".
- KerrAvon 3y ago
  The io_uring finding is the only novel tidbit.
- dang 3y ago
  Thanks! Changed now. Submitted title was "io_uring disabled across Google platforms".
  - NavinF 3y ago
    IMO the original title was way better since it doesn't bury the lede. For better or worse, very few programmers care about "Learnings from kCTF". A lot of programmers care about io_uring being disabled.
    - tptacek 3y ago
      On HN we use the original title unless it's actively misleading, or it's linkbait --- obviously, this title is anything but that. Titles are community property: it's not the prerogative of the submitter to determine the most interesting thing in the story and retitle it accordingly.
- DrBazza 3y ago
  "Learnings" - should be lessons. We don't go on 'drivings' or 'travellings' we go on journeys. Which lead me down this rabbit hole - https://english.stackexchange.com/questions/546959/why-is-learnings-considered-acceptable https://english.stackexchange.com/questions/546959/why-is-le... Observation bias, but I first noticed this word popping up in tech talks from Microsoft first.
  - insanitybit 3y ago
    How odd that a language stackexchange wouldn't just immediately quote Fowler's, which has a whole section on such words, as I recall.
p-e-w 3y ago
> As the Linux kernel is a key component not just for Google, but for the Internet, we started heavily investing in this area. We extended the VRP's scope and maximum reward in 2021 (to $50k), then again in February 2022 (to $91k), and finally in August 2022 (to $133k). So an award roughly equal to four months' salary for an average Silicon Valley engineer is considered "heavily investing" in security, when it comes to potentially industry-shattering exploits? I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.
- vasco 3y ago
  Who is lawfully paying out more for bounties on open source projects they didn't even start? You also seem to discount the monetary value of not becoming a criminal by selling exploits unlawfully. Do you want to be the person that reported a vulnerability to increase security in the world or to give more power to spy agencies or crime syndicates?
  - TechBro8615 3y ago
    How do you define "selling exploits unlawfully?" I'm not aware of any law that says only Google is allowed to purchase information disclosing exploits in open source software.
    - gretch 3y ago
      Because if this exploit is ever used in breach of the computer fraud and abuse act, now you are down as a conspirator. When you sell this to another party, are you going to report it on your taxes? If you do report it, is your buyer going to like that? If you clear all those conditions, then by all means.
      - xyzzyz 3y ago
        > When you sell this to another party, are you going to report it on your taxes? If you do report it, is your buyer going to like that? You don't need to declare who gave you the money on the income tax filing.
twunde 3y ago
For anyone who wants an easily digestible read on io_uring exploits I love my former colleague's post: https://chompie.rip/Blog+Posts/Put+an+io_uring+on+it+-+Exploiting+the+Linux+Kernel https://chompie.rip/Blog+Posts/Put+an+io_uring+on+it+-+Explo...
- stevekemp 3y ago
  That was an outstanding writeup, thanks for sharing it.
- nitinreddy88 3y ago
  This deserves its own thread HN. I don't have minimal knowledge on Kernel yet I understood many things about io_uring
  - fragmede 3y ago
    https://news.ycombinator.com/item?id=36353845 https://news.ycombinator.com/item?id=36353845 I'm surprised it hasn't been submitted before.
    - insanitybit 3y ago
      https://news.ycombinator.com/item?id=30605356 https://news.ycombinator.com/item?id=30605356 (For context) It was submitted under graplsecurity.com, chompie has since mirrored the research as the company no longer exists.
      - menaerus 3y ago
        Is it known what happened to them?
        insanitybit 3y ago
        Offers were made to join Datadog, where 5 of 7 employees joined (myself included).
laserbeam 3y ago
As a reader... I can't tell if there's a fundamental design flaw with io_uring, or just growing pains because the implementation is newish. Like how speculative execution had fundamental vulnerabilities which had to do with timing your code, not a specific implementation. The title makes it look like a design problem with io_uring, Google's reaction makes it feel like a design issue. But the article itself sounds like growing pains.
- sylware 3y ago
  Speculative execution is not fixed in modern CPUs? Or this is related to old ones?_
  - wahern 3y ago
    > Speculative execution is not fixed in modern CPUs? Very generally speaking, not really. It's a huge class and the pace of discovery of new vectors has continued nearly unabated since Spectre. See https://en.wikipedia.org/wiki/Transient_execution_CPU_vulnerability https://en.wikipedia.org/wiki/Transient_execution_CPU_vulner... The effectiveness and cost of various remediations--software, microcode, hardware--varies depending on the particular vector, so some vectors can be considered fixed, while other vectors, such as some of those relating to SMT, might never be fully fixed without effectively giving up some of the benefits of SMT. Some of the more robust theoretical remedies, like cache coloring and kernel scheduler-based process affinity schemes, still haven't seen much investment or attention, so I would expect the state of things to remain relatively crappy. It just may seem less crappy because the industry has become inured to the threat, and also perhaps because there's usually much lower hanging fruit for attackers (i.e. traditional exploits like with io_uring), so these vectors are rarely used in the wild.
    - sylware 3y ago
      Ok, then it means modern CPUs which are sold are not fixed. Weird, I was told Zen3 is fixed.
      - 3y ago
tlamponi 3y ago
> While io_uring brings performance benefits, and promptly reacts to security > issues with comprehensive security fixes (like backporting the 5.15 version > to the 5.10 stable tree), it is a fairly new part of the kernel. As such, > io_uring continues to be actively developed, but it is still affected by > severe vulnerabilities and also provides strong exploitation primitives. For > these reasons, we currently consider it safe only for use by trusted > components. As often in the kernel (and probably everywhere): there are people caring for performance and there are people caring for security, and a lot of the time there's not that much overlap. Ensuring that io_uring is covered by the Linux Security Modules (LSM) framework, that provides hooks for things like AppArmor or SELinux (and a variety of others, some even can even get stacked), was only done after it was already quite popular. So FWICT, even if io_uring is existing since a while, 1) the security experts vetting it are looking only closer since relatively recently and 2) it's architecture is quite different from other kernel user space interfaces, which certainly needs some adaptions on applying the established security mechanisms of the kernel, especially as the project still wants to stay high performance. IMO, this doesn't mean io_uring is inherently unsafe, or has no use, but that one certainly needs to evaluate if great async submission model is worth the potential security implications. E.g., in an isolate environment, where no untrusted third-party apps or people have access, it can be fine, while especially in environments that Google runs it isn't worth the trouble for now. But, I think that the features which io_uring brings are too nice to have for quite some (I/O or syscall heavy) workloads even in those environments, so there will be an incentive it to the integration into the existing Linux security stack and mechanisms, and evolve the latter to be also a good fit for this rather different interface.
- udunadan 3y ago
  Yes, it is no different than OPSEC with heavy procedures in place which take the actor's mobility at the sake of security. A universal security property.
- rfoo 3y ago
  > integration into the existing Linux security stack It's less about integration into the security stack. It's just that the implementation of io_uring is hard to get right, and at 2023 there seems to be an endless stream of race condition bugs. Quite like user namespaces ten years ago. It doesn't matter how good your security design is. Hackers just find a memory corruption bug and get around everything. To be fair it isn't endless, finding an io_uring bug which can be exploited to cause memory corruption and ultimately kernel code execution in 5.15 tree is measurably harder now. But io_uring moves very fast and it's an entirely different story for mainline kernels.
deleted 3y ago
[deleted]
retrocryptid 3y ago
"Lessons." The word you're looking for is "lessons." As in "Lessons from kCTF VRP's 42 Linux kernel exploits submissions."
- dang 3y ago
  "Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
nathants 3y ago
when i was implementing lsm hooks for mighty-snitch last year, i remember attempting the io_uring hooks but quickly gave up. the normal lsm hooks all do something well scoped, specific, and take meaningful parameters. the io_uring hooks take what appear to be a turing complete language snippets as a parameter. the hooks now all return false (disabled).

206 points by tim_sw 3y ago

Oof. I hope this doesn't get wedge io_uring into a place where people can't use it because it's disabled, so they don't, and the whole improvement gets stuck and wasted.

localplume 3y ago
[dead]

rektide 3y ago

[flagged]

cratermoon 3y ago
"io_uring vulnerabilities were used in all the submissions which bypassed our mitigations."
- wahern 3y ago
  Everything about io_uring has screamed security nightmare from day 1--the amount of kernel attack surface it opens up, the fact it put little thought into how it interacted with process and privilege boundaries and related overarching models (see, e.g., interactions with setuid and fork/clone), increased opportunity to exploit race conditions, etc. From a pure performance standpoint it's amazing, but it's circumventing a slew of interfaces that were, at least in part, designed to restrict process privilege--more importantly than merely designed, but refined and bug-fixed over decades. The use of lockless buffers for high throughput IPC works better in microkernel designs. Linux being a monolithic kernel, io_uring is like one, big trap door for exploits.
- chrismorgan 3y ago
  Rephrasing the sentiments of the article: • io_uring is quite new, and currently still having some severe security problems. • Most kernel security bugs, they can mitigate effectively. But their mitigations aren’t fully effective on io_uring. • Therefore, until the security problems die down or until they can effectively mitigate io_uring security bugs, they’re disabling it, because the security concerns outweigh the performance advantages.
  - johncolanduoni 3y ago
    > • Most kernel security bugs, they can sandbox effectively. But their sandboxing isn’t fully effective on io_uring. This is definitely not true in general for their platform: I don’t believe they do any shared-kernel sandboxing of untrusted code at this point and their primary sandbox gVisor (used for stuff like App Engine, Cloud Functions, and Cloud Run) just doesn’t simulate io_uring at all.
    - chrismorgan 3y ago
      “Sandboxing” was the wrong word to use; I was referring to what they call “mitigations”. Following through a few links, you can end up at https://github.com/thejh/linux/blob/slub-virtual/MITIGATION_README https://github.com/thejh/linux/blob/slub-virtual/MITIGATION_... which describes what it all actually is. I was figuring that sandboxing was probably close enough as a description, and maybe more easily understood, but that wasn’t wise. I’ve changed it to use their wording.
- hedora 3y ago
  The example in the article was fixed by backporting a patch from 5.15 to 5.10. 5.10 was released in October 2020. 5.15 was from November 2021. They also changed the program so they only pay out for bugs in a recent kernel. It seems more prudent to disable io_uring only on kernels that are say, over 2 years old. Alternatively, they could upgrade their known-insecure kernels. I doubt the io_uring patch is the only missing one that was produced in the last three years.
  - 7e 3y ago
    That's not my read. I read that they've found a bunch of exploits in io_uring but are only willing to talk publicly about previously disclosed ones, like the one fixed by backporting to 5.10. Just because they used that as an example doesn't mean they're refusing to backport security fixes. It just means they don't want to talk about exploits they found in newer kernels, which is understandable.
throwawaaarrgh 3y ago
[flagged]

jpgvm 3y ago

Shame but understandable. io_uring (like netlink) is essentially a completely new interface to kernel-land things so until security boundary stuff is sorted out this is probably the best move.

hedora 3y ago
No. It was completely new a few years ago, and they’re running kernels from three years ago that are missing security patches from two years ago.
- sirdarckcat 3y ago
  [citation needed] * ChromeOS, COS and Android are opensource, and they are all up to date. Mind elaborating?
  - bewaretheirs 3y ago
    With respect to Android, I was under the impression that (in part because key members of the Linux community are allergic to the notion of a stable device driver ABI) the specific release of linux used on a particular model of phone was typically frozen at some point before the time the phone shipped, and subsequently only patched as needed. The mere fact that the source is available doesn't mean that someone will take the (potentially unbounded) time to port it to a future release and test it sufficiently.
- viraptor 3y ago
  I'd like to see a source for this. Even if they do run older kennels, it doesn't mean they're missing patches. Google has significant involvement in security and enough engineers to roll their own systems. I don't expect they'd be missing any relevant patches, much less for years. (I know this was an older patch in this case, but they did mention other mitigations, so it may have been a considered choice not to backport) Given how large they are as a target, we'd also hear about them being exploited all the time if that was true.
- greesil 3y ago

tptacek 3y ago

This is an editorialized title, and should instead be "Learnings from kCTF VRP's 42 Linux kernel exploits submissions".

KerrAvon 3y ago
The io_uring finding is the only novel tidbit.
dang 3y ago
Thanks! Changed now. Submitted title was "io_uring disabled across Google platforms".
- NavinF 3y ago
  IMO the original title was way better since it doesn't bury the lede. For better or worse, very few programmers care about "Learnings from kCTF". A lot of programmers care about io_uring being disabled.
  - tptacek 3y ago
    On HN we use the original title unless it's actively misleading, or it's linkbait --- obviously, this title is anything but that. Titles are community property: it's not the prerogative of the submitter to determine the most interesting thing in the story and retitle it accordingly.
DrBazza 3y ago
"Learnings" - should be lessons. We don't go on 'drivings' or 'travellings' we go on journeys. Which lead me down this rabbit hole - https://english.stackexchange.com/questions/546959/why-is-learnings-considered-acceptable https://english.stackexchange.com/questions/546959/why-is-le... Observation bias, but I first noticed this word popping up in tech talks from Microsoft first.
- insanitybit 3y ago
  How odd that a language stackexchange wouldn't just immediately quote Fowler's, which has a whole section on such words, as I recall.

p-e-w 3y ago

> As the Linux kernel is a key component not just for Google, but for the Internet, we started heavily investing in this area. We extended the VRP's scope and maximum reward in 2021 (to $50k), then again in February 2022 (to $91k), and finally in August 2022 (to $133k). So an award roughly equal to four months' salary for an average Silicon Valley engineer is considered "heavily investing" in security, when it comes to potentially industry-shattering exploits? I imagine that anyone who knows where to look can easily find dozens of interested buyers willing to pay a lot more than that, from intelligence agencies and their contractors to crime syndicates.

vasco 3y ago
Who is lawfully paying out more for bounties on open source projects they didn't even start? You also seem to discount the monetary value of not becoming a criminal by selling exploits unlawfully. Do you want to be the person that reported a vulnerability to increase security in the world or to give more power to spy agencies or crime syndicates?
- TechBro8615 3y ago
  How do you define "selling exploits unlawfully?" I'm not aware of any law that says only Google is allowed to purchase information disclosing exploits in open source software.
  - gretch 3y ago
    Because if this exploit is ever used in breach of the computer fraud and abuse act, now you are down as a conspirator. When you sell this to another party, are you going to report it on your taxes? If you do report it, is your buyer going to like that? If you clear all those conditions, then by all means.
    - xyzzyz 3y ago
      > When you sell this to another party, are you going to report it on your taxes? If you do report it, is your buyer going to like that? You don't need to declare who gave you the money on the income tax filing.

twunde 3y ago

For anyone who wants an easily digestible read on io_uring exploits I love my former colleague's post: https://chompie.rip/Blog+Posts/Put+an+io_uring+on+it+-+Exploiting+the+Linux+Kernel https://chompie.rip/Blog+Posts/Put+an+io_uring+on+it+-+Explo...

stevekemp 3y ago
That was an outstanding writeup, thanks for sharing it.
nitinreddy88 3y ago
This deserves its own thread HN. I don't have minimal knowledge on Kernel yet I understood many things about io_uring
- fragmede 3y ago
  https://news.ycombinator.com/item?id=36353845 https://news.ycombinator.com/item?id=36353845 I'm surprised it hasn't been submitted before.
  - insanitybit 3y ago
    https://news.ycombinator.com/item?id=30605356 https://news.ycombinator.com/item?id=30605356 (For context) It was submitted under graplsecurity.com, chompie has since mirrored the research as the company no longer exists.
    - menaerus 3y ago
      Is it known what happened to them?
      - insanitybit 3y ago
        Offers were made to join Datadog, where 5 of 7 employees joined (myself included).

laserbeam 3y ago

As a reader... I can't tell if there's a fundamental design flaw with io_uring, or just growing pains because the implementation is newish. Like how speculative execution had fundamental vulnerabilities which had to do with timing your code, not a specific implementation. The title makes it look like a design problem with io_uring, Google's reaction makes it feel like a design issue. But the article itself sounds like growing pains.

sylware 3y ago
Speculative execution is not fixed in modern CPUs? Or this is related to old ones?_
- wahern 3y ago
  > Speculative execution is not fixed in modern CPUs? Very generally speaking, not really. It's a huge class and the pace of discovery of new vectors has continued nearly unabated since Spectre. See https://en.wikipedia.org/wiki/Transient_execution_CPU_vulnerability https://en.wikipedia.org/wiki/Transient_execution_CPU_vulner... The effectiveness and cost of various remediations--software, microcode, hardware--varies depending on the particular vector, so some vectors can be considered fixed, while other vectors, such as some of those relating to SMT, might never be fully fixed without effectively giving up some of the benefits of SMT. Some of the more robust theoretical remedies, like cache coloring and kernel scheduler-based process affinity schemes, still haven't seen much investment or attention, so I would expect the state of things to remain relatively crappy. It just may seem less crappy because the industry has become inured to the threat, and also perhaps because there's usually much lower hanging fruit for attackers (i.e. traditional exploits like with io_uring), so these vectors are rarely used in the wild.
  - sylware 3y ago
    Ok, then it means modern CPUs which are sold are not fixed. Weird, I was told Zen3 is fixed.
    - 3y ago

tlamponi 3y ago

> While io_uring brings performance benefits, and promptly reacts to security > issues with comprehensive security fixes (like backporting the 5.15 version > to the 5.10 stable tree), it is a fairly new part of the kernel. As such, > io_uring continues to be actively developed, but it is still affected by > severe vulnerabilities and also provides strong exploitation primitives. For > these reasons, we currently consider it safe only for use by trusted > components. As often in the kernel (and probably everywhere): there are people caring for performance and there are people caring for security, and a lot of the time there's not that much overlap. Ensuring that io_uring is covered by the Linux Security Modules (LSM) framework, that provides hooks for things like AppArmor or SELinux (and a variety of others, some even can even get stacked), was only done after it was already quite popular. So FWICT, even if io_uring is existing since a while, 1) the security experts vetting it are looking only closer since relatively recently and 2) it's architecture is quite different from other kernel user space interfaces, which certainly needs some adaptions on applying the established security mechanisms of the kernel, especially as the project still wants to stay high performance. IMO, this doesn't mean io_uring is inherently unsafe, or has no use, but that one certainly needs to evaluate if great async submission model is worth the potential security implications. E.g., in an isolate environment, where no untrusted third-party apps or people have access, it can be fine, while especially in environments that Google runs it isn't worth the trouble for now. But, I think that the features which io_uring brings are too nice to have for quite some (I/O or syscall heavy) workloads even in those environments, so there will be an incentive it to the integration into the existing Linux security stack and mechanisms, and evolve the latter to be also a good fit for this rather different interface.

udunadan 3y ago
Yes, it is no different than OPSEC with heavy procedures in place which take the actor's mobility at the sake of security. A universal security property.
rfoo 3y ago
> integration into the existing Linux security stack It's less about integration into the security stack. It's just that the implementation of io_uring is hard to get right, and at 2023 there seems to be an endless stream of race condition bugs. Quite like user namespaces ten years ago. It doesn't matter how good your security design is. Hackers just find a memory corruption bug and get around everything. To be fair it isn't endless, finding an io_uring bug which can be exploited to cause memory corruption and ultimately kernel code execution in 5.15 tree is measurably harder now. But io_uring moves very fast and it's an entirely different story for mainline kernels.

deleted 3y ago

[deleted]

retrocryptid 3y ago

"Lessons." The word you're looking for is "lessons." As in "Lessons from kCTF VRP's 42 Linux kernel exploits submissions."

dang 3y ago
"Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

nathants 3y ago

when i was implementing lsm hooks for mighty-snitch last year, i remember attempting the io_uring hooks but quickly gave up. the normal lsm hooks all do something well scoped, specific, and take meaningful parameters. the io_uring hooks take what appear to be a turing complete language snippets as a parameter. the hooks now all return false (disabled).