7 ms·

How to handle people dismissing io_uring as insecure? (2024)

122 points by nromiun 1y ago

johnisgood 1y ago
It is kind of like PHP. Their views of PHP is still stuck at PHP 5. We have PHP 8.
- nicman23 1y ago
  the work that the people in PHP did for 7 is criminally underrated
- continuational 1y ago
  Programming languages are different. They usually can't be "repaired" once broken, because they need to maintain backwards compatibility.
  - concerned_user 1y ago
    This is only true if people stick to a version of a language and don't upgrade. If you upgrade then, for example, you can't run all of your PHP 5 code in PHP 8, most of it you can but you will have to change the parts that are broken, which are the areas that are repaired in PHP 8. Same goes for other languages like C# or Python
  - pjmlp 1y ago
    They are software products like anything else.
  - nromiun 1y ago
    They don't "need" to maintain backwards compatibility. Several major languages have broken it by now.
  - jeroenhd 1y ago
    Programming languages deprecate features in standard libraries all the time. As PHP did, causing many PHP 5.3 applications to fail catastrophically once the warnings added to PHP 5.4, 5.5, and 5.6 were turned into errors. Of course, maintained software rarely ever runs into this issue. The standard libraries were the lacking part in PHP. The language itself was never a serious problem.
- xlii 1y ago
  Maybe it is (mine is for sure) but that's a "bitten by the dog once" case. You got bitten and everyone around you assures you won't get bitten again, but the pain was real and you still have a scar from the event. Why bother or invest risking another bite if there are other places to be which had never bitten before. Over my career I hated technology 3 times. First was PHP, second was Python during Python 2/3 fiasco and third was CoffeeScript. Edit: till this day one of my favorite meme is titled "PHP: Training wheels without a bike"
  - yjftsjthsd-h 1y ago
    A few years ago, I used BTRFS on a laptop. Single disk, no RAID of any kind, OpenSUSE (which favored BTRFS, so I expected it to be as well supported as could be had), nothing fancy. After losing the root filesystem twice, I decided that maybe I shouldn't trust BTRFS. Since then, I've been told that it's totally better now, that all the problems are with bad RAID setups, and it's safe and won't lose my data. Anyways, as I type this from a laptop running on ZFS, I remain somewhat cautious.
    - jiggawatts 1y ago
      > Since then, I've been told that it's totally better now I've read through some bug reports, and I assure you that BTRFS remains a horror show. Saying that comments from the dev team "don't inspire confidence" is putting it mildly. ZFS is the diametric opposite of this, where blog posts from the team working on it made me realise that they're moving the state of the art into new territory.
      - johnisgood 1y ago
        I really want to try bcachefs, I wonder if anyone has used it (with encryption, and if their encryption supports plausible deniability or not). BTW, is ZFS on Linux good now?
        chasil 1y ago
        Bcachfs has been removed from Linux. https://forums.truenas.com/t/bcachefs-dropped-from-linux-6-17/47287 https://forums.truenas.com/t/bcachefs-dropped-from-linux-6-1... ZFS is an install option on Ubuntu. Running it taints the kernel.
- Xenoamorphous 1y ago
  It’s true of most languages that have been around for a while. E.g. people who don’t write Javascript think of it as it was pre-ES6, not to mention that many people don’t write JS directly and use Typescript instead. Same with Java, it’s somehow stuck in time in Java 6 in some people’s minds.
- WJW 1y ago
  It's with so many things! Some evergreens just from last week here on HN: - Rails is horribly slow! - Python is still stuck migrating to Python 3! - MySQL doesn't scale! - Haskell tooling sucks! - io_uring is insecure! - ... and dozens more just in the programming world. These are just off the top of my head. Probably hundreds more in the wider engineering world.
  - dijit 1y ago
    MySQL would happily clobber your data silently for decades though. I think the situation is better now, but that was worthy criticism for a long time; like anything though, those that bought into it wholeheartedly couldn’t take the criticisms. I wonder if there’s a word for that, seems to be a common issue.
    - als0 1y ago
      > those that bought into it wholeheartedly couldn’t take the criticisms. "invested"
  - hyghjiyhu 1y ago
    From my perspective python3 migration is very much complete. Package managers is the evergreen mess now.
- nurettin 1y ago
  Wow. My view is stuck at php 4. Didn't even know 5 existed. I only remember Facebook making a big deal of hhvm and php 7 and laravel being considered great but not as great as rails or django.
  - johnisgood 1y ago
    It is worth checking out PHP for when you are building backend. I did not use Laravel because I had some issues with it (lack of flexibility in some cases) so I ended up creating my stuff from scratch. I am glad I did. It is well documented, modular, maintainable, not bloated, etc. I can give you the resources I have heavily used.
    - nurettin 1y ago
      You're very kind. Thanks for the recommendation. If for some reason I have to branch out, I will check php again.
      - johnisgood 1y ago
        Hey, I have not recommended anything yet (besides PHP). :D I like "PHP & MySQL: Novice to Ninja" quite a lot. I have used it more so than the other books, so I definitely recommend it. You can find the code archive (code from the book) here: https://github.com/spbooks/phpmysql7 https://github.com/spbooks/phpmysql7. FWIW I recommend these books (in no particular order): ├── Modern PHP Security (2020).pdf ├── PHP 8 - OOP.pdf ├── PHP 8 - Quick Scripting Reference (2021).pdf ├── PHP 8 Programming Tips, Tricks, and Best Practices (2021).pdf ├── PHP MySQL - Novice to Ninja.pdf ├── PHP and MySQL for Advanced Learning.pdf ├── PHP in Action (2007).pdf ├── Programming PHP (2020).pdf └── The Art of Modern PHP 8.pdf
- dolmen 1y ago
  It is kind of like Perl. Their views of Perl is still stuck at CGIs of the 90's. I still think that Perl was and is still a better language than PHP and Python. And people never had a serious look at raku because of the Perl heritage. But that ship has sailed.
  - em-bee 1y ago
    huh? that would suggest that perl has changed so much since the 90s that those views would no longer be valid. CGIs in the 90s were written in perl 5, and perl hasn't evolved much since then. but perl was never a bad language. it had/has a quirky syntax. if you were coming from a lot of commandline work using sed, awk, tr, and many other tools to manipulate text in shell scripts, then perl made a lot of sense. and while raku may have been a great improvement, perls reputation was tied to its syntax. the readability and writeability of the code. and that's the one thing that raku did not change significantly. so unlike other examples i don't see how peoples views of perl could improve
    - johnisgood 1y ago
      I love Perl. Is Raku worth trying considering I love Perl for its performance, syntax, brevity, and so forth? Raku feels like a different language to me, and it seems not-so-serious, maybe because of the name and their mascot (which is unfair to the language).
      - em-bee 1y ago
        i haven't looked at it in detail, but if you like perl and you are looking for an interesting new language that has the spirit of perl, i'd say it is worth a try
    - topspin 1y ago
      > and perl hasn't evolved much since then Perl (5) has been evolving, quietly. The language has had many nice additions over the years. Some of it is backporting from 6.
frevib 1y ago
If you like to play around with io_uring networking, here is a very simple echo server example: https://github.com/frevib/io_uring-echo-server https://github.com/frevib/io_uring-echo-server
- deleted 1y ago
  [deleted]
vjerancrnjak 1y ago
I think it's the Wikipedia article. https://en.wikipedia.org/wiki/Io_uring https://en.wikipedia.org/wiki/Io_uring Very easy to just quote that without any io_uring experience. > In June 2023, Google's security team reported that 60% of the exploits submitted to their bug bounty program in 2022 were exploits of the Linux kernel's io_uring vulnerabilities. As a result, io_uring was disabled for apps in Android, and disabled entirely in ChromeOS as well as Google servers. Docker also consequently disabled io_uring from their default seccomp profile.
- yjftsjthsd-h 1y ago
  I like how someone helpfully added > Although initial async offload design in io_uring could be problematic, later kernels changed the thread model. After such improvements, there were no known inherent problems with it and its development is very careful with new features. Considering that a performant async framework with a user facing API is complex, it was to be expected that issues would be found initially. After initial issues have been addressed, it is not any less secure than anything else in the kernel and io_uring acceptance quickly grew in production. Some of its criticism are also based on wrong or outdated assumptions.[14] ...but the only citation is a link to this GH thread, which doesn't support the claims made.
- znpy 1y ago
  Jens Axboe replies on the very first line of the thread: > As I'm sure you know, this is all mostly centered around a) google using an old kernel on android
  - fulafel 1y ago
    But also > My hope is that this reputation will go away eventually, as less issues are found in the code. this has not yet happened like this other comment shows: https://news.ycombinator.com/item?id=44632639 https://news.ycombinator.com/item?id=44632639
- flomo 1y ago
  Without going into the weeds, there has be some vendor support, and that vendor is obviously not google. How to convince people: Get it into RHEL.
fabian2k 1y ago
I admit, I was confused a bit as well about the io_ring security reputation. Though I didn't really follow the topic, so the clarification that this was mostly about an older design on Android is quite helpful. The potential performance benefits are quite compelling, e.g. in Postgres 18 you reportedly can get a 3x speedup over the old sync behaviour in simple read queries.
yjftsjthsd-h 1y ago
> How to handle people dismissing io_uring as insecure? It is, in the general case, hard to prove something secure (because it's hard to prove a negative). It might help to show CVEs per month/year/whatever related to it vs anything else, preferably with a clear downward trend. For example, you could look at https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring ... although I struggle to read that as supporting the case you want to make. > I have had to deal with handful of these people from different sectors as well. Since I am actively working on project based on io_uring, I have had people saying all kinds of hmm... "crap", its so baseless! Can't even talk to them with actual facts. So what are those facts? Because all this thread has is people handwaving that it used to have a worse design, and everything has bugs and this isn't different, and implying that it's better. If it's better, show that.
- yorwba 1y ago
  > you could look at https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring CVE count by year: 2019: 1 2020: 1 2021: 10 2022: 15 2023: 19 2024: 21 2025: 10
  - franciscop 1y ago
    I expect CVEs to be directly proportional to project usage and popularity, and inversely proportional to maturity, which makes things a lot more complicated.
    - delusional 1y ago
      And also directly proportional to the publicity of the CVE system. If you're creative enough in your writing, any bug in any program can be filed as a CVE, and filing CVEs is much more interesting carreer wise than filing bug reports. Any decently sized project has probably seen an increase in reported CVEs over the past 5 years, simply because the number of CVEs total has grown.
    - sigmoid10 1y ago
eviks 1y ago
> Perhaps if it had been written in Rust we would've been better off ;-). So the answer is known :)
ibotty 1y ago
One problem is that you can't filter its "syscalls" as you can regular syscalls. This removes a security boundary that e.g. container runtimes regularly use. So you cannot use it in your regular kubernetes cluster without weakening its security for these pods.
- altairprime 1y ago
  Is that a true limitation that cannot be overcome? Are solutions possible and/or available, but require further work to be shipped?
- spwa4 1y ago
  What regular filter for syscalls do you use?
  - lima 1y ago
    seccomp-bpf, for instance.
  - holowoodman 1y ago
    seccomp BPF, eBPF, in a way SELinux/AppArmor/Tomoyo/..., maybe you can even call namespaces some kind of syscall filter. And then there is the auditing framework, where you can at least record which critical syscalls were performed. Nowadays its mostly a combination of eBPF, SELinux and auditd plus namespaces in case of containers. Usually in the combination that some distro ships, so nothing really fancy.
- holowoodman 1y ago
  This just reinforces the (maybe unfounded) impression that security is a secondary consideration, and performance is primary. I'd use io_uring in a heartbeat on a dedicated system where the job is only I/O and security isolation isn't a concern. But multiuser/multiapplication/networked? Not a chance.
  - weitendorf 1y ago
    I think there is a very large amount of overlap between the people who 1. know what io_uring is 2. are interested in performance enough to look at improvements based on new linux kernel system calls and talk about it in public 3. care about security in multitenant environments or the syscalls used by third party libraries I think io_uring right now probably makes a lot of sense for HPC and highly technical, performance-sensitive financial stuff, but they can be kind of insular. I don't think most linux hobbyists really need the performance benefits enough t care about it, and most businesses are using a major cloud vendor/don't have the scale or expertise to be thinking about this kind of stuff. Which leaves major cloud providers and really big businesses like Meta with their own internal clouds as the ones that stand to benefit enough to care about performance while really caring about security
perks_12 1y ago
Works on the most sophisticated io_uring project, has no idea how to defend it. Nice.
- topspin 1y ago
  It's tragic, and it bothers me that Jens Axboe's work is suffering due to this. Obviously, with the clarity of hindsight, this might have been avoided. Now the cost of the damage it high. My idea: consider formal verification. A rigorous formal proof of behavior is capable of solving actual flaws and probably capable of overcoming the bad PR. That would require a great and ongoing effort. However, given the nature of io_uring, it's likely rather amenable to formal verification, and ultimately it is probably necessary. Perhaps our new "AI" tools could greatly reduce the effort. Anyhow, that's my brilliant thought...
  - topspin 1y ago
    Replying to myself. io_uring is only a more ambitious manifestation of a pattern that has been commonplace for a long time. High performance network, storage and other hardware use circular buffers internally in their drivers to process commands and return results. io_uring is attempting to realize the same pattern, but for the entire operating system interface. That's a really big idea. It has me asking whether this isn't how all "IPC" should be done, where one generalizes all of the running processes, including the "kernel." Ultimately it has significant implications for CPU architecture, supplanting traditional patterns. When I wrote "given the nature of io_uring," I had something specific in mind wrt formal verification. io_uring fundamentally consumes a stream of bytes. At the front end of that is a great deal of validation code to ensure the steam contents are sane. At the backend, the commands are mapped to existing kernel functionality. It's the front end I think could be well served by formal verification.
weinzierl 1y ago
Convince AWS to allow it in managed container and serverless runtimes. When people see what Amazon does it is far too easy to conclude: "Huh, if Amazon blocks it there must me something to it."
txdv 1y ago
One of the most interesting aspects of software development is that it is still done by humans. Information moves slowly and is very generalised, with little attention to detail. I have seen this multiple times when developers were still reciting old benchmarks, taken out of context. It often becomes very tribal and centred around technologies.
serial_dev 1y ago
> How can I help people out when they tell me that io_uring is insecure? Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong. If you start there, you will fail to convince people of anything, because you automatically dismiss their claim, without thinking about what they might have seen and what they might think. A better starting point would be wanting to get to the bottom of it, and assess the security of io_uring. If you start from that point and you give it an honest, thorough assessment, and it turns out it "looks secure", you'll have an easier time convincing people. You might still be wrong (assessing io_uring's security is not trivial), but at least you tried to understand why people think that. And reminder: it's ok to "agree to disagree".
- VWWHFSfQ 1y ago
  > Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong. I think this is the right approach. We know that io_uring has a somewhat significant history of critical security issues. It's not enough just to point out that "these 3 critical CVEs were fixed in the last 12 months, it's secure now!" Reputation and trust has to be built over a long period of time.
- jay-barronville 1y ago
  > Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong. If you start there, you will fail to convince people of anything, because you automatically dismiss their claim, without thinking about what they might have seen and what they might think. Bingo. This is the correct approach. Very well said!
- Arch-TK 1y ago
  People are saying: "Oxygen is blue and that's why the sky is blue." Someone is replying: "The sky isn't blue because of the oxygen." You are then saying: "Well what if the people who are saying that the sky is blue because of oxygen are right." Although it gets a bit more complicated, the statement `io_uring` is insecure might be true, that's not really in dispute here. The people who are saying it, aren't saying it because they know it to be true, they are saying it because they heard about security issues in the context of `io_uring` and assumed that using `io_uring` would make your code less secure. This is incorrect, the security issues are in security features in Linux which have not been updated to handle `io_uring`. This means that your application won't be any less/more secure when using `io_uring`. But your system might be less secure if you have support for `io_uring` enabled and applications can make use of it. Moreover, the "security issues" are only undoing security related hardening you would have put in place over the baseline, they're not putting you below baseline. That's why a statement such as `io_uring` is insecure isn't very useful. If these people make the argument that: "I don't want to use `io_uring` because that would mean that security conscious system administrators would not want to run my software as a precaution." then it would make sense and nobody would be disputing it.
deleted 1y ago
[deleted]
chris_wot 1y ago
What is the best resource to actually know what io_using is and does?
- benreesman 1y ago
  https://unixism.net/loti/ https://unixism.net/loti/
yc-kraln 1y ago
I have a somewhat different problem with io_uring in practice: It's extremely hard to use /correctly/. The management of buffers which bounce across a kernel boundary and may-or-may-not end up in the same original thread lends itself to lots of subtle race conditions, resource exhaustions, and ABA issues. It's not that you can't make it work, and work well--it's that it's hard to do correctly, and very easy to make something which works 99.99% correctly, and then fails spectacularly under load or over time. I can imagine the security implications are the same.
- wtallis 1y ago
  > and may-or-may-not end up in the same original thread That sounds like a problem stemming entirely from a decision to share a ring among multiple application threads. Is there a good reason to do so? Each thread that needs to do IO can have its own ring, and submitting IO to another thread's ring seems like unnecessary complexity. The ring buffers are intended to be single-producer, single-consumer.
deleted 1y ago
[deleted]

122 points by nromiun 1y ago

johnisgood 1y ago

It is kind of like PHP. Their views of PHP is still stuck at PHP 5. We have PHP 8.

nicman23 1y ago
the work that the people in PHP did for 7 is criminally underrated
continuational 1y ago
Programming languages are different. They usually can't be "repaired" once broken, because they need to maintain backwards compatibility.
- concerned_user 1y ago
  This is only true if people stick to a version of a language and don't upgrade. If you upgrade then, for example, you can't run all of your PHP 5 code in PHP 8, most of it you can but you will have to change the parts that are broken, which are the areas that are repaired in PHP 8. Same goes for other languages like C# or Python
- pjmlp 1y ago
  They are software products like anything else.
- nromiun 1y ago
  They don't "need" to maintain backwards compatibility. Several major languages have broken it by now.
- jeroenhd 1y ago
  Programming languages deprecate features in standard libraries all the time. As PHP did, causing many PHP 5.3 applications to fail catastrophically once the warnings added to PHP 5.4, 5.5, and 5.6 were turned into errors. Of course, maintained software rarely ever runs into this issue. The standard libraries were the lacking part in PHP. The language itself was never a serious problem.
xlii 1y ago
Maybe it is (mine is for sure) but that's a "bitten by the dog once" case. You got bitten and everyone around you assures you won't get bitten again, but the pain was real and you still have a scar from the event. Why bother or invest risking another bite if there are other places to be which had never bitten before. Over my career I hated technology 3 times. First was PHP, second was Python during Python 2/3 fiasco and third was CoffeeScript. Edit: till this day one of my favorite meme is titled "PHP: Training wheels without a bike"
- yjftsjthsd-h 1y ago
  A few years ago, I used BTRFS on a laptop. Single disk, no RAID of any kind, OpenSUSE (which favored BTRFS, so I expected it to be as well supported as could be had), nothing fancy. After losing the root filesystem twice, I decided that maybe I shouldn't trust BTRFS. Since then, I've been told that it's totally better now, that all the problems are with bad RAID setups, and it's safe and won't lose my data. Anyways, as I type this from a laptop running on ZFS, I remain somewhat cautious.
  - jiggawatts 1y ago
    > Since then, I've been told that it's totally better now I've read through some bug reports, and I assure you that BTRFS remains a horror show. Saying that comments from the dev team "don't inspire confidence" is putting it mildly. ZFS is the diametric opposite of this, where blog posts from the team working on it made me realise that they're moving the state of the art into new territory.
    - johnisgood 1y ago
      I really want to try bcachefs, I wonder if anyone has used it (with encryption, and if their encryption supports plausible deniability or not). BTW, is ZFS on Linux good now?
      - chasil 1y ago
        Bcachfs has been removed from Linux. https://forums.truenas.com/t/bcachefs-dropped-from-linux-6-17/47287 https://forums.truenas.com/t/bcachefs-dropped-from-linux-6-1... ZFS is an install option on Ubuntu. Running it taints the kernel.
Xenoamorphous 1y ago
It’s true of most languages that have been around for a while. E.g. people who don’t write Javascript think of it as it was pre-ES6, not to mention that many people don’t write JS directly and use Typescript instead. Same with Java, it’s somehow stuck in time in Java 6 in some people’s minds.
WJW 1y ago
It's with so many things! Some evergreens just from last week here on HN: - Rails is horribly slow! - Python is still stuck migrating to Python 3! - MySQL doesn't scale! - Haskell tooling sucks! - io_uring is insecure! - ... and dozens more just in the programming world. These are just off the top of my head. Probably hundreds more in the wider engineering world.
- dijit 1y ago
  MySQL would happily clobber your data silently for decades though. I think the situation is better now, but that was worthy criticism for a long time; like anything though, those that bought into it wholeheartedly couldn’t take the criticisms. I wonder if there’s a word for that, seems to be a common issue.
  - als0 1y ago
    > those that bought into it wholeheartedly couldn’t take the criticisms. "invested"
- hyghjiyhu 1y ago
  From my perspective python3 migration is very much complete. Package managers is the evergreen mess now.
nurettin 1y ago
Wow. My view is stuck at php 4. Didn't even know 5 existed. I only remember Facebook making a big deal of hhvm and php 7 and laravel being considered great but not as great as rails or django.
- johnisgood 1y ago
  It is worth checking out PHP for when you are building backend. I did not use Laravel because I had some issues with it (lack of flexibility in some cases) so I ended up creating my stuff from scratch. I am glad I did. It is well documented, modular, maintainable, not bloated, etc. I can give you the resources I have heavily used.
  - nurettin 1y ago
    You're very kind. Thanks for the recommendation. If for some reason I have to branch out, I will check php again.
    - johnisgood 1y ago
      Hey, I have not recommended anything yet (besides PHP). :D I like "PHP & MySQL: Novice to Ninja" quite a lot. I have used it more so than the other books, so I definitely recommend it. You can find the code archive (code from the book) here: https://github.com/spbooks/phpmysql7 https://github.com/spbooks/phpmysql7. FWIW I recommend these books (in no particular order): ├── Modern PHP Security (2020).pdf ├── PHP 8 - OOP.pdf ├── PHP 8 - Quick Scripting Reference (2021).pdf ├── PHP 8 Programming Tips, Tricks, and Best Practices (2021).pdf ├── PHP MySQL - Novice to Ninja.pdf ├── PHP and MySQL for Advanced Learning.pdf ├── PHP in Action (2007).pdf ├── Programming PHP (2020).pdf └── The Art of Modern PHP 8.pdf
dolmen 1y ago
It is kind of like Perl. Their views of Perl is still stuck at CGIs of the 90's. I still think that Perl was and is still a better language than PHP and Python. And people never had a serious look at raku because of the Perl heritage. But that ship has sailed.
- em-bee 1y ago
  huh? that would suggest that perl has changed so much since the 90s that those views would no longer be valid. CGIs in the 90s were written in perl 5, and perl hasn't evolved much since then. but perl was never a bad language. it had/has a quirky syntax. if you were coming from a lot of commandline work using sed, awk, tr, and many other tools to manipulate text in shell scripts, then perl made a lot of sense. and while raku may have been a great improvement, perls reputation was tied to its syntax. the readability and writeability of the code. and that's the one thing that raku did not change significantly. so unlike other examples i don't see how peoples views of perl could improve
  - johnisgood 1y ago
    I love Perl. Is Raku worth trying considering I love Perl for its performance, syntax, brevity, and so forth? Raku feels like a different language to me, and it seems not-so-serious, maybe because of the name and their mascot (which is unfair to the language).
    - em-bee 1y ago
      i haven't looked at it in detail, but if you like perl and you are looking for an interesting new language that has the spirit of perl, i'd say it is worth a try
  - topspin 1y ago
    > and perl hasn't evolved much since then Perl (5) has been evolving, quietly. The language has had many nice additions over the years. Some of it is backporting from 6.

frevib 1y ago

If you like to play around with io_uring networking, here is a very simple echo server example: https://github.com/frevib/io_uring-echo-server https://github.com/frevib/io_uring-echo-server

deleted 1y ago
[deleted]

vjerancrnjak 1y ago

I think it's the Wikipedia article. https://en.wikipedia.org/wiki/Io_uring https://en.wikipedia.org/wiki/Io_uring Very easy to just quote that without any io_uring experience. > In June 2023, Google's security team reported that 60% of the exploits submitted to their bug bounty program in 2022 were exploits of the Linux kernel's io_uring vulnerabilities. As a result, io_uring was disabled for apps in Android, and disabled entirely in ChromeOS as well as Google servers. Docker also consequently disabled io_uring from their default seccomp profile.

yjftsjthsd-h 1y ago
I like how someone helpfully added > Although initial async offload design in io_uring could be problematic, later kernels changed the thread model. After such improvements, there were no known inherent problems with it and its development is very careful with new features. Considering that a performant async framework with a user facing API is complex, it was to be expected that issues would be found initially. After initial issues have been addressed, it is not any less secure than anything else in the kernel and io_uring acceptance quickly grew in production. Some of its criticism are also based on wrong or outdated assumptions.[14] ...but the only citation is a link to this GH thread, which doesn't support the claims made.
znpy 1y ago
Jens Axboe replies on the very first line of the thread: > As I'm sure you know, this is all mostly centered around a) google using an old kernel on android
- fulafel 1y ago
  But also > My hope is that this reputation will go away eventually, as less issues are found in the code. this has not yet happened like this other comment shows: https://news.ycombinator.com/item?id=44632639 https://news.ycombinator.com/item?id=44632639
flomo 1y ago
Without going into the weeds, there has be some vendor support, and that vendor is obviously not google. How to convince people: Get it into RHEL.

fabian2k 1y ago

I admit, I was confused a bit as well about the io_ring security reputation. Though I didn't really follow the topic, so the clarification that this was mostly about an older design on Android is quite helpful. The potential performance benefits are quite compelling, e.g. in Postgres 18 you reportedly can get a 3x speedup over the old sync behaviour in simple read queries.

yjftsjthsd-h 1y ago

> How to handle people dismissing io_uring as insecure? It is, in the general case, hard to prove something secure (because it's hard to prove a negative). It might help to show CVEs per month/year/whatever related to it vs anything else, preferably with a clear downward trend. For example, you could look at https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring ... although I struggle to read that as supporting the case you want to make. > I have had to deal with handful of these people from different sectors as well. Since I am actively working on project based on io_uring, I have had people saying all kinds of hmm... "crap", its so baseless! Can't even talk to them with actual facts. So what are those facts? Because all this thread has is people handwaving that it used to have a worse design, and everything has bugs and this isn't different, and implying that it's better. If it's better, show that.

yorwba 1y ago
> you could look at https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring CVE count by year: 2019: 1 2020: 1 2021: 10 2022: 15 2023: 19 2024: 21 2025: 10
- franciscop 1y ago
  I expect CVEs to be directly proportional to project usage and popularity, and inversely proportional to maturity, which makes things a lot more complicated.
  - delusional 1y ago
    And also directly proportional to the publicity of the CVE system. If you're creative enough in your writing, any bug in any program can be filed as a CVE, and filing CVEs is much more interesting carreer wise than filing bug reports. Any decently sized project has probably seen an increase in reported CVEs over the past 5 years, simply because the number of CVEs total has grown.
  - sigmoid10 1y ago

eviks 1y ago

> Perhaps if it had been written in Rust we would've been better off ;-). So the answer is known :)

ibotty 1y ago

One problem is that you can't filter its "syscalls" as you can regular syscalls. This removes a security boundary that e.g. container runtimes regularly use. So you cannot use it in your regular kubernetes cluster without weakening its security for these pods.

altairprime 1y ago
Is that a true limitation that cannot be overcome? Are solutions possible and/or available, but require further work to be shipped?
spwa4 1y ago
What regular filter for syscalls do you use?
- lima 1y ago
  seccomp-bpf, for instance.
- holowoodman 1y ago
  seccomp BPF, eBPF, in a way SELinux/AppArmor/Tomoyo/..., maybe you can even call namespaces some kind of syscall filter. And then there is the auditing framework, where you can at least record which critical syscalls were performed. Nowadays its mostly a combination of eBPF, SELinux and auditd plus namespaces in case of containers. Usually in the combination that some distro ships, so nothing really fancy.
holowoodman 1y ago
This just reinforces the (maybe unfounded) impression that security is a secondary consideration, and performance is primary. I'd use io_uring in a heartbeat on a dedicated system where the job is only I/O and security isolation isn't a concern. But multiuser/multiapplication/networked? Not a chance.
- weitendorf 1y ago
  I think there is a very large amount of overlap between the people who 1. know what io_uring is 2. are interested in performance enough to look at improvements based on new linux kernel system calls and talk about it in public 3. care about security in multitenant environments or the syscalls used by third party libraries I think io_uring right now probably makes a lot of sense for HPC and highly technical, performance-sensitive financial stuff, but they can be kind of insular. I don't think most linux hobbyists really need the performance benefits enough t care about it, and most businesses are using a major cloud vendor/don't have the scale or expertise to be thinking about this kind of stuff. Which leaves major cloud providers and really big businesses like Meta with their own internal clouds as the ones that stand to benefit enough to care about performance while really caring about security

perks_12 1y ago

Works on the most sophisticated io_uring project, has no idea how to defend it. Nice.

topspin 1y ago
It's tragic, and it bothers me that Jens Axboe's work is suffering due to this. Obviously, with the clarity of hindsight, this might have been avoided. Now the cost of the damage it high. My idea: consider formal verification. A rigorous formal proof of behavior is capable of solving actual flaws and probably capable of overcoming the bad PR. That would require a great and ongoing effort. However, given the nature of io_uring, it's likely rather amenable to formal verification, and ultimately it is probably necessary. Perhaps our new "AI" tools could greatly reduce the effort. Anyhow, that's my brilliant thought...
- topspin 1y ago
  Replying to myself. io_uring is only a more ambitious manifestation of a pattern that has been commonplace for a long time. High performance network, storage and other hardware use circular buffers internally in their drivers to process commands and return results. io_uring is attempting to realize the same pattern, but for the entire operating system interface. That's a really big idea. It has me asking whether this isn't how all "IPC" should be done, where one generalizes all of the running processes, including the "kernel." Ultimately it has significant implications for CPU architecture, supplanting traditional patterns. When I wrote "given the nature of io_uring," I had something specific in mind wrt formal verification. io_uring fundamentally consumes a stream of bytes. At the front end of that is a great deal of validation code to ensure the steam contents are sane. At the backend, the commands are mapped to existing kernel functionality. It's the front end I think could be well served by formal verification.

weinzierl 1y ago

Convince AWS to allow it in managed container and serverless runtimes. When people see what Amazon does it is far too easy to conclude: "Huh, if Amazon blocks it there must me something to it."

txdv 1y ago

One of the most interesting aspects of software development is that it is still done by humans. Information moves slowly and is very generalised, with little attention to detail. I have seen this multiple times when developers were still reciting old benchmarks, taken out of context. It often becomes very tribal and centred around technologies.

serial_dev 1y ago

> How can I help people out when they tell me that io_uring is insecure? Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong. If you start there, you will fail to convince people of anything, because you automatically dismiss their claim, without thinking about what they might have seen and what they might think. A better starting point would be wanting to get to the bottom of it, and assess the security of io_uring. If you start from that point and you give it an honest, thorough assessment, and it turns out it "looks secure", you'll have an easier time convincing people. You might still be wrong (assessing io_uring's security is not trivial), but at least you tried to understand why people think that. And reminder: it's ok to "agree to disagree".

VWWHFSfQ 1y ago
> Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong. I think this is the right approach. We know that io_uring has a somewhat significant history of critical security issues. It's not enough just to point out that "these 3 critical CVEs were fixed in the last 12 months, it's secure now!" Reputation and trust has to be built over a long period of time.
jay-barronville 1y ago
> Maybe those people are right, though? I think the discussion starts from a place that assumes other people are wrong. If you start there, you will fail to convince people of anything, because you automatically dismiss their claim, without thinking about what they might have seen and what they might think. Bingo. This is the correct approach. Very well said!
Arch-TK 1y ago
People are saying: "Oxygen is blue and that's why the sky is blue." Someone is replying: "The sky isn't blue because of the oxygen." You are then saying: "Well what if the people who are saying that the sky is blue because of oxygen are right." Although it gets a bit more complicated, the statement `io_uring` is insecure might be true, that's not really in dispute here. The people who are saying it, aren't saying it because they know it to be true, they are saying it because they heard about security issues in the context of `io_uring` and assumed that using `io_uring` would make your code less secure. This is incorrect, the security issues are in security features in Linux which have not been updated to handle `io_uring`. This means that your application won't be any less/more secure when using `io_uring`. But your system might be less secure if you have support for `io_uring` enabled and applications can make use of it. Moreover, the "security issues" are only undoing security related hardening you would have put in place over the baseline, they're not putting you below baseline. That's why a statement such as `io_uring` is insecure isn't very useful. If these people make the argument that: "I don't want to use `io_uring` because that would mean that security conscious system administrators would not want to run my software as a precaution." then it would make sense and nobody would be disputing it.

deleted 1y ago

[deleted]

chris_wot 1y ago

What is the best resource to actually know what io_using is and does?

benreesman 1y ago
https://unixism.net/loti/ https://unixism.net/loti/

yc-kraln 1y ago

I have a somewhat different problem with io_uring in practice: It's extremely hard to use /correctly/. The management of buffers which bounce across a kernel boundary and may-or-may-not end up in the same original thread lends itself to lots of subtle race conditions, resource exhaustions, and ABA issues. It's not that you can't make it work, and work well--it's that it's hard to do correctly, and very easy to make something which works 99.99% correctly, and then fails spectacularly under load or over time. I can imagine the security implications are the same.

wtallis 1y ago
> and may-or-may-not end up in the same original thread That sounds like a problem stemming entirely from a decision to share a ring among multiple application threads. Is there a good reason to do so? Each thread that needs to do IO can have its own ring, and submitting IO to another thread's ring seems like unnecessary complexity. The ring buffers are intended to be single-producer, single-consumer.

deleted 1y ago

[deleted]