20 ms·

How io_uring and eBPF Will Revolutionize Programming in Linux

• nynx 6y ago
  wasm in the kernel when?
  • layoutIfNeeded 6y ago
    Hopefully never
  • Technically 6y ago
    ....wtf why would you do this?
  • Tuna-Fish 6y ago
    Never. eBPF is a good interface for running code in the kernel because of how restricted it is. Before loading a eBPF module, the kernel does a lof of verification on it, including proving that it terminates. This is famously rather hard on Turing-complete languages. (Although in this case could be solved by giving programs a fixed time budget per invocation.)
    • emn13 6y ago
      Guaranteed termination isn't the only analysis worth having, especially since if your aim is termination (as opposed to leveraging that simplicity for other analysis) it's much easier to simply impose termination and be done with it. So, WASM isn't entirely unconstrained, and those constraints do allow some interesting analyses that true native code cannot support, e.g. symbolic execution: https://blog.trailofbits.com/2020/01/31/symbolically-executing-webassembly-in-manticore/ https://blog.trailofbits.com/2020/01/31/symbolically-executi... By it's very design it's aimed at running untrusted, even malicious code without exposing the host to security risks, while allowing the imposition of resource constraints. It doesn't sound completely absurd to me (not an expert in this) that some project might seek to use those safety guarantees as a way to avoid paying for the more heavyweight guarantees provided by the kernelspace/userspace split. A quick google find lots of people trying to get this to work; who knows - it might pay off.
  • rhencke 6y ago
    It was done in 2017: https://github.com/cervus-v/cervus https://github.com/cervus-v/cervus
    • ithkuil 6y ago
      Pity it's based on commonwa instead of wasi
      • rhencke 6y ago
        Perhaps you would enjoy the sequel, then: https://github.com/wasmerio/wasmer https://github.com/wasmerio/wasmer
  • pjmlp 6y ago
    eBPF is already it, apparently people keep forgeting that WASM isn't the first nor is going to be the last polyglot bytecode format.
    • CyberDildonics 6y ago
      I haven't seen anyone make that claim. WASM's value has always been in its pragmatic integration plan using what is already there.
      • pjmlp 6y ago
        Then you haven't been paying attention to the "rewrite in WASM" advocacy force and how WASM "has invented" polyglot bytecode.
        • CyberDildonics 6y ago
          I've been paying pretty close attention and read a lot of articles and discussions, starting when it was being created and I have never seen anything about either of those. No one actually writes WASM directly, so no one says to "rewrite in WASM". It is only an optimization, so the two uses are to speed up javascript and to compile native languages. No one even says "polygot bytecode", let alone says that WASM 'invented it'. Most programmers have already used something that uses a bytecode format, why would anyone say that?
• YarickR2 6y ago
  No it will not. Two rather specialized tools to help with rather specific issues are no reason to throw out heaps and mounds of existing and perfectly working code and solutions
  • kevincox 6y ago
    There are tons of applications that are built upon higher-level APIs that can be updated to take advantage of this automatically. Furthermore you don't need to have every single application updated to revolutionize something. The apps that need this will be quite happy to make the jump.
  • harporoeder 6y ago
    I expect these technologies to be integrated into language runtimes and webservers such that most developers won't even know it being used similar to EPOLL today. While revolutionary is an extreme characterization the performance and improved API changes are at least non trivial.
  • dclusin 6y ago
    There's a very niche conference called eBPF Summit* that has presentations from people at companies with first class engineering orgs talking about what they are doing with eBPF. The problems they are solving and the breadth of problems being solved are very impressive. [*] - https://ebpf.io/summit-2020/ https://ebpf.io/summit-2020/
    • jcelerier 6y ago
      yep, at the Linux Audio Conference we have a paper for using eBPF to process network audio packets this year for instance (https://lac2020.sciencesconf.org/program https://lac2020.sciencesconf.org/program / paper is https://lac2020.sciencesconf.org/307835 https://lac2020.sciencesconf.org/307835) - presentation is tomorrow (happens online)
    • smartbit 6y ago
      This conference was more a Cilium conference. Google will use cilium as the default cni on gke https://news.ycombinator.com/item?id=24212021 https://news.ycombinator.com/item?id=24212021 Not speaking was Kinvolk’s CTO Alban Crequy https://news.ycombinator.com/item?id=23042722 https://news.ycombinator.com/item?id=23042722
• rajnathani 6y ago
  Previous discussion: https://news.ycombinator.com/item?id=22974728 https://news.ycombinator.com/item?id=22974728
• mhh__ 6y ago
  eBPF is still a bit rough but it's already very cool what you can do already. It would be nice to see it at a high-level at the syscall interface i.e. currently if I want to attach a probe I have to find the function myself or use a library but it would he nice to have it understand elf files.
• The_rationalist 6y ago
  Project loom is gonna make Java threads automatically use io_uring and restartable sequences. Btw netty is currently actively working on io_uring support, which could enable truly non blocking sockets, which could enable truly asynchronous connection to postrgresql through JDBC, which would enable state of the art Spring performance on techempowerup benchmarks
  • vbezhenar 6y ago
    What do you mean "truly non blocking sockets"? There's API for non-blocking network sockets in Java since forever. You just have to explicitly use it. I think you mean using non-blocking sockets even for code which explicitly uses blocking sockets.
• alexgartrell 6y ago
  I wouldn't be doing my job if I failed to mention that both Alexei (eBPF) and Jens (io_uring, block) work at Facebook. Beyond them, we've got a bunch of folks working on the primitives as well as low-level userspace libraries [0] that enable us to use all of this stuff in production, so, by the time you're seeing it, we've demonstrated that it works well for all of Facebook's load balancers, container systems, etc. [0] https://github.com/facebook/folly/blob/16d6394130b0961f6d688fd4cca27fca25fbca93/folly/experimental/io/IoUring.h https://github.com/facebook/folly/blob/16d6394130b0961f6d688...
• deleted 6y ago
  [deleted]
• qchris 6y ago
  I'm genuinely curious; both of these changes seem to be exciting due to the ability for people to extend and implement specialized code/features using the kernel. Since the Linux kernel is GPLed (v2, I believe?), does this mean that the number of GPL requests related to products' operating systems is likely to increase, since groups using this extensibility will be writing code covered by the GPL which might actually be of value to other people? Or does the way io_uring and eBPF are implemented isolate the code in such a way that the extensions through their frameworks such that the GPL license won't affect them?
  • rtkaratekid 6y ago
    I don’t know about io_uring, but for BPF programs only the kernel space needs to be licensed as GPLv2. Everything on the user space side is handled with system calls or higher level libraries that aren’t GPL licensed (libbpf).
  • marcan_42 6y ago
    io_uring is a data structure, not code. It's not Turing complete, so there is absolutely no way it would extend GPL virality from the kernel into userspace. eBPF is code, and follows similar rules to kernel modules. That is, non-GPL-compatible eBPF code is allowed, but a subset of APIs (helpers, like module symbols) are only available to GPL-compatible eBPF programs.
    • Conan_Kudo 6y ago
      I wish people would stop trying to work around the GPL. It causes immense heartache and someday, the kernel developers will revolt and destroy any avenue that you try to use. After all, that's what just happened with the NVIDIA GPU drivers with Linux 5.9. Also, you're still pulling in parts of Linux into your code, so GPLv2 still applies.
      • tyingq 6y ago
        I don't see how that would apply to an application using io_uring through syscalls. It's not linking to any GPLv2 code, and the header file (io_uring.h) is dual licensed as GPLv2 and MIT. Similarly, if you choose to use the higher level liburing, it's dual LGPL and MIT licensed. It's all very deliberately not viral for user-space applications.
• merqurio 6y ago
  Also from Glauber Costa, a thread-per-core framework using io_uring written in Rust[1] and discussed in HN[2]. [1]: https://github.com/DataDog/glommio https://github.com/DataDog/glommio [2]: https://news.ycombinator.com/item?id=24976533 https://news.ycombinator.com/item?id=24976533
• hawk_ 6y ago
  One thing that I haven't been able to get is if this makes things like DPDK or user mode tcp stack unnecessary since the system call overhead is gone.
  • dathinab 6y ago
    io_uring reduces but doesn't remove the system call overhead. Only with in kernel polling mode is it close to removed. But kernel polling mode has it's own cost. If the system call overhead is no where close to being a bottle neck, i.e. you don't do system calls "that" much, e.g. because your endpoints take longer to complete then using kernel polling mode can degrade the overall system performance. And potential increase power consumption and as such heat generation. Besides that user mode tcp stacks can be more tailored for your use case which can increase performance. So all in all I would say that it depends on your use case. For some it will make user mode tcp useless or at least not worth it but for others it doesn't.
    • hawk_ 6y ago
      I am not sure how user stack tcp or DPDK would get around the power consumption issues of kernel polling. Infact the usage I am aware of pretty much involve polling in user mode because any context switch or O/S scheduling related overhead is excessive. The only thing you can do is to keep your task queue full so as to always be doing something.
    • anarazel 6y ago
      io_uring does allow to remove a lot of the syscall overhead, without polling. Many operations can be submitted with just one syscall. And ready completions can be consumed without a syscall at all. Additionally, compared to using epoll/select/.. for network IO, one can just submit a send/recv, instead of patterns like recv -> EAGAIN, epoll, recv
      • deleted 6y ago
        [deleted]
      • gpderetta 6y ago
• cb321 6y ago
  The batch system call part is not so hard on its own: https://github.com/c-blake/batch https://github.com/c-blake/batch
  • ithkuil 6y ago
    io_uring interface is designed to allow to encode arbitrary syscalls, although it seems that only a handful are actually supported
    • pengaru 6y ago
      It's under active development, more syscalls keep getting added. Contributions are welcome.
      • ithkuil 6y ago
        I'd love a TL;DR explaination for why a blanket interface is not possible. I can guess that there are different ways that syscalls handle parameters and there are different families of syscall behaviours, but I'm not sure if that's the reason. I'd love a quick intro and pointers to more details.
        • cb321 6y ago
          You can look at my code in the link I gave. It's pretty short and should work on kernels 3.x series to 5.x series. { EDIT: I realize this may make you ask your question even more strongly. :-) }
        • pengaru 6y ago
          While I won't be writing a TL;DR summary, this isn't exactly unexplored territory, and it's been documented by lwn: https://lwn.net/Articles/316806/ https://lwn.net/Articles/316806/ https://lwn.net/Articles/221887/ https://lwn.net/Articles/221887/ https://lwn.net/Articles/219954/ https://lwn.net/Articles/219954/
          • 6y ago
• shmerl 6y ago
  Are I/O libraries like Tokio for Rust using io_uring?
  • dathinab 6y ago
    Some are other will be other won't. One problem with io_uring is that it's completion based I/O where you move ownership of an buffer to the kernel which then writes to it until the operation completes. This means you might not be able to (sync) cancel an operation occurring in the background. This makes it harder to integrate into some I/O libraries, as the previous fact conflicts with RAII patterns. Another think making adaption harder is that the interfaces for reading/writing with io_uring are conceptually slightly different. Because of this e.g. Tokio currently hasn't switched yet to use io_uring but still uses readiness based async I/O as far as I know. (Which doesn't mean it won't support it in the future.) This issue might be relevant (mio is internally used by tokio for async I/O): https://github.com/tokio-rs/mio/issues/923 https://github.com/tokio-rs/mio/issues/923
  • fwip 6y ago
    Looks like it's on the roadmap: https://github.com/tokio-rs/tokio/issues/2411 https://github.com/tokio-rs/tokio/issues/2411
  • Arnavion 6y ago
    Apart from tokio (which the sibling comment covers), you can use rio [1] which specifically uses io_uring, and can be used as a Future [2] so that you can use it as part of the wider Future ecosystem, with tokio or any other executor. [1]: https://crates.io/crates/rio https://crates.io/crates/rio [2]: https://docs.rs/rio/0.9.4/rio/struct.Completion.html#impl-Future https://docs.rs/rio/0.9.4/rio/struct.Completion.html#impl-Fu...
  • coolreader18 6y ago
    There's the ringbahn[0] project, which is a couple of layers wrapping the liburing library; it seems like it's presenting itself as an orthogonal runtime to tokio. Meanwhile, looks like tokio is looking into the feasibility of using io_uring[1], though I'm not sure if it would be using ringbahn or not. [0]: https://github.com/ringbahn https://github.com/ringbahn [1]: https://github.com/tokio-rs/tokio/issues/2411 https://github.com/tokio-rs/tokio/issues/2411
• b0rsuk 6y ago
  How does it make Linux compare to Windows, OSX and *BSD?
  • Arnavion 6y ago
    The underlying principle of submitting a job to be done and letting the kernel do it (submitting a read() and waiting for it to complete), as opposed to the original model of waiting for the kernel when it's ready to let you do the job (waiting for an fd to become readable so you can call read() on it) is the same as the completion-based model of Windows async I/O (and I think BSD's kqueue too?) as opposed to the readiness-based model of epoll. The part of where this submission and completion information involves a ring buffer mapped to kernel space is unique to Linux, I believe.
    • Const-me 6y ago
      > The part of where this submission and completion information involves a ring buffer mapped to kernel space is unique to Linux RIO is very similar and predates Linux version by many years: https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/hh997032(v=ws.11) https://docs.microsoft.com/en-us/previous-versions/windows/i... Main downside, that thing is only for sockets.
      • muststopmyths 6y ago
        And plain old overlapped I/O can lock pages in memory. With Direct I/O devices can directly DMA into these buffers, so theoretically any device can provide similar functionality and use completion ports for notification. (or polling, since the possible high "interrupt rate" bottleneck of completion notifications is one of the things that motivated RIO)
        • Const-me 6y ago
          Mapping pages is relatively expensive. Not the mapping itself, but as a consequence of such update CPU has to flush at least a portion of TLB cache. With overlapped I/O the kernel has to do that for every I/O request. With RIO and now io_uring, kernels map buffers to both kernel and user addresses spaces just once on initial setup, and reuse the same buffer for many I/O operations.
• superkuh 6y ago
  I suppose this will help for the big corporate users of linux. And I suppose that's where most of the programming gets done for linux. But the rate of change and feature adoption by the big commercial pushers of linux has made linux as a desktop more troublesome due to the constant futureshock.
  • yxhuvud 6y ago
    Io_uring will be big under the hood for frameworks and/or programming languages too. But yes, adoption will take a while..
• adzm 6y ago
  This feels very very similar to IO completion ports / iocp on Windows. More modern versions of Windows even has registered buffers for completion which can be even more performant in certain scenarios. I'm looking forward to trying this out on Linux. I'm curious to see how this might work its way into libuv and c++ ASIO libraries, too.
  • leetrout 6y ago
    libuv is tracking in https://github.com/libuv/libuv/issues/1947 https://github.com/libuv/libuv/issues/1947
    • cycloptic 6y ago
      Tracking that issue was a motivator for me to begin adding support to glib: https://gitlab.gnome.org/GNOME/glib/-/issues/2084 https://gitlab.gnome.org/GNOME/glib/-/issues/2084
  • ithkuil 6y ago
    io_uring allows the kernel and the user program to communicate purely via shared memory without having to perform a system call, i.e. a context switch to the kernel. Do windows completion ports also work that way or do they involve a system call to be performed in order to consume completion events?
    • ta8645 6y ago
      You do still need a single system call (io_uring_submit) to submit each batch of entries in the submission queue. Edit: actually no it's not required in all cases. Thanks for the correction.
      • WJW 6y ago
        You can ask the kernel to poll the submission queue and skip io_uring_submit too. (Though you need elevated privileges to do this IIRC)
      • 6y ago
• glutamate 6y ago
  How do you feel about spreading genocidal hatred and undermining democracy? https://www.nytimes.com/2020/09/18/opinion/facebook-democracy.html https://www.nytimes.com/2020/09/18/opinion/facebook-democrac... https://www.vice.com/en/article/xg897a/hate-speech-on-facebook-is-pushing-ethiopia-dangerously-close-to-a-genocide https://www.vice.com/en/article/xg897a/hate-speech-on-facebo... https://www.bbc.co.uk/news/world-asia-46105934 https://www.bbc.co.uk/news/world-asia-46105934
  • dang 6y ago
    Personal attacks and flamebait are not allowed here. Please read https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and stick to the rules. We detached this comment from https://news.ycombinator.com/item?id=25222895 https://news.ycombinator.com/item?id=25222895.
• tus88 6y ago
  BSD folks are yawning right now.
  • ta8645 6y ago
    Why?
    • cyphar 6y ago
      kqueue on FreeBSD is effectively like io_uring but has existed for much longer (and Windows I/O Completion Ports predate kqueue). kqueue also gives you a way to get system events which you can't get on Linux (it has an equivalent of cn_proc that isn't awful).
      • yxhuvud 6y ago
        I would argue the api of kqueue is clunkier and less general. Io_uring can keep track of events through using event_fds, but yes, that is perhaps not optimal.
        • cyphar 6y ago
          Yes, sorry I muddled things up (and now the edit window has elapsed). io_uring isn't the same as kqueue because it doesn't permit asynchronous notification of job completion nor can you do more complicated chained operations, epoll is the Linux equivalent of kqueue (and in that comparison, kqueue is better).
      • josephg 6y ago
        Huh? Kqueue is nice, but I don't think you'll get anywhere near the performance of io_uring out of kqueue: - kqueue only tells you there's data to read (/space in the write buffer). You still need to call read() or write(), including paying the cost of the syscall. io_uring lets you batch a lot of read/write calls together and either issue a single syscall to the kernel for all calls, or have the kernel poll and never syscall at all. - kqueue doesn't let you issue fsync, or any of the other syscalls now in io_uring. fsync is essential on the write path for correctness in lots of cases, and for that you still need to dispatch to a local thread pool or something. So yeah, I prefer kqueue over linux's epoll. But io_uring seems like the new king.
• CalChris 6y ago
  This reminds me of David Wheeler's adage: All problems in computer science can be solved by another level of indirection. The rejoinder, and I don't know who gets credit for it, is: All performance problems can be solved by removing a layer of indirection.
  • sigjuice 6y ago
    An often cited corollary to the first one is, "...except for the problem of too many layers of indirection." :) https://en.wikipedia.org/wiki/Indirection https://en.wikipedia.org/wiki/Indirection
    • FeepingCreature 6y ago
      Incorrect: that problem, too, can be solved by adding another layer of indirection - then adding an optimized implementation underneath it.
  • harry8 6y ago
    Have we stopped solving all performance problems with introducing a cache? Why wasn't I told? Will I have to hand in my union card?
    • infogulch 6y ago
      No worries, your union card is safe: a cache is just a particular kind of added indirection. :)
      • CalChris 6y ago
        And some architectures have magically improved performance by reconfiguring caches as scratch pad memories. http://rexcomputing.com/ http://rexcomputing.com/
• zests 6y ago
  I am impressed with the level of linux knowledge in this thread. How do people become linux kernel hackers? Most of the developers I know (including myself) use linux but have very little awareness beyond application level programming.
  • the8472 6y ago
    You don't necessarily have to be a kernel hacker to be familiar with many of the features that the kernel provides. Just doing application debugging often requires to dig deeper until you hit some kernel balrogs. Container problems? Namespaces, Cgroups, ... Network problems? Netfilter, tc, lots of sysctl knobs, tcp algorithms (cue 1287947th thread on nagle/delayed acks/cork) Slow disk IO? Now you need to read up on syscalls and maybe find more efficient uses. Copy_file_range doesn't work as expected? Suddenly you're reading kernel release notes or source code.
  • marcosdumay 6y ago
    > How do people become linux kernel hackers? Honestly, by hacking it. There's a famous book about Linux internals that I don't remember the name (but has "Linux" and "internals" on it). But I have never seen anybody doing it by reading a book (despite how excellent it can be). You just go change what you want or read the submodule you are interested in understanding, and use the book, site or whatever when you have a problem.
    • bigyikes 6y ago
      I have never run into a problem that I thought needed to be solved in the kernel. What kinds of things have you wanted to change?
      • the_only_law 6y ago
        Conversely every problem I’ve run into in a kernel that I could potentially solve requires me to be an expert in 1-3 things outside of the kernel.
      • marcosdumay 6y ago
• kzrdude 6y ago
  The author of the black swan book explained that the covid pandemic was not what he meant by a black swan event. Because it was not something entirely unpredictable.. if we look back, we have been talking about pandemics for decades.
  • makapuf 6y ago
    Maybe, but I fail to see how this relates to Linux kernel (wrong thread, I suppose)
    • kzrdude 6y ago
      Fourth paragraph of the article brings it up.
  • throw0101a 6y ago
    > Because it was not something entirely unpredictable.. To the point that as part of the Obama-Trump transition, a literal playbook was created for pandemics, with Coronaviruses (MERS-COV, SARS) explicitly mentioned: * https://assets.documentcloud.org/documents/6819268/Pandemic-Playbook.pdf https://assets.documentcloud.org/documents/6819268/Pandemic-... They had tabletop exercise on pandemics: * https://www.politico.com/news/2020/03/16/trump-inauguration-warning-scenario-pandemic-132797 https://www.politico.com/news/2020/03/16/trump-inauguration-...
    • rbanffy 6y ago
      That someone would completely disband the team and then ignore the knowledge acquired was kind of unpredictable. I mean, what kind of person would do that?!
      • throw0101a 6y ago
        People who see government not as a way to (hopefully) better society through good implementations of good policies, but rather see government as source of power to wield for their own benefit. > Friends of the government win state contracts at high prices and borrow on easy terms from the central bank. Those on the inside grow rich by favoritism; those on the outside suffer from the general deterioration of the economy. As one shrewd observer told me on a recent visit [to Hungary], “The benefit of controlling a modern state is less the power to persecute the innocent, more the power to protect the guilty.” * https://www.theatlantic.com/magazine/archive/2017/03/how-to-build-an-autocracy/513872/ https://www.theatlantic.com/magazine/archive/2017/03/how-to-... * https://archive.is/ZIzCm https://archive.is/ZIzCm
• mwcampbell 6y ago
  > Things will never be the same again after the dust settles. And yes, I’m talking about Linux. One has to be in quite a techie bubble to equate Linux kernel features with actual world-changing events, as the author goes on to do. More on-topic though, having read the rest of the article, my guess is that while these features will let companies squeeze some more efficiency out of high-end servers, they won't change how most of us develop applications.
  • the8472 6y ago
    Any async or event-loop runtime can be almost entirely powered by io_uring. Timers, waiting for work when you're out of CPU-bound tasks, most IO syscalls, it all can go through io_uring. You'll still need a few worker threads for blocking syscalls that haven't been ported to io_uring yet but that need is greatly reduced compared to the previous state of things. So even if you're not using io_uring yourself the language standard libraries or server frameworks will. There are WIPs for netty, libuv, nginx. Other projects are exploring it or have announced intent to use it.
    • jorangreef 6y ago
      Another example, Zig landed io_uring in the std lib a month ago: https://github.com/ziglang/zig/pull/6356 https://github.com/ziglang/zig/pull/6356 I'm also really excited by how you can use io_uring to power everything (fs, networking etc.) with one easy api and a single-threaded event loop: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_uring#what-this-means-for-the-future-of-event-loops https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u... io_uring makes thread-per-core designs so much easier.
  • ben509 6y ago
    He also brings up 2020 because OMG, it's worst year EVAR. It's not a tech bubble as much as it's a journo bubble. People are reading before they're writing, so he's seeing trendy topics like 2020 and the virus. He feels he needs a hook to get his readers engaged, so he's reaching for things readers can related to. It's a bad hook. I think an editor would have cut that whole intro.
• whateveracct 6y ago
  GHC RTS integration already well in the works too :) http://wjwh.eu/posts/2020-07-26-haskell-iouring-manager.html http://wjwh.eu/posts/2020-07-26-haskell-iouring-manager.html
• pengaru 6y ago
  Coincidentally last night I announced [0] a little io_uring systemd-journald tool I've been hacking on recently for fun. No ebpf component at this time, but I do wonder if ebpf could perform journal searches in the kernel side and only send the matches back to userspace. Another thing this little project brought to my attention is the need for a compatibility layer on pre-io_uring kernels. I asked on io_uring@vger [1] last night, but nobody's responded yet, does anyone here know if there's already such a thing in existence? [0] https://lists.freedesktop.org/archives/systemd-devel/2020-November/045641.html https://lists.freedesktop.org/archives/systemd-devel/2020-No... [1] https://lore.kernel.org/io-uring/20201126043016.3yb5ggpkgvuzhudw@shells.gnugeneration.com/T/#u https://lore.kernel.org/io-uring/20201126043016.3yb5ggpkgvuz...
  • cycloptic 6y ago
    I was thinking about doing this for an event loop I was working on, but no code to show yet... you probably can get away easily with using pthreads and a sparse memfd to store the buffers.
    • cycloptic 6y ago
      (Reply since I can't edit anymore) The only catch is that with that approach, you can't poll on the fd like you can with the real thing.
      • pengaru 6y ago
        Assuming you're talking about emulating io_uring in userspace at the liburing API level, couldn't you just use a pthreads thread pool for the syscall dispatching from submitted SQEs, and when those complete their results get serialized into the CQE. For fd-based monitoring of the CQE, wouldn't a simple pipe or eventfd suffice? When CQEs get added, write to the fd, it just happens to all be in-process. I must admit I haven't gone deep into the liburing internals or the low-level io_uring API, but conceptually speaking there doesn't seem to be anything happening that can't be done in-process in userspace atop pthreads for the blocking syscalls. It just won't be fast. Am I missing some critical show-stopping detail?
• PeterCorless 6y ago
  Today I am grateful for the brilliant minds around the world that continually open up fundamentally revolutionary new ways to develop applications. To Jens, to Alexei, and to Glauber, and to all of their kindred and ilk, we raise a glass!
• pierrebai 6y ago
  Ah the funny things we resad about in 2020. In 1985... yes I said 1985, the Amiga did all I/O through sending and receiving messages. You queued a message to the port of the device / disk you wanted, when the I/O was complete you received a reply on your port. The same message port system was used to receive UI messages. And filesystems, on top of drive system, were also using port/messages. So did serial devices. Everything. Simple, asynchronous by nature. As a matter of fact, it was even more elegant than this. Devices were just DLL with a message port.
  • beagle3 6y ago
    And it worked, well, with 512K memory in 1985. The multitasking was co-operative, and there was no paging or memory protection. That didn't work as well (But worked surprisingly well, especially compared to Win3.1 which came 5-6 years later and needed much more memory to be usable). I suspect if Commodore/Amiga had done a cheaper version and did not suck so badly at planning and management, we would have been much farther along on software and hardware by now. The Amiga had 4 channel 8-bit DMA stereo sound in 1985 (which with some effort could become 13-bit 2 channel DMA stereo sound), a working multitasking system, 12-bit color high resolution graphics, and more. I think the PC had these specs as "standard" only in 1993 or so, and by "standard" I mean "you could assume there was hardware to support them, but your software needed to include specific support for at least two or three different vendors, such as Creative Labs SoundBlaster and Gravis UltraSound for sound).
    • pjmlp 6y ago
      We have to "thank" Compaq for it as well. Even with all its management flaws, the Amiga might have survived without the mass production of PC clones.
    • icedchai 6y ago
      Amiga multitasking was actually preemptive. It was only cooperative in the sense that all processes / tasks were in the same address space and without memory protection...
• phatfish 6y ago
  IBM did some great work for the Germans in the 30s and 40s.
  • girvo 6y ago
    I despise Facebook, but this is a horrible take. They’re not actively in the business of genocide.
    • stone2020 6y ago
      Myanmar?
    • pizza 6y ago
      Unless you're Rohingya
    • jmcgough 6y ago
      Even if it's through inaction, they're still liable.
  • dvfjsdhgfv 6y ago
    I understand your feelings but it's not fair to blame developers for what the management is doing.
    • itodd 6y ago
      respectfully disagree.
    • deleted 6y ago
      [deleted]
    • deleted 6y ago
      [deleted]
    • munificent 6y ago
      If you choose to give your labor to a company, you tacitly approve of what the company does with the fruits of that labor.
• grahamm 6y ago
  At SCO in the mid-90s we were playing with very similar ideas to boost DB performance. The main motivation was the same then as it is now, don't block and avoid making system calls into the kernel once up and running. Don't recall if any of the work made it into product.
• helloycombinat 6y ago
  What's the fastest way to read a file thats 32 (or 64) byte aligned (not bit, but byte)? I was thinking synchronous open/stat might not be optimal. My current code opens/stat/fadvise that I'll read it all and then mmap entire files handing them off to different threads. What calls do I want? I wish the article listed the enhanced API calls but I guess i'll be hunting those this weekend
• cassepipe 6y ago
  So at line 12, it's a macro for a loop right ? Or am I missing something ? https://gist.github.com/PeterCorless/f83c09cc62ccd60e595e4eb124c1676e#file-newstack-03-io_uring-event-loop-c https://gist.github.com/PeterCorless/f83c09cc62ccd60e595e4eb...
  • layoutIfNeeded 6y ago
    #define io_uring_for_each_cqe(ring, head, cqe) \ /* \ * io_uring_smp_load_acquire() enforces the order of tail \ * and CQE reads. \ */ \ for (head = *(ring)->cq.khead; \ (cqe = (head != io_uring_smp_load_acquire((ring)->cq.ktail) ? \ &(ring)->cq.cqes[head & (*(ring)->cq.kring_mask)] : NULL)); \ head++)
• marathonian 6y ago
  Is this similar to XNU's Mach messaging?
• chris_wot 6y ago
  Given it uses a queue that has a producer and a consumer, I wonder if a monitor will be required?
  • junon 6y ago
    Can you elaborate? What do you mean by monitor?
    • gpderetta 6y ago
      I believe they are referring to the synchronization primitive [1], conceptually a condition variable plus a mutex. [1] https://en.wikipedia.org/wiki/Monitor_(synchronization) https://en.wikipedia.org/wiki/Monitor_(synchronization)
  • gpderetta 6y ago
    I think that either futexes or polling can be used for notification.
• fefe23 6y ago
  I don't think io_uring and ebpf will revolutionize programming on Linux. In fact I hope they don't. The most important aspect of a program is correctness, not speed. Writing asynchronous code is much harder to get right. Sure, I still write asynchronous code. Mostly to find out if I can. My experience has been that async code is hard to write, is larger, hard to read, hard to verify as correct and may not even be faster for many common use cases. I also wrote some kernel code, for the same reason. To find out if I could. Most programmers have this drive, I think. They want to push themselves. And sure, go for it! Just realize that you are experimenting, and you are probably in over your head. Most of us are most of the time. Someone will have to be able to fix bugs in your code when you are unavailable. Consider how hard it is to maintain other people's code even if it is just a well-formed, synchronous series of statements. Then consider how much worse it is if that code is asynchronous and maybe has subtle timing bugs, side channels and race conditions. If I haven't convinced you yet, let me try one last argument. I invite you to profile how much actual time you spend doing syscalls. Syscalls are amazingly well optimized on Linux. The overhead is practically negligible. You can do hundreds of thousands of syscalls per second, even on old hardware. You can also easily open thousands of threads. Those also scale really well on Linux.
  • skybrian 6y ago
    I don't know what kind of programming you're doing, but in network apps, if you have a thread per client and lots of clients (like a web server), you end up with lots of threads waiting on responses from slow clients, and that takes up memory. The time blocked on the syscall has nothing to do with your own machine's performance. But on the other hand, if your server is behind a buffering proxy so it's not streaming directly over the Internet, it might not be a problem.
    • mwcampbell 6y ago
      > But on the other hand, if your server is behind a buffering proxy so it's not streaming directly over the Internet, it might not be a problem. This is one instance of a larger pattern I've been noticing. When using some languages (like Python and Ruby) in the natural, blocking way, a back-end web application typically needs multiple processes per machine, because it doesn't handle many concurrent requests per process. Combine this with the fact that each thread has to block while waiting on the client, and you have to add more complexity around the application server processes to regain efficiency. The proxy in front of those servers is one example. Another is an external database connection pool like PgBouncer. Speaking of the database, to avoid wasting memory while waiting on it, you may end up introducing caching sooner than you otherwise would. And when you do, the cache will be an external component like Redis, so all of your many processes can use it. Or you might use a background job queue just to avoid tying up one of your precious blocking threads, even for something that has to happen right away (e.g. sending email). And so on. Contrast that with something like Go or Erlang (and by extension Elixir), where the runtime offers cheap concurrency that can fully use all of your cores in a single process, built on lightweight userland threads and asynchronous I/O, while the language lets you write straightforward, sequential code. In such an environment, a lot of the operational complexity that I described above can just go away. Simple code and simple ops -- seems like a winning combination to me.
• xgdgsc 6y ago
  Thanks. Now I understand what netdata ebpf.plugin process is doing.
• dmix 6y ago
  [warning: slight offtopic) TLDR: Any recommendations on the best way to clone one harddrive to another that doesn't take forever? > Storage I/O gained an asynchronous interface tailored-fit to work with the kind of applications that really needed it at the moment and nothing else. Say you have 2x 2TB SSD harddrives and one needs to be cloned to the other. Being the clever hacker I am who grew up using linux I simply tried unmounting the drivers and trying the usually `dd` approach (using macOS). The problem: It took >20hrs for a direct duplication of the disk. The other problem: this was legal evidence from my spouses work on a harddisk provided by police, so I assumed this was the best approach. Ultimately she had to give it in late because of my genius idea which I told her wouldn't take long. Given a time constraint the next time this happened, we gave up `dd`, and did the old mounted disk copy/paste via Finder approach... which only took only 3hrs to get 1.2TB of files across into the other HD - via usb-c interfaces. I've been speculating why one was 5x+ faster than the other (besides the fact `dd` doing a bit-by-bit copy of the filesystem). My initial suspicion was options provided `dd`: > sudo dd if=/dev/rdisk2 of=/dev/rdisk3 bs=1m conv=noerror,sync I'm not 100% familiar with the options for `dd` but I do remember a time where I changed `bs=1M` to `bs=8M` helped speed up a transfer in the past. But I didn't do it for the sake of following the instructions on StackOverflow.
  • vamega 6y ago
    Probably faster to copy via finder since it’s not actually copying every byte, but only the utilized bytes. It might be faster to have multiple rsync operations (via xargs or the like), but if the disk is relatively empty I can see this being faster. Finding the right level of parallelism isn’t something I can help you with, probably needs some experimentation.
  • trentnelson 6y ago
    I wrote some Python glue that constructed a bunch of dd commands to run concurrently which helped when I last cloned a 1TB NVMe drive. Resulting commands: dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=0 seek=0 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=1302349 seek=1302349 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=2604698 seek=2604698 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=3907047 seek=3907047 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=5209396 seek=5209396 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=6511745 seek=6511745 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=7814094 seek=7814094 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=9116443 seek=9116443 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=10418792 seek=10418792 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=11721141 seek=11721141 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=13023490 seek=13023490 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=14325839 seek=14325839 count=1302358 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=4096 skip=15628197 seek=15628197 count=6 status=progress &
• secondcoming 6y ago
  Phoronix showed that a recent bug fix in io_uring negated most of the gains when they profiled redis
• 867-5309 6y ago
  hopefully this will bubble up to higher-level C-esque languages such as PHP, for which asynchronicity is still a pain
  • junon 6y ago
    This won't help the lack of async I/O in those languages that do not support the concept as a whole. If it can't handle epoll, it certainly won't handle io_uring.
• ornornor 6y ago
  Interesting. I was just surprised as this: > Joyful things like the introduction of the automobile Cars cause so much pollution, noise, traffic, and take up so much space... How can you say its introduction is joyful? About the new api: while I’m not very knowledgeable about the kernel, it seems like very good news for performance, the improvements are drastic!
  • liamwire 6y ago
    > How can you say it’s introduction is joyful? Sure, if you want to sidestep the innumerable ways the automobile, or more accuracy the internal combustion engine, have completely revolutionised society, you could make an argument... Except you can’t. Think about the increase in distance and speed at which goods and services can be rendered compared to prior modes of transportation. One simple example that comes to mind is the ambulance, in which that increase may well be the difference between life or death for an unknown but surely enormous population. A similar argument can be made for logistic supply chains delivering medicine, food, sanity products, waste removal, all of which without, mortality would surge. Plague, famine, disease are some of largest killers in human history. Please tell me again how the absolute gains measured in human lifetimes is anything other than joyful, outside of subjectivity which is an unresolvable debate.
  • gumby 6y ago
    > Cars cause so much pollution, noise, traffic, and take up so much space... How can you say its introduction is joyful? Horses generated literal tons of pollution in cities, were complicated to look after, and slow. The arrival of the car cleaned up the streets, made it easier to travel (and travel far) and very rapidly became the preferred mode of transport all over the world.
• ganafagol 6y ago
  The title of the HN post is missing a suffix of "for a few niche applications". My work is "programming in Linux", but it's not impacted by any of this since I'm working in a different area. I'm sure this is important work, but maybe tone down such claims a bit.
  • capableweb 6y ago
    "few niche applications" being any application that touches files, network or want to run code in the kernel. Sounds like a bigger target than just "niche", but I'm no Linux developer so what do I know.
    • blub 6y ago
      eBPF has been available for a while now, one year ago there were even two books published about it, one by an author of the "bcc" mentioned in this ad disguised as a technical article. It didn't revolutionize programming in Linux that I'm aware of. It found its market in observability and performance analysis. io_uring seems to be relevant mostly to people using or wanting to use AIO. Outside of a "few niche applications" this is unimportant for the majority of Linux developers. Libraries like ASIO would likely wrap it anyway since these low-level APIs are not pleasant to use.
• jamesfisher 6y ago
  Who added the two generic Covid paragraphs to the start of this otherwise good article? _Please_ stop.
  • bauerd 6y ago
    Such an odd thing to open an article about IO tech with …
• GoblinSlayer 6y ago
  So, after all async io converges on io completion port design?
• Matthias247 6y ago
  There's currently a lot of talk about io_uring, but most articles around it and usages still seem more in the exploration, research and toy project state. I'm however wondering what the actual quality level is, whether people used it successfully in production and whether there is an overview with which kernel level which feature works without any [known] bugs. When looking at the mailing list at https://lore.kernel.org/io-uring/ https://lore.kernel.org/io-uring/ it seems like it is still a very fast moving project, with a fair amount bugfixes. Given that, is it realistic to think about using any kernel in with a kernel version between 5.5 and 5.7 in production where any bug would incur an availability impact, or should this still rather be a considered an ongoing implementation effort and revisited at some 5.xy version? An extensive set of unit-tests would make it a bit easier to gain trust into that everything works reliably and stays working, but unfortunately those are still not a thing in most low-level projects.
  • junon 6y ago
    Don't use io_uring until at least 5.10 rc3, if not 5.11. SQPOLL is still to be properly added and fixed and there are some security concerns (e.g. CAP_SYS_ADMIN being replaced by CAP_SYS_NICE to start a kernel submission queue polling thread). io_uring has many tests in the companion user space library liburing, maintained by the same person that made the kernel patches (Jens Axboe). They test both the library as well as expected functionality in the kernel. io_uring is not going to give you speed ups if you use it in the same way as you would epoll or kqueue. Thus, simply sticking it into e.g. libuv without changing how the applications are built probably won't give you a lot of benefit (speculating). It comes down to how you work with the ring buffers and how much you take advantage of the highly out-of-order, memory-barrier-based shared memory approach as opposed to more "discrete" (maybe not the right word) syscalls. As of yet, I haven't personally come across a published example of a production framework that utilizes these features adequately. We have some internal IP that does, but probably won't be open sourced.
  • pure_simplicity 6y ago
    This article recently featured on HN may be of interest to answer your question: https://itnext.io/modern-storage-is-plenty-fast-it-is-the-apis-that-are-bad-6a68319fbc1a https://itnext.io/modern-storage-is-plenty-fast-it-is-the-ap... io_uring allows for better utilization of fast storage.
• TomSwirly 6y ago
  > Joyful things like the introduction of the automobile, which forever changed the landscape of cities around the world. What?!
• bogomipz 6y ago
  The author states: >"It’s beyond our scope to explain why, but this readiness mechanism really works only for network sockets and pipes — to the point that epoll() doesn’t even accept storage files." Could someone say here explain why this readiness mechanism really works only for network sockets and pipes and not for disk?
• nahuel0x 6y ago
  So asynchronous message passing is faster than syscalls? Andy Tanenbaum laughs last.