3 ms·
There are certain counterintuitive things that you have to learn if you want to be a "systems engineer", in a general sense, and this whole async thing has been
by jerf 2y ago
There are certain counterintuitive things that you have to learn if you want to be a "systems engineer", in a general sense, and this whole async thing has been one of the clearest lessons to me over the years of how seemingly identical things sometimes can not be abstracted over.
Here by "async" I don't so much mean async/await versus threads, but these kernel-level event interfaces regardless of which abstraction a programming language lays on top of them.
At the 30,000 foot view, all the async abstractions are basically the same, right? You just tell the kernel "I want to know about these things, wake me up when they happen." Surely the exact way in which they happen is not something so fundamental that you couldn't wrap an abstraction around all of them, right?
And to some extent you can, but the result is generally so lowest-common-denominator as to appeal to nobody.
Instead, every major change in how we handle async has essentially obsoleted the entire programming stack based on the previous ones. Changing from select to epoll was not just a matter of switching out the fundamental primitive, it tended to cascade up almost the entire stack. Huge swathes of code had to be rewritten to accommodate it, not just the core where you could do a bit of work and "just" swap out epoll for select.
Now we're doing it again with io_uring. You can't "just" swap out your epoll for io_uring and go zoomier. It cascades quite a ways up the stack. It turns out the guarantees that these async handlers provide are very different and very difficult to abstract. I've seen people discuss how to bring io_uring to Go and the answer seems to basically be "it breaks so much that it is questionable if it is practically possible". An ongoing discussion on an Erlang forum seems to imply it's not easy there (https://erlangforums.com/t/erlang-io-uring-support/765); https://erlangforums.com/t/erlang-io-uring-support/765); I'd bet it reaches up "less far" into the stack but it's still a huge change to BEAM, not "just" swapping out the way async events come in. I'm sure many other similar discussions are happening everywhere with regards to how to bring io_uring into existing code, both runtimes and user-level code.
This does not mean the problem is unsolvable by any means. This is not a complaint, or a pronunciation of doom, or an exhortation to panic, or anything like that. We did indeed collectively switch from select to epoll. We will collectively switch to io_uring eventually. Rust will certainly be made to work with it. I am less certain about the ability of shared libraries to be efficiently and easily written that work in both environments, though; if you lowest-common-denominator enough to work in both you're probably taking on the very disadvantages of epoll in the first place. But programmers are clever and have a lot of motivation here. I'm sure interesting solutions will emerge.
I'm just highlighting that as you grow in your programming skill and your software architecture abilities and general system engineering, this provides a very interesting window into how abstractions can not just leak a little, but leak a lot, a long ways up the stack, much farther than your intuition may suggest. Even as I am typing this, my own intuition is still telling me "Oh, how hard can this really be?" And the answer my eyes and my experience give my intuition is, "Very! Even if I can't tell you every last reason why in exhaustive detail, the evidence is clear!" If it were "just" a matter of switching, as easy as it feels like it ought to be, we'd all already be switched. But we're not, because it isn't.
- vacuity 2y agoI appreciate the insight in this comment! I see your problem, and I offer an answer (I daren't call it a solution): there is no surefire way to make an interface/abstraction withstand the test of time. It just doesn't happen, even across just a few paradigm shifts, at least not without arbitrary costs to performance, observability/debuggability, ease of use, and so on. The microkernel (in the spirit of Liedtke)/exokernel philosophy tells us to focus on providing minimal, orthogonal mechanisms that just barely allow implementing the "other stuff". But unless a monolithic system is being built for one purpose, "the other stuff" isn't meaningfully different from a microkernel; it has different "hardware" but must itself impose minimally on what is to be built above it. We build layers of components with rich interactions of meaningful abstractions, building a web of dependencies and capabilities. There is no accidental complexity here in this ideal model; to switch paradigms, one must discard exactly the set of components and layers that are incompatible with the new paradigm. Consider Linux async mechanisms. They are provided by a monolithic kernel that dictates massive swathes of what worldview a program is developed in. When select was found lacking, it took time for epoll to arrive. Then io_uring took its sweet time. When the kernel is lacking, the kernel must change, and that is painful. Now consider a hypothetical microkernel/exokernel where a program just gets bare asynchronous notifications about hardware and from other programs. Async abstractions must be built on top, in services and libraries, to make programming feasible. Say the analogous epoll library is found lacking. Someone must uproot it and perhaps lower layers and build an io_uring library instead. I will not say this is always less pain that before, although it is decidedly not the same as changing a kernel. But perhaps it is less painful in most cases. I do not think it is ever more painful. This is the essential pain brought about by stability and change.