8 ms·

BSD kqueue is a mountain of technical debt (2021)

121 points by thunderbong 2y ago

caboteria 2y ago
(2021)
Joker_vD 2y ago
So the argument is that extending kqueue's interface to handle more and more event is worse than turning more and more events into xxxfd subsystems. Why is that worse, again? Like, not in "it's not a composable design" abstract sense but in concrete "you can't do this and that, which are useful things" sense?
- jsnell 2y ago
  Yeah, the article really doesn't make a case as much as assert it. One way to think about this is whether all these non-file fds are useful for a variety of generic operations, or whether they're only used for epoll and one-off system calls or ioctls specific to that type of fd. If it's the latter, it seems hard to believe that there actually is some kind of composability advantage. So, what can you do with them? 1. You can use these fds with poll()/select(), not just epoll. That's not a big deal, since nobody really should be using either of them. But it does also mean that you should ideally be able to use them with any future event notification syscalls as well. And the invention of new mechanisms is not hypothetical, since Linux added io_uring. I'd be curious to know how well io_uring supports arbitrary fd types, and whether they got that support "for free" or if io_uring needs to be extended for every new fd type. 2. You can typically read() structured data from those fds describing the specific event. With kqueue all that data would need to be passed through struct kevent, which needs to be fixed size and generic enough to be shared between all the different event types. 3. You can pass individual fds between processes, either via UDS or fork(). I expect you would not be able to do that for individual kqueue filters. 4. You can close() the fd to release the underlying kernel resource. This doesn't seem interesting, just listing it for completeness. So there's probably enough smoke there that one could at least argue the case. It's too bad the author didn't.
  - timewizard 2y ago
    [dead]
  - stevefolta 2y ago
    > ...nobody really should be using either of them. What's wrong with poll(), at least for smaller values of nfds? And what should one use instead when writing POSIX-compatible portable code, where epoll() and kqueue() don't exist?
    - somat 2y ago
      Fun fact, on openbsd(at least, I don't know about freebsd) select and poll were reworked to use kqueue internally. The thread is an interesting read as it sounds like the naive approach has negative performance implications, knowing openbsd I suspect the main motivation was to reduce maintenance burden, that is, make it easier to reason about the internals of event driven mechanisms by only having one such system to worry about. https://marc.info/?l=openbsd-tech&m=162444132017561&w=2 https://marc.info/?l=openbsd-tech&m=162444132017561&w=2
- ajross 2y ago
  Do you have a third party driver that exposes a file descriptor for some oddball purpose? epoll can wait on that. It's absolutely true that within the bounds of what the OS supports, both interfaces are complete. But that's sort of a specious point. One requires a giant tree of extra types be amended every time something changes, while the other exploits a nearly-five-decade old abstraction that already encapsulates the idea of "something on which you might want to wait".
  - markjdb 2y ago
    The same is true of kqueue/kevent though... the driver just needs to decide which filters it wants to implement. There's no need to extend kqueue when adding some custom driver or fd type. One just needs to define some semantics for the existing filters.
    - ajross 2y ago
      > the driver just needs to decide That's pretty much the definition of technical debt though. "This interface works fine for you if you take steps to handle it specially according to its requirements". It makes kqueue into a point of friction for everything in the system wanting to provide a file descriptor-like API.
      - netbsdusers 2y ago
        No it isn't. Letting files be poll/select/epoll'd isn't free either. They don't get support for that by magic. A poll operation has to be coded, and this is just as a much a "point of friction" then as supporting kqueue. (It bears mentioning as well that on at least DragonFly BSD and OpenBSD, they have reimplemented poll()/select() to use kqueue's knotes mechanism, and so now you only have to write a VOP_KQFILTER implementation and not a VOP_POLL too.)
        ajross 2y ago
        > Letting files be poll/select/epoll'd isn't free either. Yes, but those slashes are showing the lie in the statement. Letting files be polled/selected isn't "free", but it's standard. The poll() method has been in struct file_operations for literally decades[1]. Adding "epoll support" requires no meaningful changes to the API, for any device that ever supported select(). That kind of evolutionary flexibility (the opposite of "technical debt") is generally regarded as good design. And it's something that epoll had designed in and something that queue lacks, having decided to go its own way. And it's not unreasonable to call that out, IMHO. [1] It's present in commit 1da177e4c3f4 ("Linux-2.6.12-rc2"), which is the very first git commit. I know people maintain archives of older trees, but I'm too lazy to dig. Suffice it to say that epoll relies on an interface that is likely older than many of the driver developers using it.
- Someone 2y ago
  I would think it’s the other way around. If you have separate subsystems for events of types A, B, C, etc. there’s no easy way you can have a process wait for “an event of type A or C, whichever comes first”. There also is the thing that, sometimes, different types of event are identical types, lower down in the stack. A simple example are keyboard and mouse events generated from USB devices. Why would the low-level USB stack have to do a switch on device type to figure out what subsystem to post an event to?
- IshKebab 2y ago
  Because "wait for an event" is a good abstraction for many event types. The epoll interface is a generic wait interface whereas the kqueue one is a hardcoded list of things you can wait for. Think about async programming you have a single `await` keyword that can await lots of different things.
  - toast0 2y ago
    How many event types do you need though? And would you actually need to do less work to add support for a new one to a user program? It would be different work, for sure, but I don't think it would be meaningfully less work.
baggy_trough 2y ago
I must have missed where the mountain of technical debt was. The article just says epoll has a more composable design.
- vacuity 2y ago
  To be fair (?), the title seems like classic clickbait. The thesis seems to just be > While I agree that epoll doesn’t come close, I think that’s actually a feature that has lead to a much more flexible and composable design. You could read it as "kqueue is more prone to tech debt than epoll if events keep getting added".
klabb3 2y ago
While I don’t doubt that the author has good reason for their opinion, the reason is handwaved here, no? Where is the tech debt? In the kernel only? Does it affect user space? For who is it more or less composable? I personally think ”async IO” (a term I don’t love, but anyway), should be agreed upon and settled across OSs. Not necessarily exact syscall equivalence, but the major design choices needs to be the same. For instance, who owns the buffers (kernel or user), and should it be completion or ready-based event systems? The reason I think this is so important to agree upon, is that language runtimes (and other custom runtimes like libuv) need to be simplified in order to allow for healthy growth. Any language that doesn’t have cross-platform support for ”async IO” is completely nerfed, increasing the barrier of entry for things that otherwise would be ”graduate level” projects, into deep OS/kernel knowledge that is hard to find outside FAANG and a few other spaces.
- BobbyTables2 2y ago
  I fully agree. Synchronous I/O looks appealing for beginners and simple projects. It seems like the simplest path for things where I/O isn’t the primary focus. Then one suddenly finds themselves juggling ephemeral threads to “background” tasks or such. The application quickly becomes a multi-threaded monster and still suffers responsiveness issues from synchronous use. The Tower of Babel created from third party runtimes further adds to the mess even for those who try to do better.
  - worik 2y ago
    > Synchronous I/O looks appealing for beginners and simple projects. Yes. And. Synchronous I/O is suitable for almost all projects When it is unsuitable it is catastrophicly unsuitable,
    - astrange 2y ago
      Synchronous I/O is actually better than async for many programs, partly because almost all async systems were designed without thinking about priority inversions and can't resolve them properly. If you have futures, you have this problem.
marcodiego 2y ago
I remember that when io_uring was in it's early stages several pointed to kqueue (also nt'wait_for_multiple_something or iocp and solaris event ports). By coming later, I think io_uring was able to better fit modern reality and also avoid problems from previous implementations. Hope security issues with it are solved and usage becomes mostly transparent through userspace libs, it looks like a high performance strategy for our current computing hardware.
- petesergeant 2y ago
  > I think io_uring was able to better fit modern reality Genuinely would be fascinated to understand what that means
  - nesarkvechnep 2y ago
    I'm wondering what has fundamentally changed in computing between the creation of kqueue and io_uring so the latter is able to better fit modern reality.
    - vacuity 2y ago
      I don't know that there were relevant changes between the advent of each, so much as kqueue just didn't have existing features in mind. I assume GP is referring to the ring buffer design and/or completion-based processing, as part of the ability to batch syscall processing. This is reminiscent of external work like FlexSC and can be viewed as mechanical sympathy.
    - deleted 2y ago
      [deleted]
    - mananaysiempre 2y ago
      I don’t know which things here are actually relevant to the differences between the two, but of course there have been changes. Core counts are much higher. You can stripe some enterprise SSDs and get bandwidth within an order of magnitude or so of your RAM. Yet clocks aren’t that much higher, and user-supervisor transitions are comparatively much more expensive. There’s a reason Lemire’s talk on simdjson is called “Data engineering at the speed of your disk”.
Validark 2y ago
A good start to an article, but unfortunately it feels like it was cut off before the asserted argument was demonstrated to be true.
deleted 2y ago
[deleted]
jcalvinowens 2y ago
I'll save you some time if you haven't read this yet: > Hopefully, as you can see, epoll can automatically monitor any kind of kernel resource without having to be modified, due to its composable design, which makes it superior to kqueue from the perspective of having less technical debt. That's literally the entire argument. That's not what technical debt means. The author doesn't reference a single line of source code: this is vapid clickbait.
- steeleduncan 2y ago
  > That's not what technical debt means Undoubtedly, I'm not sure why technical debt is in the title > this is vapid clickbait This seems unfair. It is a well written discussion of the difference between kqueue on BSDs and epoll on linux, and a historical overview of predecessor APIs. It just has nothing to do with technical debt, which, given the title, is admittedly odd
  - codr7 2y ago
    I'm curious: What would be the difference between suboptimal decisions made earlier in the history of a code base and technical debt?
deleted 2y ago
[deleted]
StayTrue 2y ago
Technical debt has become a meaningless term.
mwkaufma 2y ago
Making a mountain out of a molehill.
nobodyandproud 2y ago
Is this correct? How I interpreted the author: - kqueue : Designed to work off of concrete event filters. New event types mean kqueue must be rengineered support it (at the very least, a new event filter type). Versus - epoll : Designed to work off of kernel handles. The actual event type isn’t the concern of epoll and therefore epoll itself remains unaffected if a new event type were introduced. I’m guessing composibility is mentioned because of this decoupling? I’d think a better explanation would be single-responsibility, but I’m likely not understanding the author correctly here.
- adrian_b 2y ago
  I have not looked at kqueue, but there is no reason to believe that it would be affected by the introduction of a new event filter more than epoll is affected by the introduction of a new system call that creates a new kind of pseudo-file handles that can be passed to epoll. The event filters must have a standard interface, so adding one more kind should not matter for kqueue more than the existence of a new kind of "file" handle matters for epoll. The epoll API makes opaque for the user the differences between the kinds of kernel handles, but at some point inside the kernel the handles must be disambiguated and appropriate actions for each kind of handle must be performed, exactly like in kqueue various kinds of event filters must be invoked.
  - nobodyandproud 2y ago
    Right. I’ve never worked in the kernel space, but I wanted to understand the author. As you mentioned, event handlers still only care for certain messages and ignore others, so the disambiguation must happen. Rereading it; what’s also not clear to me is whether epoll is a single queue, or one queue for each subsystem and handle type. From a handler’s perspective, it seems like an implementation detail? Again, I don’t know about this stuff; just hoping someone knowledgeable can help clear it up.
- markjdb 2y ago
  I don't think the article does a good job of arguing its premise, which I think is that kqueue is a less general interface than epoll. When adding a new descriptor type, one can define semantics for existing filters (e.g., EVFILT_READ) as one pleases. To give an example, FreeBSD has EVFILT_PROCDESC to watch for events on process descriptors, which are basically analogous to pidfds. Right now, using that filter kevent() can tell you that the process referenced by a procdesc has exited. That could have been defined using the EVFILT_READ filter instead of or in addition to adding EVFILT_PROCDESC. There was no specific need to introduce EVFILT_PROCDESC, except that the implementor presumably wanted to leave space to add additional event types, and it seemed cleaner to introduce a new EVFILT_PROCDESC filter. Process descriptors don't implement EVFILT_READ today, but there's no reason they can't. So if one wants to define a new event type using kevent(), one has the option of adding a new definition (new filter, new note type for an existing filter, etc.), or adding a new type of file descriptor which implements EVFILT_READ and other "standard" filters. kqueue doesn't really constrain you either way. In FreeBSD, most of the filter types correspond to non-fd-based events. But nothing stops one from adding new fd types for similar purposes. For instance, we have both EVFILT_TIMER (a non-fd event filter) and timerfd (which implements EVFILT_READ and in particular didn't need a new filter). Both are roughly equivalent; the latter behaves more like a regular file descriptor from kqueue's perspective, which might be better, but it'd be nice to see an example illustrating how. One could argue that the simultaneous existence of timerfds and EVFILT_TIMER is technical debt, but that's not really kqueue's fault. EVFILT_TIMER has been around forever, and timerfd was added to improve Linux compatibility. So, I think the article is misguided. In particular, the claim that "any time you want kqueue to do something new, you have to add a new type of event filter" is just wrong. I'm not arguing that there isn't technical debt here, but it's not really because of kqueue's design.
WesolyKubeczek 2y ago
I went to read the article hoping that it would shed light on some nitty-gritty internals of the Free(or any other)BSD kernel and tell a story. The story would be, of course, about how the brave and bold developers undertook a journey to add something to the kqueue mechanism, or fix a bug in it, or increase its performance, or improve its scalability, and how on their way there they hit the legendary, impenetrable Mountain of Technical Debt. How they tried to force their way over the mountain first, but how the dangers and the cold drove them to try to go in depth, via disused tunnels made by some hard-working fearless people many moons ago. In there, our brave team discovers long-decayed badly burned bodies, rusted tools, and fragmentary diaries made by some of those people. They learn that the hard-working tunnelling people undertook a quest not unlike their own; they also felt that the Mountain of Technical Debt could actually present an opportunity to enrich themselves, and while mining it for Bugfixes and Improvements here and there, they accidentally awoke an Ancient Evil of Regression-Rich Behavior, the Mother of All Heisenbugs. The hard-working tunnelling people were no strangers to the battle, and they faced the enemy, but it overpowered them with Crippling Regressions and Massive Burnout. Only a few survivors lived to tell the tale, and while what they told was confusing, the clear message was to avoid the Mountain and never approach the tunnels again. At least this was what everyone remembered, but almost everyone forgot what evil fought them and drove them out. Our heroes realize that their quest has suddenly become much more dangerous than they thought, but they don't want to stop halfway, and they can be hard-working and hard-fighting too, and their tools are much more superior than those of their predecessors. They also think that they are much more resistant to burnout. Of course they go on, and eventually they meet the creature. Each attempt at improving kqueue yields such obscure regressions and bugs elsewhere in the system, that half of the team succumbs to Burnout unleashed on them despite youth, vigor, tools, and strength. Two brave souls, armed with the best debuggers, fuzzers, and in-depth code analyzers, manage to escape the Mother of Heisenbugs, wounding it, and damaging some of the Mountain itself. They describe their findings in a much more coherent way than their surviving predecessors have managed to, and conclude that the Mountain and the creature cannot be defeated on their home ground, not until a viable replacement arises which will be compatible with and and all the quirks the software grew to rely on. This replacement would starve the creature and the Mountain would erode. The story of the kqueue Replacement Building would be a separate one, in a different genre, with our brave heroes acting as wise advisors to the new cast of protagonists, detailing challenges they all run into, and how their combined wisdom, intelligence, and superb teamwork and camaraderie help them overcome hardships — sometimes barely, but still. Needless to say, the article failed me greatly. And it didn't even make an attempt to describe where the debt was!
sc68cal 2y ago
kqueue was if not the, then one of the first attempts to solve the issue of being more performant than select(). So this whole post boils down to "they didn't get it right the first time" to which I say, okay sure, but we only know that now after kqueue proved that event based queues were better than select. This post is ahistorical and anti-social.
ryao 2y ago
I don’t see how adding a new syscall to monitor something new via epoll is more desirable than adding a new BSD event filter. Either way, you are modifying the syscall interface. The initial epoll_create() had to be revised with epoll_create1(). It is even cited on slide 53 as being an example of a past design failure: https://www.man7.org/conf/lpc2014/designing_linux_kernel_APIs.pdf https://www.man7.org/conf/lpc2014/designing_linux_kernel_API... Interestingly, the BSD kqueue made the same mistake and had to be revised for the same reason, which is why kqueue1() was made. However, by the points on the technical checklist there for syscall design, kqueue is an excellent design, minus the one mistake of not supporting O_CLOEXEC at the beginning.
ori_b 2y ago
Counterpoint -- epoll is fundamentally broken: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... You can end up with events on file descriptors that you don't even have open -- they leak across process boundaries. And that means that if the file descriptor gets reused, you can end up with events on the wrong file descriptor.
- zzo38computer 2y ago
  From looking at the man page, it look like epoll does not return the file descriptor of the event; it returns a union containing user-defined data, although one of the union's fields is called "fd" because it is presumably intended to be used as the file descriptor. However, this is still subject to the problems you mention, as well as that you presumably can no longer control events for a file descriptor if you do longer have it, so it is still seems a problem. Putting the file descriptor in the "struct epoll_event" instead of "epoll_data_t" would have avoided the problem of events on the wrong file descriptor, but that still might not be good enough. (It could specify -1 as the file descriptor for events of file descriptors that you do not have access to.) Some of this is just the problem with POSIX in general. I did consider such problems (of file descriptors and of event handling) in my own ideas of operating system design, which uses capabilities. I would want to avoid the mess that other systems is doing, too. A capability in this case is fundamentally a different data type than numbers (and IPC treats them differently), although I am not sure yet how to handle this in a suitable way within a process (tagged memory might do, although not all computers use tagged memory). (Note that most I/O is done using IPC and not by system calls; the number of system calls is limited and is usually only used for managing the capabilities and event handling. This would also improve security as well as other things.)
  - ori_b 2y ago
    > From looking at the man page, it look like epoll does not return the file descriptor of the event; it returns a union containing user-defined data, although one of the union's fields is called "fd" because it is presumably intended to be used as the file descriptor. Yes -- so how do you unregister an event after you close the file, or dup the file descriptor? You have no way to do that, you just keep getting events for a resource you no longer have open, so long as the parent or child still has it open! It's incredible that you can set things up in a way that you get events on a resource you can no longer refer to in any way.
kev009 2y ago
This is pretty facepalm, has the author actually used any of these extended epoll APIs? inotify - deprecated. aio - complete footgun in linux. Whatever point is trying to be made here is lost in reality, I fail to see how having a bunch of additional function calls is superior to event filters and there is no coherent argument laid out.. how is kqueue not composable if it too has added event types (i.e. Mach ports they mention)? You're going to have to modify the kernel in either case.
netbsdusers 2y ago
Five points in reply: 1. Signalfd is a mountain of technical debt. It's not like a file at all. It reads entirely different things depending on which process is reading from a single common open file description, and it interacts with epoll in a most bizarre way, in that the process that added the signalfd to the poll set is the one whose signals will notify readiness - even if that process then closed the epoll and signalfd. 2. Signalfd is a mountain of technical debt because they built it over signals. Signals are of two forms: the one is asynchronous, the other synchronous. The one should be replaced with a generic message queue in a technical debt free design. The other is debatable. (Mach and also Fuchsia have an "Exception Port" instead, a message port someone can read from to deal with exceptions incurred by some thread.) 3. Regarding: > epoll can automatically monitor any kind of kernel resource without having to be modified, due to its composable design Well, so can kqueue - EVFILT_READ/EVFILT_WRITE will work just as well on any file, be it a socket, an eventfd, or even another kqueue (though I don't think anyone who isn't dealing with technical debt should be adding kqueues to kqueues, nor epolls to epolls.) No need to modify kqueue! But in either case, though, you've got to modify something. Whether that's trying to fit signals into a file and getting signalfd, or whether it's adding a signal event filter and getting EVFILT_SIGNAL, something has to be done. 4. FreeBSD's process descriptors predate Linux's pidfds. They are also better and less technically indebted because they are pure capabilities (they were invented as part of the Capsicum object-capability model based security architecture for FreeBSD) while pidfds are not: what you can do with a pidfd is decided on based on the ambient authority of the process trying to, say, signal a pidfd, and not on rights intrinsic to the fact of having a pidfd. In fact, these pidfds are not even like traditional Unix open file descriptions, whose rights are based on the credential of who opened them. This makes privilege-separation architectures impossible with pidfds, but I digress. 5. The author ignored the actual points people argued against epoll with, viz. that 1) epoll's edge triggering behaviour is useless, and 2) that epoll's conflation of file descriptor with open file description is a terribly leaky abstraction: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-broken-22/ https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-...
- vacuity 2y ago
SMP-UX 2y ago
You don't have to necessarily use kqueue on a BSD. In areas where it might make more sense performance wise you can obviously use poll. I'm not opposed to introducing an alternative interface like epoll, but it's only recently that Linux has managed to bridge the network performance gap. I attribute that to a lot of money being poured into it with pure and raw optimization from all of the Linux sponsors that they get which is really cool and great for them but it does not necessarily make gnu/Linux a purer system. Both approaches have their merits. There's no reason to sit here and claim that one is better than the other
delta64 2y ago
It's slightly odd to say 'wow, epoll can wait on any type of kernel handle, like timerfd and signalfd!' given those *fd interfaces were added just for select/poll/epoll. Strikes me as remarkably ugly design...

121 points by thunderbong 2y ago

(2021)

So the argument is that extending kqueue's interface to handle more and more event is worse than turning more and more events into xxxfd subsystems. Why is that worse, again? Like, not in "it's not a composable design" abstract sense but in concrete "you can't do this and that, which are useful things" sense?

jsnell 2y ago
Yeah, the article really doesn't make a case as much as assert it. One way to think about this is whether all these non-file fds are useful for a variety of generic operations, or whether they're only used for epoll and one-off system calls or ioctls specific to that type of fd. If it's the latter, it seems hard to believe that there actually is some kind of composability advantage. So, what can you do with them? 1. You can use these fds with poll()/select(), not just epoll. That's not a big deal, since nobody really should be using either of them. But it does also mean that you should ideally be able to use them with any future event notification syscalls as well. And the invention of new mechanisms is not hypothetical, since Linux added io_uring. I'd be curious to know how well io_uring supports arbitrary fd types, and whether they got that support "for free" or if io_uring needs to be extended for every new fd type. 2. You can typically read() structured data from those fds describing the specific event. With kqueue all that data would need to be passed through struct kevent, which needs to be fixed size and generic enough to be shared between all the different event types. 3. You can pass individual fds between processes, either via UDS or fork(). I expect you would not be able to do that for individual kqueue filters. 4. You can close() the fd to release the underlying kernel resource. This doesn't seem interesting, just listing it for completeness. So there's probably enough smoke there that one could at least argue the case. It's too bad the author didn't.
- timewizard 2y ago
  [dead]
- stevefolta 2y ago
  > ...nobody really should be using either of them. What's wrong with poll(), at least for smaller values of nfds? And what should one use instead when writing POSIX-compatible portable code, where epoll() and kqueue() don't exist?
  - somat 2y ago
    Fun fact, on openbsd(at least, I don't know about freebsd) select and poll were reworked to use kqueue internally. The thread is an interesting read as it sounds like the naive approach has negative performance implications, knowing openbsd I suspect the main motivation was to reduce maintenance burden, that is, make it easier to reason about the internals of event driven mechanisms by only having one such system to worry about. https://marc.info/?l=openbsd-tech&m=162444132017561&w=2 https://marc.info/?l=openbsd-tech&m=162444132017561&w=2
ajross 2y ago
Do you have a third party driver that exposes a file descriptor for some oddball purpose? epoll can wait on that. It's absolutely true that within the bounds of what the OS supports, both interfaces are complete. But that's sort of a specious point. One requires a giant tree of extra types be amended every time something changes, while the other exploits a nearly-five-decade old abstraction that already encapsulates the idea of "something on which you might want to wait".
- markjdb 2y ago
  The same is true of kqueue/kevent though... the driver just needs to decide which filters it wants to implement. There's no need to extend kqueue when adding some custom driver or fd type. One just needs to define some semantics for the existing filters.
  - ajross 2y ago
    > the driver just needs to decide That's pretty much the definition of technical debt though. "This interface works fine for you if you take steps to handle it specially according to its requirements". It makes kqueue into a point of friction for everything in the system wanting to provide a file descriptor-like API.
    - netbsdusers 2y ago
      No it isn't. Letting files be poll/select/epoll'd isn't free either. They don't get support for that by magic. A poll operation has to be coded, and this is just as a much a "point of friction" then as supporting kqueue. (It bears mentioning as well that on at least DragonFly BSD and OpenBSD, they have reimplemented poll()/select() to use kqueue's knotes mechanism, and so now you only have to write a VOP_KQFILTER implementation and not a VOP_POLL too.)
      - ajross 2y ago
        > Letting files be poll/select/epoll'd isn't free either. Yes, but those slashes are showing the lie in the statement. Letting files be polled/selected isn't "free", but it's standard. The poll() method has been in struct file_operations for literally decades[1]. Adding "epoll support" requires no meaningful changes to the API, for any device that ever supported select(). That kind of evolutionary flexibility (the opposite of "technical debt") is generally regarded as good design. And it's something that epoll had designed in and something that queue lacks, having decided to go its own way. And it's not unreasonable to call that out, IMHO. [1] It's present in commit 1da177e4c3f4 ("Linux-2.6.12-rc2"), which is the very first git commit. I know people maintain archives of older trees, but I'm too lazy to dig. Suffice it to say that epoll relies on an interface that is likely older than many of the driver developers using it.
Someone 2y ago
I would think it’s the other way around. If you have separate subsystems for events of types A, B, C, etc. there’s no easy way you can have a process wait for “an event of type A or C, whichever comes first”. There also is the thing that, sometimes, different types of event are identical types, lower down in the stack. A simple example are keyboard and mouse events generated from USB devices. Why would the low-level USB stack have to do a switch on device type to figure out what subsystem to post an event to?
IshKebab 2y ago
Because "wait for an event" is a good abstraction for many event types. The epoll interface is a generic wait interface whereas the kqueue one is a hardcoded list of things you can wait for. Think about async programming you have a single `await` keyword that can await lots of different things.
- toast0 2y ago
  How many event types do you need though? And would you actually need to do less work to add support for a new one to a user program? It would be different work, for sure, but I don't think it would be meaningfully less work.

baggy_trough 2y ago

I must have missed where the mountain of technical debt was. The article just says epoll has a more composable design.

vacuity 2y ago
To be fair (?), the title seems like classic clickbait. The thesis seems to just be > While I agree that epoll doesn’t come close, I think that’s actually a feature that has lead to a much more flexible and composable design. You could read it as "kqueue is more prone to tech debt than epoll if events keep getting added".

klabb3 2y ago

While I don’t doubt that the author has good reason for their opinion, the reason is handwaved here, no? Where is the tech debt? In the kernel only? Does it affect user space? For who is it more or less composable? I personally think ”async IO” (a term I don’t love, but anyway), should be agreed upon and settled across OSs. Not necessarily exact syscall equivalence, but the major design choices needs to be the same. For instance, who owns the buffers (kernel or user), and should it be completion or ready-based event systems? The reason I think this is so important to agree upon, is that language runtimes (and other custom runtimes like libuv) need to be simplified in order to allow for healthy growth. Any language that doesn’t have cross-platform support for ”async IO” is completely nerfed, increasing the barrier of entry for things that otherwise would be ”graduate level” projects, into deep OS/kernel knowledge that is hard to find outside FAANG and a few other spaces.

BobbyTables2 2y ago
I fully agree. Synchronous I/O looks appealing for beginners and simple projects. It seems like the simplest path for things where I/O isn’t the primary focus. Then one suddenly finds themselves juggling ephemeral threads to “background” tasks or such. The application quickly becomes a multi-threaded monster and still suffers responsiveness issues from synchronous use. The Tower of Babel created from third party runtimes further adds to the mess even for those who try to do better.
- worik 2y ago
  > Synchronous I/O looks appealing for beginners and simple projects. Yes. And. Synchronous I/O is suitable for almost all projects When it is unsuitable it is catastrophicly unsuitable,
  - astrange 2y ago
    Synchronous I/O is actually better than async for many programs, partly because almost all async systems were designed without thinking about priority inversions and can't resolve them properly. If you have futures, you have this problem.

marcodiego 2y ago

I remember that when io_uring was in it's early stages several pointed to kqueue (also nt'wait_for_multiple_something or iocp and solaris event ports). By coming later, I think io_uring was able to better fit modern reality and also avoid problems from previous implementations. Hope security issues with it are solved and usage becomes mostly transparent through userspace libs, it looks like a high performance strategy for our current computing hardware.

petesergeant 2y ago
> I think io_uring was able to better fit modern reality Genuinely would be fascinated to understand what that means
- nesarkvechnep 2y ago
  I'm wondering what has fundamentally changed in computing between the creation of kqueue and io_uring so the latter is able to better fit modern reality.
  - vacuity 2y ago
    I don't know that there were relevant changes between the advent of each, so much as kqueue just didn't have existing features in mind. I assume GP is referring to the ring buffer design and/or completion-based processing, as part of the ability to batch syscall processing. This is reminiscent of external work like FlexSC and can be viewed as mechanical sympathy.
  - deleted 2y ago
    [deleted]
  - mananaysiempre 2y ago
    I don’t know which things here are actually relevant to the differences between the two, but of course there have been changes. Core counts are much higher. You can stripe some enterprise SSDs and get bandwidth within an order of magnitude or so of your RAM. Yet clocks aren’t that much higher, and user-supervisor transitions are comparatively much more expensive. There’s a reason Lemire’s talk on simdjson is called “Data engineering at the speed of your disk”.

Validark 2y ago

A good start to an article, but unfortunately it feels like it was cut off before the asserted argument was demonstrated to be true.

deleted 2y ago

[deleted]

jcalvinowens 2y ago

I'll save you some time if you haven't read this yet: > Hopefully, as you can see, epoll can automatically monitor any kind of kernel resource without having to be modified, due to its composable design, which makes it superior to kqueue from the perspective of having less technical debt. That's literally the entire argument. That's not what technical debt means. The author doesn't reference a single line of source code: this is vapid clickbait.

steeleduncan 2y ago
> That's not what technical debt means Undoubtedly, I'm not sure why technical debt is in the title > this is vapid clickbait This seems unfair. It is a well written discussion of the difference between kqueue on BSDs and epoll on linux, and a historical overview of predecessor APIs. It just has nothing to do with technical debt, which, given the title, is admittedly odd
- codr7 2y ago
  I'm curious: What would be the difference between suboptimal decisions made earlier in the history of a code base and technical debt?

deleted 2y ago

[deleted]

StayTrue 2y ago

Technical debt has become a meaningless term.

mwkaufma 2y ago

Making a mountain out of a molehill.

nobodyandproud 2y ago

Is this correct? How I interpreted the author: - kqueue : Designed to work off of concrete event filters. New event types mean kqueue must be rengineered support it (at the very least, a new event filter type). Versus - epoll : Designed to work off of kernel handles. The actual event type isn’t the concern of epoll and therefore epoll itself remains unaffected if a new event type were introduced. I’m guessing composibility is mentioned because of this decoupling? I’d think a better explanation would be single-responsibility, but I’m likely not understanding the author correctly here.

adrian_b 2y ago
I have not looked at kqueue, but there is no reason to believe that it would be affected by the introduction of a new event filter more than epoll is affected by the introduction of a new system call that creates a new kind of pseudo-file handles that can be passed to epoll. The event filters must have a standard interface, so adding one more kind should not matter for kqueue more than the existence of a new kind of "file" handle matters for epoll. The epoll API makes opaque for the user the differences between the kinds of kernel handles, but at some point inside the kernel the handles must be disambiguated and appropriate actions for each kind of handle must be performed, exactly like in kqueue various kinds of event filters must be invoked.
- nobodyandproud 2y ago
  Right. I’ve never worked in the kernel space, but I wanted to understand the author. As you mentioned, event handlers still only care for certain messages and ignore others, so the disambiguation must happen. Rereading it; what’s also not clear to me is whether epoll is a single queue, or one queue for each subsystem and handle type. From a handler’s perspective, it seems like an implementation detail? Again, I don’t know about this stuff; just hoping someone knowledgeable can help clear it up.
markjdb 2y ago
I don't think the article does a good job of arguing its premise, which I think is that kqueue is a less general interface than epoll. When adding a new descriptor type, one can define semantics for existing filters (e.g., EVFILT_READ) as one pleases. To give an example, FreeBSD has EVFILT_PROCDESC to watch for events on process descriptors, which are basically analogous to pidfds. Right now, using that filter kevent() can tell you that the process referenced by a procdesc has exited. That could have been defined using the EVFILT_READ filter instead of or in addition to adding EVFILT_PROCDESC. There was no specific need to introduce EVFILT_PROCDESC, except that the implementor presumably wanted to leave space to add additional event types, and it seemed cleaner to introduce a new EVFILT_PROCDESC filter. Process descriptors don't implement EVFILT_READ today, but there's no reason they can't. So if one wants to define a new event type using kevent(), one has the option of adding a new definition (new filter, new note type for an existing filter, etc.), or adding a new type of file descriptor which implements EVFILT_READ and other "standard" filters. kqueue doesn't really constrain you either way. In FreeBSD, most of the filter types correspond to non-fd-based events. But nothing stops one from adding new fd types for similar purposes. For instance, we have both EVFILT_TIMER (a non-fd event filter) and timerfd (which implements EVFILT_READ and in particular didn't need a new filter). Both are roughly equivalent; the latter behaves more like a regular file descriptor from kqueue's perspective, which might be better, but it'd be nice to see an example illustrating how. One could argue that the simultaneous existence of timerfds and EVFILT_TIMER is technical debt, but that's not really kqueue's fault. EVFILT_TIMER has been around forever, and timerfd was added to improve Linux compatibility. So, I think the article is misguided. In particular, the claim that "any time you want kqueue to do something new, you have to add a new type of event filter" is just wrong. I'm not arguing that there isn't technical debt here, but it's not really because of kqueue's design.

WesolyKubeczek 2y ago

I went to read the article hoping that it would shed light on some nitty-gritty internals of the Free(or any other)BSD kernel and tell a story. The story would be, of course, about how the brave and bold developers undertook a journey to add something to the kqueue mechanism, or fix a bug in it, or increase its performance, or improve its scalability, and how on their way there they hit the legendary, impenetrable Mountain of Technical Debt. How they tried to force their way over the mountain first, but how the dangers and the cold drove them to try to go in depth, via disused tunnels made by some hard-working fearless people many moons ago. In there, our brave team discovers long-decayed badly burned bodies, rusted tools, and fragmentary diaries made by some of those people. They learn that the hard-working tunnelling people undertook a quest not unlike their own; they also felt that the Mountain of Technical Debt could actually present an opportunity to enrich themselves, and while mining it for Bugfixes and Improvements here and there, they accidentally awoke an Ancient Evil of Regression-Rich Behavior, the Mother of All Heisenbugs. The hard-working tunnelling people were no strangers to the battle, and they faced the enemy, but it overpowered them with Crippling Regressions and Massive Burnout. Only a few survivors lived to tell the tale, and while what they told was confusing, the clear message was to avoid the Mountain and never approach the tunnels again. At least this was what everyone remembered, but almost everyone forgot what evil fought them and drove them out. Our heroes realize that their quest has suddenly become much more dangerous than they thought, but they don't want to stop halfway, and they can be hard-working and hard-fighting too, and their tools are much more superior than those of their predecessors. They also think that they are much more resistant to burnout. Of course they go on, and eventually they meet the creature. Each attempt at improving kqueue yields such obscure regressions and bugs elsewhere in the system, that half of the team succumbs to Burnout unleashed on them despite youth, vigor, tools, and strength. Two brave souls, armed with the best debuggers, fuzzers, and in-depth code analyzers, manage to escape the Mother of Heisenbugs, wounding it, and damaging some of the Mountain itself. They describe their findings in a much more coherent way than their surviving predecessors have managed to, and conclude that the Mountain and the creature cannot be defeated on their home ground, not until a viable replacement arises which will be compatible with and and all the quirks the software grew to rely on. This replacement would starve the creature and the Mountain would erode. The story of the kqueue Replacement Building would be a separate one, in a different genre, with our brave heroes acting as wise advisors to the new cast of protagonists, detailing challenges they all run into, and how their combined wisdom, intelligence, and superb teamwork and camaraderie help them overcome hardships — sometimes barely, but still. Needless to say, the article failed me greatly. And it didn't even make an attempt to describe where the debt was!

sc68cal 2y ago

kqueue was if not the, then one of the first attempts to solve the issue of being more performant than select(). So this whole post boils down to "they didn't get it right the first time" to which I say, okay sure, but we only know that now after kqueue proved that event based queues were better than select. This post is ahistorical and anti-social.

ryao 2y ago

I don’t see how adding a new syscall to monitor something new via epoll is more desirable than adding a new BSD event filter. Either way, you are modifying the syscall interface. The initial epoll_create() had to be revised with epoll_create1(). It is even cited on slide 53 as being an example of a past design failure: https://www.man7.org/conf/lpc2014/designing_linux_kernel_APIs.pdf https://www.man7.org/conf/lpc2014/designing_linux_kernel_API... Interestingly, the BSD kqueue made the same mistake and had to be revised for the same reason, which is why kqueue1() was made. However, by the points on the technical checklist there for syscall design, kqueue is an excellent design, minus the one mistake of not supporting O_CLOEXEC at the beginning.

ori_b 2y ago

Counterpoint -- epoll is fundamentally broken: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... You can end up with events on file descriptors that you don't even have open -- they leak across process boundaries. And that means that if the file descriptor gets reused, you can end up with events on the wrong file descriptor.

zzo38computer 2y ago
From looking at the man page, it look like epoll does not return the file descriptor of the event; it returns a union containing user-defined data, although one of the union's fields is called "fd" because it is presumably intended to be used as the file descriptor. However, this is still subject to the problems you mention, as well as that you presumably can no longer control events for a file descriptor if you do longer have it, so it is still seems a problem. Putting the file descriptor in the "struct epoll_event" instead of "epoll_data_t" would have avoided the problem of events on the wrong file descriptor, but that still might not be good enough. (It could specify -1 as the file descriptor for events of file descriptors that you do not have access to.) Some of this is just the problem with POSIX in general. I did consider such problems (of file descriptors and of event handling) in my own ideas of operating system design, which uses capabilities. I would want to avoid the mess that other systems is doing, too. A capability in this case is fundamentally a different data type than numbers (and IPC treats them differently), although I am not sure yet how to handle this in a suitable way within a process (tagged memory might do, although not all computers use tagged memory). (Note that most I/O is done using IPC and not by system calls; the number of system calls is limited and is usually only used for managing the capabilities and event handling. This would also improve security as well as other things.)
- ori_b 2y ago
  > From looking at the man page, it look like epoll does not return the file descriptor of the event; it returns a union containing user-defined data, although one of the union's fields is called "fd" because it is presumably intended to be used as the file descriptor. Yes -- so how do you unregister an event after you close the file, or dup the file descriptor? You have no way to do that, you just keep getting events for a resource you no longer have open, so long as the parent or child still has it open! It's incredible that you can set things up in a way that you get events on a resource you can no longer refer to in any way.

kev009 2y ago

This is pretty facepalm, has the author actually used any of these extended epoll APIs? inotify - deprecated. aio - complete footgun in linux. Whatever point is trying to be made here is lost in reality, I fail to see how having a bunch of additional function calls is superior to event filters and there is no coherent argument laid out.. how is kqueue not composable if it too has added event types (i.e. Mach ports they mention)? You're going to have to modify the kernel in either case.

netbsdusers 2y ago

Five points in reply: 1. Signalfd is a mountain of technical debt. It's not like a file at all. It reads entirely different things depending on which process is reading from a single common open file description, and it interacts with epoll in a most bizarre way, in that the process that added the signalfd to the poll set is the one whose signals will notify readiness - even if that process then closed the epoll and signalfd. 2. Signalfd is a mountain of technical debt because they built it over signals. Signals are of two forms: the one is asynchronous, the other synchronous. The one should be replaced with a generic message queue in a technical debt free design. The other is debatable. (Mach and also Fuchsia have an "Exception Port" instead, a message port someone can read from to deal with exceptions incurred by some thread.) 3. Regarding: > epoll can automatically monitor any kind of kernel resource without having to be modified, due to its composable design Well, so can kqueue - EVFILT_READ/EVFILT_WRITE will work just as well on any file, be it a socket, an eventfd, or even another kqueue (though I don't think anyone who isn't dealing with technical debt should be adding kqueues to kqueues, nor epolls to epolls.) No need to modify kqueue! But in either case, though, you've got to modify something. Whether that's trying to fit signals into a file and getting signalfd, or whether it's adding a signal event filter and getting EVFILT_SIGNAL, something has to be done. 4. FreeBSD's process descriptors predate Linux's pidfds. They are also better and less technically indebted because they are pure capabilities (they were invented as part of the Capsicum object-capability model based security architecture for FreeBSD) while pidfds are not: what you can do with a pidfd is decided on based on the ambient authority of the process trying to, say, signal a pidfd, and not on rights intrinsic to the fact of having a pidfd. In fact, these pidfds are not even like traditional Unix open file descriptions, whose rights are based on the credential of who opened them. This makes privilege-separation architectures impossible with pidfds, but I digress. 5. The author ignored the actual points people argued against epoll with, viz. that 1) epoll's edge triggering behaviour is useless, and 2) that epoll's conflation of file descriptor with open file description is a terribly leaky abstraction: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-broken-22/ https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-...

vacuity 2y ago

SMP-UX 2y ago

You don't have to necessarily use kqueue on a BSD. In areas where it might make more sense performance wise you can obviously use poll. I'm not opposed to introducing an alternative interface like epoll, but it's only recently that Linux has managed to bridge the network performance gap. I attribute that to a lot of money being poured into it with pure and raw optimization from all of the Linux sponsors that they get which is really cool and great for them but it does not necessarily make gnu/Linux a purer system. Both approaches have their merits. There's no reason to sit here and claim that one is better than the other

delta64 2y ago

It's slightly odd to say 'wow, epoll can wait on any type of kernel handle, like timerfd and signalfd!' given those *fd interfaces were added just for select/poll/epoll. Strikes me as remarkably ugly design...