4 ms·

Yes, both kqueue and IOCP are modelled the same: notify on operation completion, rather than when a file descriptor can perform an operation.

• wahern 10y ago
  kqueue is not completion based, at least not principally. kqueue is a kernel framework and userspace API for event signaling. The functional components are called event filters, and most of them signal readiness. On FreeBSD there is a kqueue filter for async I/O, EVFILT_AIO, an extension of POSIX AIO. It's analogous to Linux's io_submit IOCB_FLAG_RESFD flag that will signal AIO completion through an eventfd. EVFILT_AIO is a more consistent API than IOCB_FLAG_RESFD, but doesn't strictly distinguish BSD from Linux in terms of capability.[1] What makes kqueue better than the Linux epoll family of interfaces (epoll, eventfd, signalfd, inotify, etc) is that the framework was much more carefully designed and generally has saner behavior. For example, from the very beginning kqueue had EVFILT_SIGNAL for signal events. It wasn't until 7 years later that Linux got signalfd. But on Linux only a single signalfd listener will receive notification for a particular signal number, as it's basically a hack built atop the traditional signal delivery mechanism, for which there can only be a single handler. On BSD there can be multiple EVFILT_SIGNAL listeners for a signal number, and all will receive a notification. So if you have multiple components that might want to reset themselves on SIGHUP, they can't do that independently. Thus, signalfd is not much better from a functionality standpoint than just manually installing a signal handler which uses eventfd; it's merely more convenient for most cases where there's only a signal listener, but it doesn't compose well and so fails Interface Design 101. Apropos the article, kqueue will properly delete a descriptor event when you close it, regardless of whether it's been dup'd, and even though both Linux and BSD share the same distinctions between descriptors and file table entry objects.[2] It's things like that which make kqueue superior as a practical matter, above and beyond the better architectural design. So much of the annoying aspects of Linux semantics has to do with premature optimization of particular use cases. Somewhat similarly, so much that's annoying about Windows is that their interfaces are too high-level. If Microsoft's architectural decisions aren't well suited for your particular use case you're basically SOL. For example, IOCP is implemented in userland basically using something like epoll combined with a thread pool (think libdispatch on macOS where the number of slave threads is globally balanced). Microsoft could expose that readiness signaling mechanism so you could roll your own event loops (and thus make integration of projects like Node.js much simpler and cleaner), but they refuse to do so. And so while IOCP is great for the 80% of cases where you really just want completion and are happy to use their well-optimized framework, for the other 20% of cases where the design or performance isn't a good fit you have no good recourse. Windows, and to a lesser extent Linux, often violates the core design principle of keeping easy tasks easy and difficult tasks possible. They make easy things very easy but the difficult stuff often remains impossible as a practical matter. [1] Actually, EVFILT_READ and EVFILT_WRITE do support setting a low water mark, so you can theoretically condition readiness signaling on kernel buffer availability. But this is basically a kqueue-based extension to the traditional BSD Sockets API options SO_RCVLOWAT and SO_SNDLOWAT. You could theoretically get the same behavior on Linux, at least on a per socket basis (as opposed to the per event basis for kqueue) by setting those options. Alas, Linux doesn't obey SO_RCVLOWAT or SO_SNDLOWAT when polling. :( Speculating, I bet it's probably a performance optimization netting some single-digit performance improvement for the original expected use cases but, once again, keeping something difficult impossible. In any event, I hesitate to call this completion signaling because the buffer contents could be modified but the already posted event remain outstanding (if not in kernel land, in userspace land). By it's nature Windows IOCP forecloses that possibility completely. [2] In Unix there are universally three kernel data structures involved with any file-like resource: the file descriptor object, the open file table entry object, and the resource object itself. So a single socket object could be referenced by multiple file table entry objects, which in turn could each be referenced by multiple file descriptor objects. dup() will create a new descriptor object referencing the same file table entry object. Opening /proc/self/fd/N will actually create a new file table entry object referencing the resource object. So if you ever find yourself actually facing the epoll issue discussed in the article, one hack solution (unverified!) is to create new descriptor references using /proc/self/fd/N rather than dup(). I discovered this distinction when investigating the difference between /dev/fd on BSD and /dev/fd on Linux. On Linux /dev/fd is a symlink to /proc/self/fd, and so opening /dev/fd/N creates a new file table entry object, which can have different access flags and (IIRC) file position cursors from the original descriptor. On BSD /dev/fd/N is analogous to dup() and so will share the same access flags and position cursor. I imagine that someday confusion on these matters will indirectly lead to an application exploit, just because it's so esoteric and involves semantics people rarely consider. Redirecting /dev/fd/N from the shell might result in unintentional sharing of access flags or cursors. Whereas using /dev/fd/N or /proc/self/fd/N on Linux might result in an unintentional lack of sharing if the programmer was expecting dup-like behavior. I forgot what are the semantics of /dev/fd on Solaris or /.vol/DEV#/INO# on macOS. It would pay to check and document your code before relying on them.