4 ms·

sd-event was announced to be the next systemd-private interface turned public after sd-bus a while ago, and here we are. If anything, it only further convinces

sd-event was announced to be the next systemd-private interface turned public after sd-bus a while ago, and here we are. If anything, it only further convinces me what a leaky abstraction epoll(7) is, plus all the myriad of *fd calls that really just canonize things like the self-pipe trick in the completely wrong layers. I'd probably use something like libkqueue if I wanted a minimal event loop.

bcantrill 11y ago
Agreed -- implementing epoll(7) for LX-branded zones on SmartOS[1] was a bit of a shock in terms of how unbelievably broken the abstraction is. I had assumed (ha!) that epoll(7) would have been inspired by the prior art: I/O completion ports on Windows, kqueue on the BSDs, and event ports on Solaris. Instead, despite being the last to the party, epoll(7) is but a more efficient variant of poll(2) -- retaining the broken-by-design aspects of poll(2) with respect to multi-threaded programs. (For details on these conditions, consult the strange Q&A in the epoll(7) man page[2] or the aborted attempt to rectify it with EPOLL_CTL_DISABLE[3].) I suppose that sd-event may represent an improvement in that it attempts to patch some of the leaks in the abstraction -- but I (certainly) agree that kqueue represents a much better design, and to the degree that libkqueue is able to provide that abstraction on Linux, a much better alternative. [1] http://www.slideshare.net/bcantrill/illumos-lx http://www.slideshare.net/bcantrill/illumos-lx [2] http://man7.org/linux/man-pages/man7/epoll.7.html http://man7.org/linux/man-pages/man7/epoll.7.html [3] https://lwn.net/Articles/520198/ https://lwn.net/Articles/520198/
- protomyth 11y ago
  Your BSD Now interview[1] where you talk about epoll/kthread is informative and entertaining. The whole interview is worth watching. 1) http://www.bsdnow.tv/episodes/2015_08_19-ubuntu_slaughters_kittens http://www.bsdnow.tv/episodes/2015_08_19-ubuntu_slaughters_k... around 55:59
  - twoodfin 11y ago
    The story about restricting rm -rf / without violating the letter of the standard is also great stuff.
    - protomyth 11y ago
      That's probably a good question to ask about any piece of software or standard: "Be honest, Is there a way that your software or software written to the specification could render the computer unusable for someone not trying to render the computer unusable?"
- kbwt 11y ago
  A major problem with sd-event as an epoll replacement lies in how it handles multi-threading: It doesn't. The most error-prone part of using epoll in a multi-threaded environment is free-ing of the per-descriptor state. You never quite know when the last epoll_wait that still knew about that descriptor has fired. There are multiple approaches for dealing with this problem, including RCU-like waiting for each thread to pass epoll_wait once, recycling descriptor states by including an anti-race generation counter in the unused bits of the event user data, or garbage collection. In contrast, with IOCP or other completion-based programming models, you can reference count the active I/O operations and free once the count reaches zero.
osivertsson 11y ago
Do you, or anyone else, have any experience from libkqueue? After a quick glance I'm not convinced that it is battle-tested enough, and the Android port is not building, Solaris port not really working, etc.
- JdeBP 11y ago
  kqueue and kevent themselves are available across all of the BSDs; in FreeBSD, OpenBSD, NetBSD, and their various derivatives including Darwin and others. * https://www.freebsd.org/cgi/man.cgi?query=kevent https://www.freebsd.org/cgi/man.cgi?query=kevent * http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/kevent.2 http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/... * http://man.netbsd.org/HEAD/usr/share/man/html2/kevent.html http://man.netbsd.org/HEAD/usr/share/man/html2/kevent.html * https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/kevent.2.html https://developer.apple.com/library/mac/documentation/Darwin... This most definitely has been "battle tested" and it long since has provided a means for programs to use a single mechanism to wait for I/O events, directory modifications, signals, child processes, and timers. And it has some interesting abilities, too, such as the ability to keep track of grand-child processes forked off by a child, which one can use to good advantage to handle poorly written daemons that erroneously fork-child-then-exit-parent. libkqueue should not be conflated with this. It provides a workalike API. But there are differences. Some of these are of necessity. Because of the way that Linux signalfds work, for starters, it has been quietly re-specified (the alteration to the wording of the manual page being a subtle one that isn't marked in any way) so that EVFILT_SIGNAL code has to be written differently when using libkqueue on Linux to when using real kevent/kqueue on a BSD. Others are accidents of implementation. One can get spurious wakeups from libkqueue that one doesn't get from real BSD kevent(). Its Linux portion also contains a number of bugs. Internal unexposed file descriptors, for all of the signalfds, timerfds, epolls, inotifys, and whatnot that it opens, are not marked close-on-exec, meaning that there's a significant security concern in any program that uses libkqueue in a privileged process and then forks and executes unprivileged children. The implementation of EVFILT_VNODE/NOTE_WRITE for directories is simply wrong. There are pitfalls when using EVFILT_READ with terminal devices. And its inotify usage doesn't account for the fact that notifications are not always the same number of bytes long. The first two of those bugs are fairly easy to fix. I never got to the bottom of the third. The fourth is more complex, but is addressable I think. The main problem is not fixing these bugs, but getting those fixes available across a multiplicity of Linux distributions, some of which are operating from years-old versions of the library. So even though I wanted (and still want as a matter of fact) to get libkqueue made better, I had to go with dropping libkqueue and implementing my own kqueue()/kevent() workalikes for Linux. * https://www.mail-archive.com/supervision@list.skarnet.org/msg00830.html https://www.mail-archive.com/supervision@list.skarnet.org/ms... * https://www.mail-archive.com/supervision@list.skarnet.org/msg00840.html https://www.mail-archive.com/supervision@list.skarnet.org/ms... * https://www.mail-archive.com/supervision@list.skarnet.org/msg01043.html https://www.mail-archive.com/supervision@list.skarnet.org/ms...
JoshTriplett 11y ago
> it only further convinces me what a leaky abstraction epoll(7) is epoll isn't an abstraction at all. It was never designed to be. epoll was designed as a far more scalable replacement for poll and select.
- vezzy-fnord 11y ago
  That's a very baffling thing to say. Just what is a file descriptor to begin with but an abstraction? And what is a mechanism to multiplex on file descriptors other than an abstraction?
  - JoshTriplett 11y ago
    epoll isn't designed to hide anything from you. If anything, it's designed to expose more of the low-level details to allow building a more scalable event loop. For instance, see the whole edge-triggered versus level-triggered bit, a distinction wholly unavailable in poll or select. In particular, epoll wasn't designed to be a "higher-level" event loop than poll or select; while more powerful, it's arguably lower-level. Hence my comment that it isn't an abstraction.

bcantrill 11y ago

Agreed -- implementing epoll(7) for LX-branded zones on SmartOS[1] was a bit of a shock in terms of how unbelievably broken the abstraction is. I had assumed (ha!) that epoll(7) would have been inspired by the prior art: I/O completion ports on Windows, kqueue on the BSDs, and event ports on Solaris. Instead, despite being the last to the party, epoll(7) is but a more efficient variant of poll(2) -- retaining the broken-by-design aspects of poll(2) with respect to multi-threaded programs. (For details on these conditions, consult the strange Q&A in the epoll(7) man page[2] or the aborted attempt to rectify it with EPOLL_CTL_DISABLE[3].) I suppose that sd-event may represent an improvement in that it attempts to patch some of the leaks in the abstraction -- but I (certainly) agree that kqueue represents a much better design, and to the degree that libkqueue is able to provide that abstraction on Linux, a much better alternative. [1] http://www.slideshare.net/bcantrill/illumos-lx http://www.slideshare.net/bcantrill/illumos-lx [2] http://man7.org/linux/man-pages/man7/epoll.7.html http://man7.org/linux/man-pages/man7/epoll.7.html [3] https://lwn.net/Articles/520198/ https://lwn.net/Articles/520198/

protomyth 11y ago
Your BSD Now interview[1] where you talk about epoll/kthread is informative and entertaining. The whole interview is worth watching. 1) http://www.bsdnow.tv/episodes/2015_08_19-ubuntu_slaughters_kittens http://www.bsdnow.tv/episodes/2015_08_19-ubuntu_slaughters_k... around 55:59
- twoodfin 11y ago
  The story about restricting rm -rf / without violating the letter of the standard is also great stuff.
  - protomyth 11y ago
    That's probably a good question to ask about any piece of software or standard: "Be honest, Is there a way that your software or software written to the specification could render the computer unusable for someone not trying to render the computer unusable?"
kbwt 11y ago
A major problem with sd-event as an epoll replacement lies in how it handles multi-threading: It doesn't. The most error-prone part of using epoll in a multi-threaded environment is free-ing of the per-descriptor state. You never quite know when the last epoll_wait that still knew about that descriptor has fired. There are multiple approaches for dealing with this problem, including RCU-like waiting for each thread to pass epoll_wait once, recycling descriptor states by including an anti-race generation counter in the unused bits of the event user data, or garbage collection. In contrast, with IOCP or other completion-based programming models, you can reference count the active I/O operations and free once the count reaches zero.

osivertsson 11y ago

Do you, or anyone else, have any experience from libkqueue? After a quick glance I'm not convinced that it is battle-tested enough, and the Android port is not building, Solaris port not really working, etc.

JdeBP 11y ago
kqueue and kevent themselves are available across all of the BSDs; in FreeBSD, OpenBSD, NetBSD, and their various derivatives including Darwin and others. * https://www.freebsd.org/cgi/man.cgi?query=kevent https://www.freebsd.org/cgi/man.cgi?query=kevent * http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/kevent.2 http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/... * http://man.netbsd.org/HEAD/usr/share/man/html2/kevent.html http://man.netbsd.org/HEAD/usr/share/man/html2/kevent.html * https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/kevent.2.html https://developer.apple.com/library/mac/documentation/Darwin... This most definitely has been "battle tested" and it long since has provided a means for programs to use a single mechanism to wait for I/O events, directory modifications, signals, child processes, and timers. And it has some interesting abilities, too, such as the ability to keep track of grand-child processes forked off by a child, which one can use to good advantage to handle poorly written daemons that erroneously fork-child-then-exit-parent. libkqueue should not be conflated with this. It provides a workalike API. But there are differences. Some of these are of necessity. Because of the way that Linux signalfds work, for starters, it has been quietly re-specified (the alteration to the wording of the manual page being a subtle one that isn't marked in any way) so that EVFILT_SIGNAL code has to be written differently when using libkqueue on Linux to when using real kevent/kqueue on a BSD. Others are accidents of implementation. One can get spurious wakeups from libkqueue that one doesn't get from real BSD kevent(). Its Linux portion also contains a number of bugs. Internal unexposed file descriptors, for all of the signalfds, timerfds, epolls, inotifys, and whatnot that it opens, are not marked close-on-exec, meaning that there's a significant security concern in any program that uses libkqueue in a privileged process and then forks and executes unprivileged children. The implementation of EVFILT_VNODE/NOTE_WRITE for directories is simply wrong. There are pitfalls when using EVFILT_READ with terminal devices. And its inotify usage doesn't account for the fact that notifications are not always the same number of bytes long. The first two of those bugs are fairly easy to fix. I never got to the bottom of the third. The fourth is more complex, but is addressable I think. The main problem is not fixing these bugs, but getting those fixes available across a multiplicity of Linux distributions, some of which are operating from years-old versions of the library. So even though I wanted (and still want as a matter of fact) to get libkqueue made better, I had to go with dropping libkqueue and implementing my own kqueue()/kevent() workalikes for Linux. * https://www.mail-archive.com/supervision@list.skarnet.org/msg00830.html https://www.mail-archive.com/supervision@list.skarnet.org/ms... * https://www.mail-archive.com/supervision@list.skarnet.org/msg00840.html https://www.mail-archive.com/supervision@list.skarnet.org/ms... * https://www.mail-archive.com/supervision@list.skarnet.org/msg01043.html https://www.mail-archive.com/supervision@list.skarnet.org/ms...

JoshTriplett 11y ago

> it only further convinces me what a leaky abstraction epoll(7) is epoll isn't an abstraction at all. It was never designed to be. epoll was designed as a far more scalable replacement for poll and select.

vezzy-fnord 11y ago
That's a very baffling thing to say. Just what is a file descriptor to begin with but an abstraction? And what is a mechanism to multiplex on file descriptors other than an abstraction?
- JoshTriplett 11y ago
  epoll isn't designed to hide anything from you. If anything, it's designed to expose more of the low-level details to allow building a more scalable event loop. For instance, see the whole edge-triggered versus level-triggered bit, a distinction wholly unavailable in poll or select. In particular, epoll wasn't designed to be a "higher-level" event loop than poll or select; while more powerful, it's arguably lower-level. Hence my comment that it isn't an abstraction.