6 ms·

I highly recommend that you do NOT use signalfd to get notification of signals through epoll. Instead, block (mask) the signal, set a signal handler, and use ep

I highly recommend that you do NOT use signalfd to get notification of signals through epoll. Instead, block (mask) the signal, set a signal handler, and use epoll_pwait() to atomically unblock it while you wait for events. Note that in this setup, your signal handler callback need not be async-signal-safe, since you know the precise state of the calling thread: it's invoking epoll_pwait(). This sidesteps most of the pain of using signals which might otherwise make you think you want signalfd. Two reasons not to use signalfd: 1. signalfd has weird semantics that don't match what you'd normally expect from a file descriptor. When you read from a signalfd, it tells you signals queued on the thread that called read(), NOT the thread that created the signalfd. Worse, if you add signalfd to an epoll, the epoll will report readiness based on the thread that used epoll_ctl() to add the signalfd, which may be different from the thread that is reading from the epoll. So you might get a notification that the signalfd is ready, but then read the signalfd and find there are no signals, and then wait on the epoll again just to have it tell you again that this signalfd is ready. 2. It turns out that signalfd's implementation has some severe lock contention issues. I learned this through my own experimentation recently. In my experiment, I had 5000 threads each waiting on an epoll that included a signalfd. When delivering a thread-specific signal to each of the 5000 threads at once, the process spent 2+ MINUTES of CPU time spinning on spinlocks in the kernel before completing all the event deliveries. The time spent was O(n^2) to the number of threads. When I switched to an epoll_pwait()-based implementation, the same task took a few milliseconds. Here's the PR where I switched KJ's event loop (used in Cap'n Proto and Cloudflare Workers) to use epoll_pwait(): https://github.com/capnproto/capnproto/pull/1511 https://github.com/capnproto/capnproto/pull/1511

kelnos 4y ago
The big downside of using a traditional signal handler is that the only way to get your own data into the handler function is through global variables (or thread locals). While you can certainly make an exception just for that one thing, it feels gross to do so. And you can also just defer processing to your main loop by setting a flag or writing to a pipe, but those things still need to be global variables. I didn't know about signalfd's limitations before reading your post, and was happy that signalfd could eliminate the need for global variables when doing signal handling. Shame that's not really the case.
- kentonv 4y ago
  In my case I use a thread_local pointer that I initialize right before epoll_pwait and set back to null immediately after. The pointer points to the same data structures that I would otherwise use to handle signalfd events. Yeah it's a little icky to use the global but I think it ends up semantically equivalent.
  - kelnos 4y ago
    Makes sense, and is probably the "safest" you can get. Since, as you say, you know exactly the state of everything on that thread when you're in your handler, you can also know that your thread local was set properly before the epoll_pwait() call. It's probably code I'd want to isolate somewhere, with big warnings so any future reader understands why it is how it is, but I agree it's probably the safest way to do it.
  - deleted 4y ago
    [deleted]
  - wahern 4y ago
    Unfortunately, thread-local storage is not async-signal safe. You're relying (knowingly, I presume, but others should be warned) on implementation details. But, yeah, signalfd leaves much to be desired. *BSD kqueue EVFILT_SIGNAL has much saner semantics.
    - kentonv 4y ago
      > Unfortunately, thread-local storage is not async-signal safe. Doesn't matter, because the signal handler in this case is strictly called "during" invocation of epoll_pwait, so there's no risk of it interrupting the initialization of a TLS object. The usual rules about async signal safety do not need to be followed here; it's as if epoll_wait()'s implementation made a plain old function call to the signal handler. (Also, since we're talking about epoll, we can assume Linux, which means we can assume ELF, which means it's pretty easy to use thread_local in a way that requires no initialization by allocating it in the ELF TLS section. But yes, that's relying on implementation details I suppose.) > kqueue EVFILT_SIGNAL Having recently implement kqueue support in my event loop I have to say I'm disappointed by EVFILT_SIGNAL. It does not play well with signals that target a specific thread (pthread_kill()) -- on FreeBSD, all threads will get the kqueue event, while on MacOS, none of them do. Fortunately EVFILT_USER provides a reasonable alternative for efficient inter-thread signaling. (I don't like using a pipe or socketpair as that involves allocating a whole two file descriptors and a kernel buffer, and it requires a redundant syscall on the receiving end to read the dummy byte out of the buffer. If you're just trying to tell another thread "hey I added something to your work queue, please wake up and check", that's a waste.)
tlsalmin 4y ago
I have to disagree here. Not recommending signalfd for the mentioned use cases might be reasonable, just as reasonable as it is to use threads for a specific use case. For a single threaded non-blocking-FD using client/server signalfd removes the risk of doing too much in the signal handler and brings signals nicely into the event loop. This just happens to be 99% of the functionality I have to do. I'd only use more than one signalfd if each signalfd only catches a specific signal. E.g. main context handles Sigterm and a background process library handles sigchld.
- kentonv 4y ago
  > removes the risk of doing too much in the signal handler Not a concern here, since the signal handler is restricted from running at any time other than during epoll_pwait(), so the usual async-signal-safety concerns don't apply. In fact I think the code ends up cleaner than with signalfd, and there are fewer syscalls (no separate read() needed).
  - tlsalmin 4y ago
    This is getting into taste territory but sure just getting the signal in epoll_pwait removes the need for FD reading. But introduces the need for the thread local global context and the need to handle EINTR from epoll right? I can see how checking if signals were called in the EINTR-branch is nice though. Anyway nice to know the p-version use case.
FPGAhacker 4y ago
You should do a write up of item 2.
cryptonector 4y ago
Using a self-pipe (signal handler writes one byte to one end, the other end is used in your event loop) is the most portable and reliable way to handle signals.
- kentonv 4y ago
  Yeah but it involves several extra syscalls, two file descriptors, and a kernel buffer that never has any actual data in it... poll() + longjmp() out of the signal handler is actually surprisingly portable and much more efficient (but breaks all sorts of abstractions in ways that terrify people).
  - cryptonector 4y ago
    The overhead is very limited. The memory overhead is fixed, and the compute overhead is linear with the rate at which signals arrive (which is typically super low). Plus the whole thing is very easy to reason about. And you never have to worry about masking signals that you handle with a self-pipe. Add in the portability, and it's a very clear win.

kelnos 4y ago

The big downside of using a traditional signal handler is that the only way to get your own data into the handler function is through global variables (or thread locals). While you can certainly make an exception just for that one thing, it feels gross to do so. And you can also just defer processing to your main loop by setting a flag or writing to a pipe, but those things still need to be global variables. I didn't know about signalfd's limitations before reading your post, and was happy that signalfd could eliminate the need for global variables when doing signal handling. Shame that's not really the case.

kentonv 4y ago
In my case I use a thread_local pointer that I initialize right before epoll_pwait and set back to null immediately after. The pointer points to the same data structures that I would otherwise use to handle signalfd events. Yeah it's a little icky to use the global but I think it ends up semantically equivalent.
- kelnos 4y ago
  Makes sense, and is probably the "safest" you can get. Since, as you say, you know exactly the state of everything on that thread when you're in your handler, you can also know that your thread local was set properly before the epoll_pwait() call. It's probably code I'd want to isolate somewhere, with big warnings so any future reader understands why it is how it is, but I agree it's probably the safest way to do it.
- deleted 4y ago
  [deleted]
- wahern 4y ago
  Unfortunately, thread-local storage is not async-signal safe. You're relying (knowingly, I presume, but others should be warned) on implementation details. But, yeah, signalfd leaves much to be desired. *BSD kqueue EVFILT_SIGNAL has much saner semantics.
  - kentonv 4y ago
    > Unfortunately, thread-local storage is not async-signal safe. Doesn't matter, because the signal handler in this case is strictly called "during" invocation of epoll_pwait, so there's no risk of it interrupting the initialization of a TLS object. The usual rules about async signal safety do not need to be followed here; it's as if epoll_wait()'s implementation made a plain old function call to the signal handler. (Also, since we're talking about epoll, we can assume Linux, which means we can assume ELF, which means it's pretty easy to use thread_local in a way that requires no initialization by allocating it in the ELF TLS section. But yes, that's relying on implementation details I suppose.) > kqueue EVFILT_SIGNAL Having recently implement kqueue support in my event loop I have to say I'm disappointed by EVFILT_SIGNAL. It does not play well with signals that target a specific thread (pthread_kill()) -- on FreeBSD, all threads will get the kqueue event, while on MacOS, none of them do. Fortunately EVFILT_USER provides a reasonable alternative for efficient inter-thread signaling. (I don't like using a pipe or socketpair as that involves allocating a whole two file descriptors and a kernel buffer, and it requires a redundant syscall on the receiving end to read the dummy byte out of the buffer. If you're just trying to tell another thread "hey I added something to your work queue, please wake up and check", that's a waste.)

tlsalmin 4y ago

I have to disagree here. Not recommending signalfd for the mentioned use cases might be reasonable, just as reasonable as it is to use threads for a specific use case. For a single threaded non-blocking-FD using client/server signalfd removes the risk of doing too much in the signal handler and brings signals nicely into the event loop. This just happens to be 99% of the functionality I have to do. I'd only use more than one signalfd if each signalfd only catches a specific signal. E.g. main context handles Sigterm and a background process library handles sigchld.

kentonv 4y ago
> removes the risk of doing too much in the signal handler Not a concern here, since the signal handler is restricted from running at any time other than during epoll_pwait(), so the usual async-signal-safety concerns don't apply. In fact I think the code ends up cleaner than with signalfd, and there are fewer syscalls (no separate read() needed).
- tlsalmin 4y ago
  This is getting into taste territory but sure just getting the signal in epoll_pwait removes the need for FD reading. But introduces the need for the thread local global context and the need to handle EINTR from epoll right? I can see how checking if signals were called in the EINTR-branch is nice though. Anyway nice to know the p-version use case.

FPGAhacker 4y ago

You should do a write up of item 2.

cryptonector 4y ago

Using a self-pipe (signal handler writes one byte to one end, the other end is used in your event loop) is the most portable and reliable way to handle signals.

kentonv 4y ago
Yeah but it involves several extra syscalls, two file descriptors, and a kernel buffer that never has any actual data in it... poll() + longjmp() out of the signal handler is actually surprisingly portable and much more efficient (but breaks all sorts of abstractions in ways that terrify people).
- cryptonector 4y ago
  The overhead is very limited. The memory overhead is fixed, and the compute overhead is linear with the rate at which signals arrive (which is typically super low). Plus the whole thing is very easy to reason about. And you never have to worry about masking signals that you handle with a self-pipe. Add in the portability, and it's a very clear win.