8 ms·

Linux Programming: Signals the easy way

184 points by dkarapetyan 9y ago

shmerl 9y ago
That's needed for C/C++. I wonder, how something like Rust for example deal with it? Should one also worry about it by default?
- sigjuice 9y ago
  If you are running as a *nix process, then yes. It doesn't matter if it is Rust, assembly code, ...
- geofft 9y ago
  For the signals that you must handle with a handler (SIGSEGV etc.), Rust doesn't help you. Your code must be "async-signal-safe", that is, everything must work correctly despite being called between two arbitrary opcodes in the host program. Importantly, this means anything that allocates is impermissible, since you might be in the middle of an allocation, and your allocator almost certainly has a per-thread lock and doesn't expect to be called reentrantly on the same thread. And there's no compile-time tracking (e.g., no function type) for functions that don't allocate. You also might be called in the middle of an unsafe block modifying global data, etc. I've considered designing some sort of complicated DSL in Rust to construct async-signal-safe handlers, but it didn't seem worth the complexity. (Although, now that I write this comment out, I think you can handle SIGSEGV safely from another thread as long as your first thread calls for help in an async-signal-safe way, so maybe that would be a library worth writing.) For the rest, using sigwaitinfo or signalfd works just as well in Rust as in C/C++, but not really better or worse. There aren't any good libraries for it (which I keep meaning to do something about), but the underlying POSIX APIs are usable equally well in all three languages. As for whether you should worry about it? Try not to use signals, regardless of language. If you're writing some sort of demand-paging system using SIGSEGV handlers, I hope you know what you're doing already. If you're thinking about SIGALRM, there are other ways to get a notification after a fixed time (and Rust's mio/tokio, for instance, gets you timers in your event loop). If you're trying to use SIGUSR1/SIGUSR2, please rethink your design - you'll be much happier with something like a UNIX socket listening for connections or a localhost-only URL for debugging or whatever. If you want to handle SIGINT (intercept Ctrl-C at a terminal), SIGQUIT (intercept Ctrl-\ at a terminal), SIGTSTP (intercept Ctrl-Z at a terminal), SIGTERM (intercept the "kill" command's default signal), SIGHUP (do something when the terminal exits), SIGWINCH (get notified when the terminal resizes), or SIGCHLD (get notified when a child process exits), you do have to use signals, but all of these can be run through sigwaitinfo or signalfd. And you don't have to actively worry about them if you don't want to customize their behavior, or if (for SIGINT, SIGHUP, etc.) you just want to ignore them and not do anything special. The only odd case is SIGPIPE, and Rust's standard library blocks SIGPIPE for you and returns EPIPE from a system call instead, and unblocks it in child processes, so you don't need to worry about it. (The default SIGPIPE behavior is basically legacy compatibility with existing C code from the very early days of UNIX.)
  - 0x0 9y ago
    Couldn't a non-libc "standard library" for a hypotethical language be written to be re-entrant safe so dealing with signals would be less painful? For example having a separate malloc-like heap for allocs inside signal handlers etc?
    - gpderetta 9y ago
      mmap is sometimes used as an async signal safe allocation function. IIRC mmap async signal safety is not guaranteed by POSIX but it often is in practice on many platforms. It is not cheap though. edit: I should have read ben0x539 comment which suggests the same thing.
    - geofft 9y ago
      Signal handlers can interrupt other signal handlers. But you probably want to avoid this feature, lest you go mad, so let's ignore it. Yes, you could write a library for use inside signal handlers that shares no state with the actual libc. Then you could also safely printf, for instance, although you'd jump ahead of any printfs in the host program that are still buffered, etc. But it's not just libc's globals that are a problem, it's your own globals and those of any other libraries, and if you intend to do nontrivial work in the signal handler (such that having a compiler check that you're async-signal-safe is worth doing), you probably are actually trying to share state with the main program. For instance, say you're installing SEGV handler to virtually map a large data file into memory, fetching it from some remote file storage API as needed. You want access to some data structures from the main program to figure out what to map, probably the ability to make HTTPS connections, etc. And you have to do that all from the signal handler, because whatever pointer segfaulted has to be working by the time you return from your signal handler. You might be able to do this with Rust plus cleverness, by putting the signal handler in a separate statically-linked crate and using the ownership system to track the fact that the signal handler needs read-only access to some data. (Or with dlmopen, in C.) But it would still be very complex.
  - scottlamb 9y ago
    > For the rest, using sigwaitinfo or signalfd works just as well in Rust as in C/C++, but not really better or worse. There aren't any good libraries for it (which I keep meaning to do something about), but the underlying POSIX APIs are usable equally well in all three languages. There are a couple crates which should suffice for basic SIGTERM/SIGHUP/SIGINT handling: https://github.com/BurntSushi/chan-signal https://github.com/BurntSushi/chan-signal creates a sigwait thread which sends signals over a channel. https://github.com/alexcrichton/tokio-signal https://github.com/alexcrichton/tokio-signal uses the self-pipe trick to integrate with a tokio event loop.
- AdieuToLogic 9y ago
  POSIX signal handling[0] transcends what programming language is used, as it is a process/OS level concern. Regarding Rust specifically, it would seem signal handling remains an open item[1] though some degree of support appears present. 0 - http://man7.org/linux/man-pages/man7/signal.7.html http://man7.org/linux/man-pages/man7/signal.7.html 1 - https://github.com/rust-lang/rfcs/issues/1368 https://github.com/rust-lang/rfcs/issues/1368
- dpc_pw 9y ago
  There's a crate for that. `nix` crate has `signalfd` support. I have recently used it to implement terminal resize in TUI text editor, all in one event loop (`mio` crate).
bigger_cheese 9y ago
Is Signalfd viable at all (http://man7.org/linux/man-pages/man2/signalfd.2.html http://man7.org/linux/man-pages/man2/signalfd.2.html) supposedly it was created as a better alternative i.e you can poll for signals from it. Has anyone tried using it?
- oconnor663 9y ago
  The "signalfd is useless" article comes up every so often. The title might be an exaggeration, but there are problems to be aware of: https://ldpreload.com/blog/signalfd-is-useless https://ldpreload.com/blog/signalfd-is-useless In fact it looks like geofft is in this thread below :)
  - catern 9y ago
    The article can be boiled down to the last sentence, which retracts the rest of the article: >So my claim that there’s a better alternative to signalfd is wrong, but it’s still true that signalfd doesn’t solve any of the other troubles with signal handling: it just saves you a separate thread. Of course this sentence is disingenuous. Avoiding a separate thread is a huge gain. Dealing with concurrency is one of the main problems with signals. So yes, signalfd is absolutely hugely useful, allows for a lot of simplification, and should be the first tool you reach for if you want to handle signals on Linux and don't need portability.
    - charles-salvia 9y ago
      Basically, the main problem with signalfd is a problem with the Linux implementation of signals in general: if your program requires some kind of logic where each individual signal delivery somehow "counts" in some way, you may run into a problem because the kernel may coalesce/collapse multiple identical signals delivered consecutively into a single signal delivery. So if your program has some kind of logic that relies on responding to each individual signal, you may have a problem. (Note that this problem does not apply to POSIX real time signals.) But as far as I know, the only reason you'd ever care to actually count individual signal delivery events is with SIGCHLD. So when waiting for SIGCHLD, using signalfd is not sufficient. You'd need to also use one of the wait/waitid/waitpid variants to ensure that you can respond appropriately to each individual SIGCHLD delivered.
RcouF1uZ4gsC 9y ago
Boost ASIO makes signal handling pretty painless in C++. It handles the signal and calls a callback in which you can use any function, not just signal safe functions. http://www.boost.org/doc/libs/1_64_0/doc/html/boost_asio/reference/signal_set.html http://www.boost.org/doc/libs/1_64_0/doc/html/boost_asio/ref...
peter_retief 9y ago
I needed to write something that needed signal processing a few years ago, its very difficult to find simple help on the subject. Anyway, nice take, I will bookmark for future reference ;)
jwilk 9y ago
> Some signals will be specific threads anyway. Huh?
- peterbotond 9y ago
  Yes, I would like more details on this also. ++huh?
- bleeep 9y ago
  If I do "kill -HUP <pid>", the SIGHUP is delivered to a process. Each thread in the process has a separate mask of signals its willing to handle. One of those threads that have SIGHUP unmasked will have the signal delivered to it. If a thread dereferences a null pointer, the sigsegv is only relevant to the thread that dereffed the NULL pointer - that's the only thing that can do anything with it. It makes no sense to send that signal to another thread - it's a signal generated synchronously by the thread itself, rather than asynchronously from anywhere else.
- gpderetta 9y ago
  A word is probably missing: > Some signals will be specific to threads anyway. I.e. some signals are not sent to the process but to specifc threads. See 'bleep' reply for details.
exDM69 9y ago
Can someone explain the use of "volatile int running" here? I'm not sure it's correct but it might work here because the only place it's where it's read from without locking is the main thread (`while(cond->running)`), which is also the only place where it's written to (with locks this time). I'd err on the side of caution here and go with the rule of thumb "volatile is almost always wrong". The slightest change of the code (e.g. allowing another thread to terminate the app) will cause it to fail. It's not much effort to change it to __atomic_load/__atomic_store and remove the volatile qualifier or lock/unlock the mutex every time it's being accessed (costs about 50 nanoseconds).
- peterbotond 9y ago
  volatile just means to keep updating the memory and not keep a copy in the cpu cache.
  - bleeep 9y ago
    That's wildly inaccurate. It's a language level feature that indicates values can change in ways other than what's suggested by the code that manipulate them.
- gpderetta 9y ago
  volatile is usually neither necessary nor sufficient for concurrency as it doesn't have correct semantics. In some simple notification cases (just a polled exit flag for example), given a few assumptions about the compiler and machine model, volatile might have worked. But C11 has _Atomic and there is no excuse for volatile. In the specific example given in the article, not only volatile, but even _Atomic would be completely superfluous. 'running' it is set once in main before any threads is spawned so there is no concurrency issues; after that every read and write access is done under the protection of the 'lock' mutex, so there is no need for any special qualification in POSIX or C11. Interestingly, a 'volatile sig_atomic_t' used in a signal handler is both required and sufficient to communicate with the thread of execution interrupted by a signal handler and is the only portable use of the keyword. edit: there is another portable use in conjunction with {sig,}{set,long}jmp of course.
pif 9y ago
> There are developers out there that try to catch signals like SIGKILL, SIGABRT, SIGSTOP. This is a pointless attempt the signal will NEVER be delivered to the program. Partially false: SIGABRT can be caught. Indeed, that's one of the methods used by the abort() function to stop: https://github.com/lattera/glibc/blob/master/stdlib/abort.c https://github.com/lattera/glibc/blob/master/stdlib/abort.c
gpderetta 9y ago
> The worst offender I have seen was somebody trying to take a pthread lock inside the signal handler. Then trying to fix the "deadlock" by making the lock recursive!! This was so that a pthread_cond_signal could be sent to get the application to exit. we must have been looking at the same codebase... for posterity, the correct solution is to use a posix semaphore which is async signal safe.

184 points by dkarapetyan 9y ago

That's needed for C/C++. I wonder, how something like Rust for example deal with it? Should one also worry about it by default?

sigjuice 9y ago
If you are running as a *nix process, then yes. It doesn't matter if it is Rust, assembly code, ...
geofft 9y ago
For the signals that you must handle with a handler (SIGSEGV etc.), Rust doesn't help you. Your code must be "async-signal-safe", that is, everything must work correctly despite being called between two arbitrary opcodes in the host program. Importantly, this means anything that allocates is impermissible, since you might be in the middle of an allocation, and your allocator almost certainly has a per-thread lock and doesn't expect to be called reentrantly on the same thread. And there's no compile-time tracking (e.g., no function type) for functions that don't allocate. You also might be called in the middle of an unsafe block modifying global data, etc. I've considered designing some sort of complicated DSL in Rust to construct async-signal-safe handlers, but it didn't seem worth the complexity. (Although, now that I write this comment out, I think you can handle SIGSEGV safely from another thread as long as your first thread calls for help in an async-signal-safe way, so maybe that would be a library worth writing.) For the rest, using sigwaitinfo or signalfd works just as well in Rust as in C/C++, but not really better or worse. There aren't any good libraries for it (which I keep meaning to do something about), but the underlying POSIX APIs are usable equally well in all three languages. As for whether you should worry about it? Try not to use signals, regardless of language. If you're writing some sort of demand-paging system using SIGSEGV handlers, I hope you know what you're doing already. If you're thinking about SIGALRM, there are other ways to get a notification after a fixed time (and Rust's mio/tokio, for instance, gets you timers in your event loop). If you're trying to use SIGUSR1/SIGUSR2, please rethink your design - you'll be much happier with something like a UNIX socket listening for connections or a localhost-only URL for debugging or whatever. If you want to handle SIGINT (intercept Ctrl-C at a terminal), SIGQUIT (intercept Ctrl-\ at a terminal), SIGTSTP (intercept Ctrl-Z at a terminal), SIGTERM (intercept the "kill" command's default signal), SIGHUP (do something when the terminal exits), SIGWINCH (get notified when the terminal resizes), or SIGCHLD (get notified when a child process exits), you do have to use signals, but all of these can be run through sigwaitinfo or signalfd. And you don't have to actively worry about them if you don't want to customize their behavior, or if (for SIGINT, SIGHUP, etc.) you just want to ignore them and not do anything special. The only odd case is SIGPIPE, and Rust's standard library blocks SIGPIPE for you and returns EPIPE from a system call instead, and unblocks it in child processes, so you don't need to worry about it. (The default SIGPIPE behavior is basically legacy compatibility with existing C code from the very early days of UNIX.)
- 0x0 9y ago
  Couldn't a non-libc "standard library" for a hypotethical language be written to be re-entrant safe so dealing with signals would be less painful? For example having a separate malloc-like heap for allocs inside signal handlers etc?
  - gpderetta 9y ago
    mmap is sometimes used as an async signal safe allocation function. IIRC mmap async signal safety is not guaranteed by POSIX but it often is in practice on many platforms. It is not cheap though. edit: I should have read ben0x539 comment which suggests the same thing.
  - geofft 9y ago
    Signal handlers can interrupt other signal handlers. But you probably want to avoid this feature, lest you go mad, so let's ignore it. Yes, you could write a library for use inside signal handlers that shares no state with the actual libc. Then you could also safely printf, for instance, although you'd jump ahead of any printfs in the host program that are still buffered, etc. But it's not just libc's globals that are a problem, it's your own globals and those of any other libraries, and if you intend to do nontrivial work in the signal handler (such that having a compiler check that you're async-signal-safe is worth doing), you probably are actually trying to share state with the main program. For instance, say you're installing SEGV handler to virtually map a large data file into memory, fetching it from some remote file storage API as needed. You want access to some data structures from the main program to figure out what to map, probably the ability to make HTTPS connections, etc. And you have to do that all from the signal handler, because whatever pointer segfaulted has to be working by the time you return from your signal handler. You might be able to do this with Rust plus cleverness, by putting the signal handler in a separate statically-linked crate and using the ownership system to track the fact that the signal handler needs read-only access to some data. (Or with dlmopen, in C.) But it would still be very complex.
- scottlamb 9y ago
  > For the rest, using sigwaitinfo or signalfd works just as well in Rust as in C/C++, but not really better or worse. There aren't any good libraries for it (which I keep meaning to do something about), but the underlying POSIX APIs are usable equally well in all three languages. There are a couple crates which should suffice for basic SIGTERM/SIGHUP/SIGINT handling: https://github.com/BurntSushi/chan-signal https://github.com/BurntSushi/chan-signal creates a sigwait thread which sends signals over a channel. https://github.com/alexcrichton/tokio-signal https://github.com/alexcrichton/tokio-signal uses the self-pipe trick to integrate with a tokio event loop.
AdieuToLogic 9y ago
POSIX signal handling[0] transcends what programming language is used, as it is a process/OS level concern. Regarding Rust specifically, it would seem signal handling remains an open item[1] though some degree of support appears present. 0 - http://man7.org/linux/man-pages/man7/signal.7.html http://man7.org/linux/man-pages/man7/signal.7.html 1 - https://github.com/rust-lang/rfcs/issues/1368 https://github.com/rust-lang/rfcs/issues/1368
dpc_pw 9y ago
There's a crate for that. `nix` crate has `signalfd` support. I have recently used it to implement terminal resize in TUI text editor, all in one event loop (`mio` crate).

bigger_cheese 9y ago

Is Signalfd viable at all (http://man7.org/linux/man-pages/man2/signalfd.2.html http://man7.org/linux/man-pages/man2/signalfd.2.html) supposedly it was created as a better alternative i.e you can poll for signals from it. Has anyone tried using it?

oconnor663 9y ago
The "signalfd is useless" article comes up every so often. The title might be an exaggeration, but there are problems to be aware of: https://ldpreload.com/blog/signalfd-is-useless https://ldpreload.com/blog/signalfd-is-useless In fact it looks like geofft is in this thread below :)
- catern 9y ago
  The article can be boiled down to the last sentence, which retracts the rest of the article: >So my claim that there’s a better alternative to signalfd is wrong, but it’s still true that signalfd doesn’t solve any of the other troubles with signal handling: it just saves you a separate thread. Of course this sentence is disingenuous. Avoiding a separate thread is a huge gain. Dealing with concurrency is one of the main problems with signals. So yes, signalfd is absolutely hugely useful, allows for a lot of simplification, and should be the first tool you reach for if you want to handle signals on Linux and don't need portability.
  - charles-salvia 9y ago
    Basically, the main problem with signalfd is a problem with the Linux implementation of signals in general: if your program requires some kind of logic where each individual signal delivery somehow "counts" in some way, you may run into a problem because the kernel may coalesce/collapse multiple identical signals delivered consecutively into a single signal delivery. So if your program has some kind of logic that relies on responding to each individual signal, you may have a problem. (Note that this problem does not apply to POSIX real time signals.) But as far as I know, the only reason you'd ever care to actually count individual signal delivery events is with SIGCHLD. So when waiting for SIGCHLD, using signalfd is not sufficient. You'd need to also use one of the wait/waitid/waitpid variants to ensure that you can respond appropriately to each individual SIGCHLD delivered.

RcouF1uZ4gsC 9y ago

Boost ASIO makes signal handling pretty painless in C++. It handles the signal and calls a callback in which you can use any function, not just signal safe functions. http://www.boost.org/doc/libs/1_64_0/doc/html/boost_asio/reference/signal_set.html http://www.boost.org/doc/libs/1_64_0/doc/html/boost_asio/ref...

peter_retief 9y ago

I needed to write something that needed signal processing a few years ago, its very difficult to find simple help on the subject. Anyway, nice take, I will bookmark for future reference ;)

jwilk 9y ago

> Some signals will be specific threads anyway. Huh?

peterbotond 9y ago
Yes, I would like more details on this also. ++huh?
bleeep 9y ago
If I do "kill -HUP <pid>", the SIGHUP is delivered to a process. Each thread in the process has a separate mask of signals its willing to handle. One of those threads that have SIGHUP unmasked will have the signal delivered to it. If a thread dereferences a null pointer, the sigsegv is only relevant to the thread that dereffed the NULL pointer - that's the only thing that can do anything with it. It makes no sense to send that signal to another thread - it's a signal generated synchronously by the thread itself, rather than asynchronously from anywhere else.
gpderetta 9y ago
A word is probably missing: > Some signals will be specific to threads anyway. I.e. some signals are not sent to the process but to specifc threads. See 'bleep' reply for details.

exDM69 9y ago

Can someone explain the use of "volatile int running" here? I'm not sure it's correct but it might work here because the only place it's where it's read from without locking is the main thread (`while(cond->running)`), which is also the only place where it's written to (with locks this time). I'd err on the side of caution here and go with the rule of thumb "volatile is almost always wrong". The slightest change of the code (e.g. allowing another thread to terminate the app) will cause it to fail. It's not much effort to change it to __atomic_load/__atomic_store and remove the volatile qualifier or lock/unlock the mutex every time it's being accessed (costs about 50 nanoseconds).

peterbotond 9y ago
volatile just means to keep updating the memory and not keep a copy in the cpu cache.
- bleeep 9y ago
  That's wildly inaccurate. It's a language level feature that indicates values can change in ways other than what's suggested by the code that manipulate them.
gpderetta 9y ago
volatile is usually neither necessary nor sufficient for concurrency as it doesn't have correct semantics. In some simple notification cases (just a polled exit flag for example), given a few assumptions about the compiler and machine model, volatile might have worked. But C11 has _Atomic and there is no excuse for volatile. In the specific example given in the article, not only volatile, but even _Atomic would be completely superfluous. 'running' it is set once in main before any threads is spawned so there is no concurrency issues; after that every read and write access is done under the protection of the 'lock' mutex, so there is no need for any special qualification in POSIX or C11. Interestingly, a 'volatile sig_atomic_t' used in a signal handler is both required and sufficient to communicate with the thread of execution interrupted by a signal handler and is the only portable use of the keyword. edit: there is another portable use in conjunction with {sig,}{set,long}jmp of course.

pif 9y ago

> There are developers out there that try to catch signals like SIGKILL, SIGABRT, SIGSTOP. This is a pointless attempt the signal will NEVER be delivered to the program. Partially false: SIGABRT can be caught. Indeed, that's one of the methods used by the abort() function to stop: https://github.com/lattera/glibc/blob/master/stdlib/abort.c https://github.com/lattera/glibc/blob/master/stdlib/abort.c

gpderetta 9y ago

> The worst offender I have seen was somebody trying to take a pthread lock inside the signal handler. Then trying to fix the "deadlock" by making the lock recursive!! This was so that a pthread_cond_signal could be sent to get the application to exit. we must have been looking at the same codebase... for posterity, the correct solution is to use a posix semaphore which is async signal safe.