9 ms·

Should you be scared of Unix signals? (2016)

257 points by gavinhoward 3y ago

gavinhoward 3y ago
While Julia starts scared and ends up feeling better, I've only gotten more scared of Unix signals over time. Context: I've written a robust command-line utility [0] that must handle signals, and unlike most utilities that are I/O-bound, mine is CPU-bound. An I/O-bound utility can easily use signalfd() (subject to the gotchas in the "signalfd() is useless" post that Julia links to, which you should also read). signalfd() will work for I/O-bound utilities because it turns those signals into I/O. Perfect. However, in a CPU-bound program, signals are used specifically to interrupt execution. This is, to put it mildly, as difficult as writing code for interrupts in the embedded space. Why? Because that's really what you're doing: handling an interrupt that can happen at any time. My solution was something I wish on no one: I used setjmp() and longjmp(). Horrors! Yep. And it gets worse: I had to longjmp() out of the signal handler. AH! And it gets worse: to ensure that there were no memory leaks, I had to keep a stack of jmp_bufs and manually jump to each one, which would be in a function where memory had to be cleaned up. Cue screams of bloody murder You may insist that longjmp()'ing out of signal handler is not allowed; it actually is [1], but unlike most other "async-signal-safe" functions, you need to ensure you don't interrupt a syscall or other code that is not async-signal-safe. So it gets worse: I have a signal lock that the signal handler checks. If it's not locked, the signal handler will longjmp() out of the signal handler. If it is locked, the signal handler sets a flag and returns. Then the code that unlocks signals checks for the flag and does a longjmp() if it's set. He's dead, Jim! I have another project that is a framework in C. This framework needs to handle signals for clients. It has to be general, so it has to handle CPU-bound clients. So I had to implement the same thing. I was able to make it easier, but it is also harder because I have that one thing that messes up every Unix API: threads. Nuclear mushroom cloud So should you be scared? It depends; if you can get away with signalfd() and know its gotchas, maybe not. But if you need anything more complex, yes, be very afraid. [0]: https://git.gavinhoward.com/gavin/bc https://git.gavinhoward.com/gavin/bc [1]: https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/ https://pubs.opengroup.org/onlinepubs/9699919799.2008edition...
- fzeindl 3y ago
  Have you ever read up on communicating sequential processes? In that model you have one or multiple threads executing work while new messages arrive on a queue. In such an architecture handling signals is trivial.
  - boffinAudio 3y ago
    Yeah, I was going to mention that if the long-running/CPU-bound work was put in a worker thread with a message queue in which state messages are received/handled, it'd be relatively trivial to handle the signal across these constraints, since signals are only delivered upon context switches. But if you've just got one very tight loop and are running like a bat out of hell, well you deserve the hell you're in. ;) longjmp() is definitely not the right way to do this. A thread dedicated to handling signals (or doing the CPU work) which properly tells the main() thread to die/quit/clean up is how I'd do it .. even if you only spawn a thread for signal handling, its still way cleaner than that longjmp() business .. which is very difficult to understand, even with the context you've given in this thread (I read your code - its nasty) ...
    - gavinhoward 3y ago
      Why do I deserve that for having a CPU-bound program? And why is longjmp() the wrong thing to do?
      - boffinAudio 3y ago
        Because signals are delivered on context switch, and you're not allowing any of that to happen. Its not a well-behaved process in that case .. not that you shouldn't be able to just burn as much CPU as you want, just that you would need to open up some space to communicate with the OS in the meantime, and having a signal-handler-thread or a cpu-working-thread is how to do it properly. (Do you also handle SIGHUP? SIGSTOP?) longjmp() is just asking for hell 6 months later when you come back and have to work out why you did such a hokey thing in the first place. EDIT: it should be noted that the "side-effect" of using longjmp is .. a context switch .. so you gave yourself some space to behave properly with the OS. Its just that the more future-proof way to do it is to have a thread for work (you have this already with main()) and a thread for signals (you'd just add this to service signals only and give main() a way to cleanly exit) ... Not too complex, and a tad bit easier to read 6 months later ..
signa11 3y ago
if you are writing multithreaded programs, and considering dealing with signals, then yes.
wilkystyle 3y ago
I know this has been said before, but I absolutely love Julia's style of writing. It's humble and accessible, and full of great humor that is always good natured, as opposed to the more common snarky or negative humor: > Signals are a way for Unix processes to communicate! Except for SIGKILL. When you get sent SIGKILL nobody communicates with you, you just die immediately.
- epcoa 3y ago
  What about SIGSTOP?
  - dspillett 3y ago
    Also not communication as it can't be ignored and your process just pauses, if you consider it a thing in its own right. Really though it is a pair with SIGCONT which is communication: SIGSTOP happens (which your process doesn't even see) then later SIGCONT happens which you do get notified about (“hey, process, you were just paused for a bit, you might want to reassess your surroundings”). This is little different from the scheduler not giving your process any time for a while, what difference there is being the while could be a lot longer and you get explicitly told that it has happened. The other possibility is that your process never wakes up because a SIGKILL happens first. Or maybe SIGTERM, I'm not sure whether or not a SIGCONT will be sent first in that instance. There is also the friendlier SIGSTSP which can be handled (or even ignored) but is otherwise identical to SIGSTOP. This is what you get sent if a user hits ctrl+z in most interactive shells (or a more polite process than one that just sends SIGSTOP wants you to pause).
    - epcoa 3y ago
      > if you consider it a thing in its own right. There is no guarantee you’re getting a SIGCONT, so it is, and you could have stopped there. Furthermore, they aren’t paired in any meaningful way. You can get a SIGCONT without any STOP (you can just do this with kill)
  - deleted 3y ago
    [deleted]
dale_glass 3y ago
For once, Betteridge's Law of Headlines doesn't hold. Signals suck and are best avoided. When handling the concept in a vacuum, like writing a simple program from scratch that uses them, sure, they're usable. But add threads, libraries, and complex code and it quickly risks becoming a huge pain. It's one of those things from UNIX that are in principle not that bad of an idea, but just never grew up with the times.
- bluetomcat 3y ago
  It’s a mess that originates from the notion that terminals must be able to control and interrupt processes started from a shell. Events from the terminal are translated to SIGxxxx and acted upon by well-behaving programs. Had there been no terminals from the beginning, we would probably have a different abstraction for interrupting processes.
  - pjc50 3y ago
    There's also the signals that can originate from "within", like SIGILL, the SIGPIPE for interrupting otherwise blocking IO, and SIGCHILD for subprocess management.
- deleted 3y ago
  [deleted]
foobarian 3y ago
This goes right back to New Jersey vs. MIT/Stanford. Good background reading: [1] [1] https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html
rzzzt 3y ago
Wasn't there some additional magic around BSD vs. System V-style signal handling? In one of these the signal handler is restored to the default after each invocation, so the first thing you have to do is reinstate your own handler (and you still might get surprised/terminated by a second signal that arrives too quickly to do this, and gets handled by the default signal handler). Or is that an arcane thing that no longer applies to modern systems?
- fanf2 3y ago
  Yes, you are describing traditional System V style signals. BSD made signal handlers persistent. Modern (since the 1990s) code uses sigaction() to set up signal handlers which allows you to choose the sensible (BSD) semantics.
tuyiown 3y ago
I'm no UNIX guru but two things I've gathered: 1. For signal handling: if your signal handler needs to do more that setting a global value that ought to be read regularly by your program, consider that it might not be the mechanism you need, or that you're trying to hack around code design that have implications you really might not want to deal with. 2. For signal sending for process group control: read carefully manuals and forum helps to understand semantic subtleties between groups and sessions, as sending signals from a terminal with ctrl-<d,c,z,…> does not do the same thing than signaling processes from a program system call, which can really sway intuitions you might have on the subject. (edit: fixed wrong example with ctrl-<…> signals)
- fanf2 3y ago
  If your program has an async loop (select or poll etc.) then you need to know about the self-pipe trick for avoiding problems with signals. https://cr.yp.to/docs/selfpipe.html https://cr.yp.to/docs/selfpipe.html
  - Sesse__ 3y ago
    The article explicitly mentioned signalfd, which is a less hackish way of doing the same.
    - fanf2 3y ago
      signalfd() is less portable and has caveats around signal masks and subprocesses that you can avoid with the self-pipe trick.
- RickHull 3y ago
  This pattern of the signal handler merely setting a global var seems robust. Of course, now the programmer must decide when and where to check the global value.
  - Sesse__ 3y ago
    It must also be of an atomic type. POSIX specifies exactly one, namely “volatile sig_atomic_t”. I guess in C++11 etc. you can also use the language-provided atomics.
nonrandomstring 3y ago
Yes. Run. Never look behind you. One of the most terrifying tar-pits of my life came from trying to solve a process concurrency problem by monitoring PID files and using Unix signals to pause and resume things.
- h2odragon 3y ago
  Yeah well chainsaws are wonderful tools, too; but if you go to juggling them you're gonna have a bad time.
  - nonrandomstring 3y ago
    I'd give your reply a thumbs-up, but....
welder 3y ago
The author forgot an important thing to remember about signals: Signals received by a PARENT process are propagated to all it's CHILD processes. > Should You Be Scared of Unix Signals? You should never be scared of anything in life... as long as you've properly prepared, researched, and tested.
- Joker_vD 3y ago
  > Signals received by a PARENT process are propagated to all it's CHILD processes. They are most definitely not, unless the parent does implement such propagation on its own.
  - tuyiown 3y ago
    > unless the parent does implement such propagation or if the signal is send to 0, or -PGID or if signal target PID is a group leader / session id or any other subtleties But you're right the general case is that children won't be interrupted.
  - fanf2 3y ago
    Right. Perhaps welder is confused by the behaviour of process groups and controlling terminals.
    - welder 3y ago
      Yep, my bad. I forgot it's only because systemd defaults to control-group.
- sunshowers 3y ago
  > Signals received by a PARENT process are propagated to all it's CHILD processes. This is not true, outside of the context of process groups or if the parent implements this manually (edit: or some OS-specific features). I gave a talk on this at RustConf this year and the video for it should be out soon, but here are the slides: https://docs.google.com/presentation/d/e/2PACX-1vQuV0pYyWQulFbnRHfgvtHqQ_ZVyrnkmfQxT0eQfjEpVjnptUIG8uMx9FuUDz3wxtWqVN5QA9C4biZT/pub?start=false&loop=false&delayms=3000 https://docs.google.com/presentation/d/e/2PACX-1vQuV0pYyWQul...
meindnoch 3y ago
The first time I encountered signals and EINTR, I was baffled. It felt like I was missing something obvious. This couldn't be how things really work. This must be a joke or something. I must be reading an outdated documentation, and there's a sane solution for this, right?
- dale_glass 3y ago
  Yeah, it's using a modern framework like Qt that wraps around all this nonsense and provides something more comfortable to work with.
- BobbyTables2 3y ago
  Hehe, consider something like wanting to log an error message from a signal handler catching SIGFAULT... Except the standard logger route my isn’t signal safe, nor much of the basic library routines one would want to use! The only sane solution is to block/mask all signals, perhaps polling for them if really needed. Anything else is on the path to insanity.
  - Spivak 3y ago
    Right which is why you don't actually do much of anything in the signal handler except save that it happened somewhere to be processed by your application later. https://docs.python.org/3/library/signal.html#execution-of-python-signal-handlers https://docs.python.org/3/library/signal.html#execution-of-p... Here's how Python does it.
    - skitter 3y ago
      Uh, that Examples section calls print inside the signal handler. Python's IO stack is non-reentrant. I hope nobody follows these examples and gets exceptions at runtime because of it.
      - eesmith 3y ago
        "A Python signal handler does not get executed inside the low-level (C) signal handler. Instead, the low-level signal handler sets a flag which tells the virtual machine to execute the corresponding Python signal handler at a later point(for example at the next bytecode instruction)" The low-level C handler set the flag then returns. Only some time later does the Python run-time call the associated Python handler that you see in the Examples section.
claytonjy 3y ago
The most important thing I know about signals is that in dockerized applications, your ENTRYPOINT or CMD must use the exec form (`["command", "arg1", "arg2"]`) in order for your application to receive signals. A lot of applications, like web/API frameworks, will gracefully shutdown if they receive SIGTERM or SIGINT, but if that signal doesn't propagate to the app you'll need to SIGKILL instead. Kubernetes will do this for you, waiting for your process to shutdown and then ungracefully killing it when it doesn't do so in time. But you can't do env var substitution with the exec form...
- corytheboyd 3y ago
  Isn’t this something that Tini solves? Really easy to add to compose services with “init: true”
  - Spivak 3y ago
    It really should be the default, I can't think of any containerized application that actually upholds the responsibilities of PID1.
    - corytheboyd 3y ago
      Yeah I don’t disagree at all. I’ve seen so many people just put up with waiting for their containers to get SIGKILLed by the docker timeout, when they could instead use tini to forward SIGINT to trigger their graceful shutdowns. Drives me nuts haha
  - claytonjy 3y ago
    I think it's supposed to? I'm generally running on k8s with base images that don't include tini, so I've settled for exec-style CMD instead
    - corytheboyd 3y ago
      Ah gotcha. I believe it can be baked into images as well, per the entrypoint example in the readme: https://github.com/krallin/tini https://github.com/krallin/tini Not sure how this will fare IRL in k8s as I haven’t much experience there. It’s still silly that this is the default behavior where you need something like Tini, but I digress.
pjmlp 3y ago
It depends on how much one cares to write portable UNIX code, that depends on how signals work across implementations.
ravi-delia 3y ago
If anyone is curious about how signals are used for garbage collection (specifically SIGSEGV) check out this article: https://medium.com/@MartinCracauer/generational-garbage-collection-write-barriers-write-protection-and-userfaultfd-2-8b0e796b8f7f https://medium.com/@MartinCracauer/generational-garbage-coll... It's handy for knowing which areas of the heap have been untouched (and thus don't need to be scavenged in a generational gc).
- hayley-patton 3y ago
  SBCL now uses the "write-bitmap"/"card marking" scheme on some platforms, which is a tiny bit faster (1-2% mentioned on sbcl-devel). More interesting is that doing the touch-detection in software allows for finer grained precision (e.g. #+mark-region-gc uses 128 byte cards) than hardware (e.g. 4kiB pages on x86-64, 16kiB on M1) which can drastically affect scavenging time [0]. The precision is also really nice for non-moving generational schemes: if old and new objects exist on the same card, writes to new objects (which are more common too!) will cause old objects to needlessly be scanned by GC, which is called "card pollution" by Demers et al [1], so reducing the card size reduces the likelihood of that happening. [0] https://tschatzl.github.io/2022/02/15/card-table-card-size.html https://tschatzl.github.io/2022/02/15/card-table-card-size.h... [1] https://dl.acm.org/doi/pdf/10.1145/96709.96735 https://dl.acm.org/doi/pdf/10.1145/96709.96735
  - ravi-delia 3y ago
    Oh neat, I thought it was still unoptimized compared to gengc (which afaik does still use write protection). I'll have to check it out, see if I have a machine it'll run faster on
    - hayley-patton 3y ago
      gencgc uses software protection on some but not all architectures -- I recall x86-64 and MIPS but not ARM though. On x86-64 with SBCL 2.3.8 for example: * (disassemble #'(setf svref)) ; disassembly for (SETF SVREF) ; Size: 39 bytes. Origin: #x5349B00D ; (SETF SVREF) ; 0D: 483B77F9 CMP RSI, [RDI-7] ; 11: 731D JAE L0 ; 13: 488D44B701 LEA RAX, [RDI+RSI*4+1] ↓ card marking here ; 18: 48C1E80A SHR RAX, 10 ; 1C: 25FFFF0F00 AND EAX, 1048575 ; 21: 41C6040400 MOV BYTE PTR [R12+RAX], 0 ↓ pointer store here ; 26: 488954B701 MOV [RDI+RSI*4+1], RDX ; 2B: C9 LEAVE ; 2C: F8 CLC ; 2D: C3 RET ; 2E: CC10 INT3 16 ; Invalid argument count trap ; 30: L0: CC24 INT3 36 ; INVALID-VECTOR-INDEX-ERROR ; 32: 1C BYTE #X1C ; RDI(d) ; 33: 19 BYTE #X19 ; RSI(a)
notbeuller 3y ago
I was talking to someone at a vintage computer meetup this weekend and we ended up talking about "fear" in exactly this context - fear of asking, or fear that we couldn't know something or that it was too hard. From my perspective it was for exactly things like signals that were not intuitive or broke my mental model of linear programming (threads too, when they first became popular). He was an artist and his fear had been around not understanding what other people are doing with their art, or if his art was getting his message across. In both our cases, we benefitted from having mentors that helped explain (like this fine article does) by breaking things down and then... letting us learn from the broken down pieces rather than all at once. A mentor can help us conceptualize difficult concepts by showing us, intuitively where to draw our abstractions. Overcoming the fear for both of us came down to acknowledging it, "owning it", and being ok with it. (Sorry if this is rambling, but it was one of the most rewarding conversations I've had in years, and touched at least tangentially on this article.) (And I'm still have fear of MMUs, but I've never had to write anything at that level, although it's on my programming bucket list...)
- helpfulContrib 3y ago
  [dead]
- convolvatron 3y ago
  until I wrote code to manage the page tables, I did too. there's really nothing there that you don't already know.
sriram_malhar 3y ago
In the past I have used a non-blocking pre-allocated ring buffer to hold signal information, so that the sig handler is not held for too long. Using sem_post, it wakes up a separate thread (waiting using sem_wait). This has been quite pain-free, but not entirely sure how to smoke out bugs in the system.
1f60c 3y ago
I think this is also good life advice generally: > don’t be afraid! be careful!
helpfulContrib 3y ago
[dead]
extraduder_ire 3y ago
I've probably learned more about unix signals from using them on a running machine than I did when I had to write c code using them. Both nowadays, from when I have to send a SIGHUP to GNOME to restart it when some extension has a memory leak and it locks up. (Really wish the wayland version had that functionality) And quite a while back when I'd often SIGSTOP firefox so my crummy core2duo machine could play videos without skipping frames.
GuB-42 3y ago
I remember someone telling me that the only thing you should be allowed to do in a signal handler is to set a semaphore (or similar synchronization primitive). The actual logic is to be done in the main application. That's a similar idea as the signalfd function the article mentions. Obviously an oversimplification, but I think, a good rule of thumb.
monocasa 3y ago
So signals make sense once you understand where they came from. The writers of Unix needed a kernel->user downcall mechanism. Back then, user mode code was thought of as being like what we know call unikernels with the kernel thought of a bit more like a hypervisor. The individual user sessions were, early on in Unix's history, only a couple more primitives on top of what we'd later think of as DOS style environments, simply multiplexed on the hardware. From that perspective, mirroring the microcode->kernel downcall mechanism (interrupts) at the kernel->user boundary makes plenty of sense. Once you see them for what they are (interrupts for user mode instead of kernel mode), all of their goofiness makes sense. * SIGSEGV, SIGILL, etc. map to processor exceptions. * SIGIO, SIGWINCH, etc. map to I/O device interrupts, including SIGALRM as a timer interrupt. * SIGSTOP, SIGKILL, etc. map to platform interrupts like NMIs. * SIGUSR, etc. map to IPIs. It's a one shot downcall mechanism that can happen at any time, so all of the same way you think about concurrency during an interrupt applies here as well. You can't take a regular mutex because it might already be held by the context you interrupted. Less of a deal now, but any non-reentrant code shared with the regular user context is off the table. So just like how in interrupt context, large swaths of the kernel are off limits, large swaths of user mode are off limits except very carefully written, very context dependent code. I even ported an RTOS to run in *nix user mode for CI/development purposes, where the interrupt controller abstraction was simply implemented as signals. The code didn't look any different than you'd expect from running under a hypervisor with a paravirtualized interrupt controller.
- scottlamb 3y ago
  > * SIGSEGV, SIGILL, etc. map to processor exceptions. That's the idea anyway. But you can also `kill -SEGV $SOMEPID`. There are (now, since threading became a thing on Unix) three categories of signals: * async, process-directed, as with the command above, SIGALRM, or the like. * async, thread-directed, as with `pthread_kill`. * sync, thread-directed, as with the processor exceptions you mentioned. The signal numbers generally are good enough, but the `si_code` member of `siginfo_t` will more authoritatively distinguish the source of a signal. Anyway, I agree it's useful to think about the type of signal you're interested in handling. I think most people are primarily interested in async, process-directed signals for graceful shutdown on `SIGTERM`/`SIGINT` and/or config reloading on `SIGHUP`. Fortunately, that's the easiest to handle safely. One of my favorite ways is to not use a signal handler at all. Mask out the signal in main() before creating any threads, and create a thread which loops over sigwaitinfo.
KerrAvon 3y ago
it's mentioned in the article, but it's worth emphasizing that malloc is unsafe to use inside a signal handler. This should indeed scare you.
- colonwqbang 3y ago
  Now go read the source code for GNU libc printf and grep for "malloc".
matheusmoreira 3y ago
Absolutely. Asynchronous signal safety is among the murkiest waters of systems programming. It's pointless to even try to do anything in a signal handler, it's not safe to do anything more complex than setting a flag. The sanest way to handle signals seems to be signalfd. You just turn off normal signal delivery and handle them by epolling a signals file descriptor instead. Not portable of course, it's a Linux feature.
- usefulcat 3y ago
  > it's not safe to do anything more complex than setting a flag Though I generally agree with the sentiment, there are other limited but still useful things you can do, for example write() is allowed: const char msg[] = "\ncaught signal, finishing..\n"; write(2, msg, sizeof(msg)); // 1 == stdout, 2 == stderr
  - nimish 3y ago
    Which is how the infamous self-pipe trick works: https://cr.yp.to/docs/selfpipe.html https://cr.yp.to/docs/selfpipe.html Nowadays signalfd and pidfd are the better options.
  - scottlamb 3y ago
    write is async signal-safe, but you should save errno before the write and restore it afterward...
- cmrdporcupine 3y ago
  I love that userfaultfd now exists as an alternative to trapping sigsegv. I wish it was a bit more flexible though (I'd like to be able to associate user defined meta-data with the fault depending on address, for example.)
- agapon 3y ago
  A portable way to do this could be with sigtimedwait or sigwait plus sigpending or multi-threading.
- and0 3y ago
pxeger1 3y ago
I think the description of SIGWINCH is subtly wrong. AIUI, it’s sent by the terminal program to the child process (e.g. the shell). It is true if by “terminal program” you mean “program that runs in a terminal” (and arguably there are things which are both, e.g. tmux or emacs)
mark_undoio 3y ago
A weird (at first sight) thing about signal handlers - the kernel doesn't really know, or care, if you're running in one. Which makes sense because signal handlers can nest, so ideally the kernel wouldn't have to maintain some state for each enter/exit of signal handlers to be tracked. Instead, the kernel sets up the necessary conditions for things to act like a signal handler expects and puts some state on your the stack representing your registers, active signal mask, etc, plus some state to ensure a sigreturn syscall will run. If you return from your handler, that'll take effect and your state is restored to where you were before. If you go into a nested signal handler, the kernel pushes another pile of state on the stack to return to where you are now. You're free to change that saved state before returning - or just siglongjmp out of there and discard it, if you know what you're doing. Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers. From the kernel's point of view, there's nothing magic about them at all but usually we're stuck with libc restrictions into the bargain.
- 10000truths 3y ago
  It also makes sense because the kernel has no way of knowing. When you exec a binary, all that the kernel sees is an ELF image with a blob of bytes in the .text section. It doesn't know which part(s) of those blob of bytes is a signal handler. Even if it did know, the kernel has no way of verifying that the function is only ever invoked by the signal (userspace could manually call the signal handler function).
- ptsneves 3y ago
  > Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers. And this is a huge thing. People do all kinds of operations in signal handlers completely oblivious to the pitfalls. Pitfalls which often do not manifest, making it a great "it works for me" territory. I once raised a ticket on fluentbit[1] about it but they have abused signal handlers so thoroughly that I do not think they can mitigate the issue without a major rewriting of the signal and crash handling. [1] https://github.com/fluent/fluent-bit/issues/4836 https://github.com/fluent/fluent-bit/issues/4836
  - 3y ago
wofo 3y ago
As someone who once in a while dabbles in Linux systems programming, is there any reason to use signalfd over timerfd when all you want is a timer? The article suggests you can use signals as timers, which left me wondering...
- colonwqbang 3y ago
  No, timerfd is a better tool for that now. But it wasn't invented until 2008 or so.
camel-cdr 3y ago
I recently learned about a really cool usecase of signals: https://repo.hca.bsc.es/gitlab/epi-public/risc-v-vector-simulation-environment/-/wikis/vehave-user-guide https://repo.hca.bsc.es/gitlab/epi-public/risc-v-vector-simu... It's a userspace risc-v vector extension emulator that handles SIGILL signals to emulate the missing vector instructions on risc-v cpus without the vector extension. It can just be inserted in to any binary with LD_PRELOAD.
- nayuki 3y ago
  I used the Intel Software Development Emulator (SDE) about a decade ago. It lets you develop and test software on a CPU that doesn't have the necessary vector extensions (AVX, etc.). I presume it uses the SIGILL trapping technique. In practice, I found it to be about 1000× slower than native - but that's why it's intended to be a development tool. https://www.intel.com/content/www/us/en/developer/articles/tool/software-development-emulator.html https://www.intel.com/content/www/us/en/developer/articles/t...
leoc 3y ago
I found the "Unix signals" section of the 2010 LWN article "Ghosts of Unix past, part 3: Unfixable designs" by Neil Brown https://lwn.net/Articles/414618/ https://lwn.net/Articles/414618/ very interesting, though I don't know enough to recommend it firmly. (Previous HN discussions: https://news.ycombinator.com/item?id=27183784 https://news.ycombinator.com/item?id=27183784 on part 3, https://news.ycombinator.com/item?id=1941189 https://news.ycombinator.com/item?id=1941189 and https://news.ycombinator.com/item?id=1971106 https://news.ycombinator.com/item?id=1971106 on the whole series.)
er0k 3y ago
(2016)
cinntaile 3y ago
If you want portability, keep in mind that the man pages specify that you should use sigaction instead of signal when using a function as a signal handler rather than SIG_IGN or SIG_DFL. [0] [0] https://man7.org/linux/man-pages/man2/signal.2.html https://man7.org/linux/man-pages/man2/signal.2.html
lmm 3y ago
IMO it's fine to be scared of these, and fine to be scared of threads (and I have similar feelings about floating-point). You don't actually need threads; threads can't do anything that processes can't, indeed in the Linux 2.4 days threads and processes were the same thing to the scheduler. The fact that there are programs in common use that handle signals does not convince me that it's possible to handle signals correctly in all cases; lots of programs get used for decades while being subtly broken (see the famous "you are not expected to understand this").
cryptonector 3y ago
Should you be scared of Unix signals? Yes, so turn them all into self-pipes and handle them in an async I/O event loop if at all possible, and stop thinking about blocking them.
deleted 3y ago
[deleted]
bjourne 3y ago
> do something with garbage collection (but what? I’m confused about this still.) It's an almost hilarious technique. In a garbage collected, multi-threaded virtual machine the runtime may at some point decide it is time for garbage collection. However, all threads must first be stopped because allocating memory while the garbage collector runs leads to badness and/or the garbage collector moves objects referenced by threads. There is no direct way to signal to a thread that it should stop and having the thread poll some flag would hurt performance. Instead the compiler inserts dummy instructions (safepoints) at the top of every loop body or function entry point which writes to a page the runtime controls: mov [addr], eax Then when it's time to stop all threads, it just write protects the page! Every thread eventually stumbles upon the bomb. Boom! SIGSEGV! Boom! SIGSEGV! Boom! When every thread has exploded the garbage collector can run in peace. Then the runtime removes the write protection and resumes executing all threads. Something very similar is used on Windows (though I think it's called events and not signals) and sometimes you use read-protected pages rather than write-protected ones, but the principle is the same.
- hayley-patton 3y ago
  You probably want to read and not write, as writing will cause a lot of cache ping-pong and slow the mutator down when it doesn't SEGV on the dummy access <https://users.cecs.anu.edu.au/~steveb/pubs/papers/yieldpoint-ismm-2015.pdf#page=8 https://users.cecs.anu.edu.au/~steveb/pubs/papers/yieldpoint...>
- tubs 3y ago
  Have you got any links to how they continue the threads after the gc has run? I was looking for something similar but couldn't find much.
  - themoonisachees 3y ago
    Naïve approach is to have each segv handler poll something to tell when GC is done, and simply return when it can. State is then restored by the kernel.
  - jlokier 3y ago
dekhn 3y ago
Signals scare the hell out of me- that was the first point in learning about UNIX programming that made me question everything I'd learned about what is available to you as a function implementor/UNIX user in user space. It got even worse when I moved to threads and sudddenly I couldn't reconcile threads with signals (which are very process-oriented). Reentrant code gives me the same willies- or more correctly, it was a big surprise that people wrote code that depended on static values to store state between invocations.
c-cube 3y ago
I'm surprised that SIGPIPE is not mentioned (and barely present in comments). It's 90% of why I'm scared of signals and every time I write something mixing IO and threads it eventually rears its disgusting head. Can someone shine a light on what the heck it's designed to do?
- convolvatron 3y ago
  if you have a piped series of processes at the command line, and the middle process dies, how does the earlier part of the pipe get cleaned up? more tangentially, the older inetd model of spawning a process per connection used it for cleanup as well.
  - c-cube 3y ago
    The earlier part tries to write, gets `0` or `-1` with an error, concludes the pipe is closed, and exits? The doc for `write` even specifies the error code `EPIPE` that is set alongside with the `SIGPIPE`. Why the signal, when it's redundant?
doubloon 3y ago
i built a robot with two processes. one process is the brain, and all the brain does it wait for SIGUSR1. when it stops getting SIGUSR1 for like 2 tenths of a second in a row, it "ceases all motor function". the other process, which takes input and game controller stuff and sends camera feeds and turns on lights and measures temperature and blah blah blah, it continuously sends a SIGUSR1 to the brain process. it's like a dead man's switch for a robot. doesn't matter what goes wrong, basically when it stops sending SIGUSR1 then it means bad things have happened and the robot stops itself. now what if it gets stuck in a loop continuously sending SIGUSR1 erroneously? yes, that could be a problem, but it's never happened. unlike before i implemented this where, say, interference with the wifi or dead batteries or whatever could leave the robot running away in 'move forward mode' forever.
zX41ZdbW 3y ago
We are using a fuzzing method in ClickHouse that involves sending signals to random threads: https://github.com/ClickHouse/ClickHouse/blob/master/src/Common/ThreadFuzzer.h https://github.com/ClickHouse/ClickHouse/blob/master/src/Com... We immediately found bugs in many dependent libraries of one of three types: 1. A library does not check for EINTR. 2. A library that waits for something does check for EINTR but does not update a timeout, therefore stopping sooner than needed. 3. A library that waits for something does check for EINTR but incorrectly updates a timeout and waiting indefinitely.

257 points by gavinhoward 3y ago

While Julia starts scared and ends up feeling better, I've only gotten more scared of Unix signals over time. Context: I've written a robust command-line utility [0] that must handle signals, and unlike most utilities that are I/O-bound, mine is CPU-bound. An I/O-bound utility can easily use signalfd() (subject to the gotchas in the "signalfd() is useless" post that Julia links to, which you should also read). signalfd() will work for I/O-bound utilities because it turns those signals into I/O. Perfect. However, in a CPU-bound program, signals are used specifically to interrupt execution. This is, to put it mildly, as difficult as writing code for interrupts in the embedded space. Why? Because that's really what you're doing: handling an interrupt that can happen at any time. My solution was something I wish on no one: I used setjmp() and longjmp(). Horrors! Yep. And it gets worse: I had to longjmp() out of the signal handler. AH! And it gets worse: to ensure that there were no memory leaks, I had to keep a stack of jmp_bufs and manually jump to each one, which would be in a function where memory had to be cleaned up. Cue screams of bloody murder You may insist that longjmp()'ing out of signal handler is not allowed; it actually is [1], but unlike most other "async-signal-safe" functions, you need to ensure you don't interrupt a syscall or other code that is not async-signal-safe. So it gets worse: I have a signal lock that the signal handler checks. If it's not locked, the signal handler will longjmp() out of the signal handler. If it is locked, the signal handler sets a flag and returns. Then the code that unlocks signals checks for the flag and does a longjmp() if it's set. He's dead, Jim! I have another project that is a framework in C. This framework needs to handle signals for clients. It has to be general, so it has to handle CPU-bound clients. So I had to implement the same thing. I was able to make it easier, but it is also harder because I have that one thing that messes up every Unix API: threads. Nuclear mushroom cloud So should you be scared? It depends; if you can get away with signalfd() and know its gotchas, maybe not. But if you need anything more complex, yes, be very afraid. [0]: https://git.gavinhoward.com/gavin/bc https://git.gavinhoward.com/gavin/bc [1]: https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/ https://pubs.opengroup.org/onlinepubs/9699919799.2008edition...

fzeindl 3y ago
Have you ever read up on communicating sequential processes? In that model you have one or multiple threads executing work while new messages arrive on a queue. In such an architecture handling signals is trivial.
- boffinAudio 3y ago
  Yeah, I was going to mention that if the long-running/CPU-bound work was put in a worker thread with a message queue in which state messages are received/handled, it'd be relatively trivial to handle the signal across these constraints, since signals are only delivered upon context switches. But if you've just got one very tight loop and are running like a bat out of hell, well you deserve the hell you're in. ;) longjmp() is definitely not the right way to do this. A thread dedicated to handling signals (or doing the CPU work) which properly tells the main() thread to die/quit/clean up is how I'd do it .. even if you only spawn a thread for signal handling, its still way cleaner than that longjmp() business .. which is very difficult to understand, even with the context you've given in this thread (I read your code - its nasty) ...
  - gavinhoward 3y ago
    Why do I deserve that for having a CPU-bound program? And why is longjmp() the wrong thing to do?
    - boffinAudio 3y ago
      Because signals are delivered on context switch, and you're not allowing any of that to happen. Its not a well-behaved process in that case .. not that you shouldn't be able to just burn as much CPU as you want, just that you would need to open up some space to communicate with the OS in the meantime, and having a signal-handler-thread or a cpu-working-thread is how to do it properly. (Do you also handle SIGHUP? SIGSTOP?) longjmp() is just asking for hell 6 months later when you come back and have to work out why you did such a hokey thing in the first place. EDIT: it should be noted that the "side-effect" of using longjmp is .. a context switch .. so you gave yourself some space to behave properly with the OS. Its just that the more future-proof way to do it is to have a thread for work (you have this already with main()) and a thread for signals (you'd just add this to service signals only and give main() a way to cleanly exit) ... Not too complex, and a tad bit easier to read 6 months later ..

signa11 3y ago

if you are writing multithreaded programs, and considering dealing with signals, then yes.

wilkystyle 3y ago

I know this has been said before, but I absolutely love Julia's style of writing. It's humble and accessible, and full of great humor that is always good natured, as opposed to the more common snarky or negative humor: > Signals are a way for Unix processes to communicate! Except for SIGKILL. When you get sent SIGKILL nobody communicates with you, you just die immediately.

epcoa 3y ago
What about SIGSTOP?
- dspillett 3y ago
  Also not communication as it can't be ignored and your process just pauses, if you consider it a thing in its own right. Really though it is a pair with SIGCONT which is communication: SIGSTOP happens (which your process doesn't even see) then later SIGCONT happens which you do get notified about (“hey, process, you were just paused for a bit, you might want to reassess your surroundings”). This is little different from the scheduler not giving your process any time for a while, what difference there is being the while could be a lot longer and you get explicitly told that it has happened. The other possibility is that your process never wakes up because a SIGKILL happens first. Or maybe SIGTERM, I'm not sure whether or not a SIGCONT will be sent first in that instance. There is also the friendlier SIGSTSP which can be handled (or even ignored) but is otherwise identical to SIGSTOP. This is what you get sent if a user hits ctrl+z in most interactive shells (or a more polite process than one that just sends SIGSTOP wants you to pause).
  - epcoa 3y ago
    > if you consider it a thing in its own right. There is no guarantee you’re getting a SIGCONT, so it is, and you could have stopped there. Furthermore, they aren’t paired in any meaningful way. You can get a SIGCONT without any STOP (you can just do this with kill)
- deleted 3y ago
  [deleted]

dale_glass 3y ago

For once, Betteridge's Law of Headlines doesn't hold. Signals suck and are best avoided. When handling the concept in a vacuum, like writing a simple program from scratch that uses them, sure, they're usable. But add threads, libraries, and complex code and it quickly risks becoming a huge pain. It's one of those things from UNIX that are in principle not that bad of an idea, but just never grew up with the times.

bluetomcat 3y ago
It’s a mess that originates from the notion that terminals must be able to control and interrupt processes started from a shell. Events from the terminal are translated to SIGxxxx and acted upon by well-behaving programs. Had there been no terminals from the beginning, we would probably have a different abstraction for interrupting processes.
- pjc50 3y ago
  There's also the signals that can originate from "within", like SIGILL, the SIGPIPE for interrupting otherwise blocking IO, and SIGCHILD for subprocess management.
deleted 3y ago
[deleted]

foobarian 3y ago

This goes right back to New Jersey vs. MIT/Stanford. Good background reading: [1] [1] https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html

rzzzt 3y ago

Wasn't there some additional magic around BSD vs. System V-style signal handling? In one of these the signal handler is restored to the default after each invocation, so the first thing you have to do is reinstate your own handler (and you still might get surprised/terminated by a second signal that arrives too quickly to do this, and gets handled by the default signal handler). Or is that an arcane thing that no longer applies to modern systems?

fanf2 3y ago
Yes, you are describing traditional System V style signals. BSD made signal handlers persistent. Modern (since the 1990s) code uses sigaction() to set up signal handlers which allows you to choose the sensible (BSD) semantics.

tuyiown 3y ago

I'm no UNIX guru but two things I've gathered: 1. For signal handling: if your signal handler needs to do more that setting a global value that ought to be read regularly by your program, consider that it might not be the mechanism you need, or that you're trying to hack around code design that have implications you really might not want to deal with. 2. For signal sending for process group control: read carefully manuals and forum helps to understand semantic subtleties between groups and sessions, as sending signals from a terminal with ctrl-<d,c,z,…> does not do the same thing than signaling processes from a program system call, which can really sway intuitions you might have on the subject. (edit: fixed wrong example with ctrl-<…> signals)

fanf2 3y ago
If your program has an async loop (select or poll etc.) then you need to know about the self-pipe trick for avoiding problems with signals. https://cr.yp.to/docs/selfpipe.html https://cr.yp.to/docs/selfpipe.html
- Sesse__ 3y ago
  The article explicitly mentioned signalfd, which is a less hackish way of doing the same.
  - fanf2 3y ago
    signalfd() is less portable and has caveats around signal masks and subprocesses that you can avoid with the self-pipe trick.
RickHull 3y ago
This pattern of the signal handler merely setting a global var seems robust. Of course, now the programmer must decide when and where to check the global value.
- Sesse__ 3y ago
  It must also be of an atomic type. POSIX specifies exactly one, namely “volatile sig_atomic_t”. I guess in C++11 etc. you can also use the language-provided atomics.

nonrandomstring 3y ago

Yes. Run. Never look behind you. One of the most terrifying tar-pits of my life came from trying to solve a process concurrency problem by monitoring PID files and using Unix signals to pause and resume things.

h2odragon 3y ago
Yeah well chainsaws are wonderful tools, too; but if you go to juggling them you're gonna have a bad time.
- nonrandomstring 3y ago
  I'd give your reply a thumbs-up, but....

welder 3y ago

The author forgot an important thing to remember about signals: Signals received by a PARENT process are propagated to all it's CHILD processes. > Should You Be Scared of Unix Signals? You should never be scared of anything in life... as long as you've properly prepared, researched, and tested.

Joker_vD 3y ago
> Signals received by a PARENT process are propagated to all it's CHILD processes. They are most definitely not, unless the parent does implement such propagation on its own.
- tuyiown 3y ago
  > unless the parent does implement such propagation or if the signal is send to 0, or -PGID or if signal target PID is a group leader / session id or any other subtleties But you're right the general case is that children won't be interrupted.
- fanf2 3y ago
  Right. Perhaps welder is confused by the behaviour of process groups and controlling terminals.
  - welder 3y ago
    Yep, my bad. I forgot it's only because systemd defaults to control-group.
sunshowers 3y ago
> Signals received by a PARENT process are propagated to all it's CHILD processes. This is not true, outside of the context of process groups or if the parent implements this manually (edit: or some OS-specific features). I gave a talk on this at RustConf this year and the video for it should be out soon, but here are the slides: https://docs.google.com/presentation/d/e/2PACX-1vQuV0pYyWQulFbnRHfgvtHqQ_ZVyrnkmfQxT0eQfjEpVjnptUIG8uMx9FuUDz3wxtWqVN5QA9C4biZT/pub?start=false&loop=false&delayms=3000 https://docs.google.com/presentation/d/e/2PACX-1vQuV0pYyWQul...

meindnoch 3y ago

The first time I encountered signals and EINTR, I was baffled. It felt like I was missing something obvious. This couldn't be how things really work. This must be a joke or something. I must be reading an outdated documentation, and there's a sane solution for this, right?

dale_glass 3y ago
Yeah, it's using a modern framework like Qt that wraps around all this nonsense and provides something more comfortable to work with.
BobbyTables2 3y ago
Hehe, consider something like wanting to log an error message from a signal handler catching SIGFAULT... Except the standard logger route my isn’t signal safe, nor much of the basic library routines one would want to use! The only sane solution is to block/mask all signals, perhaps polling for them if really needed. Anything else is on the path to insanity.
- Spivak 3y ago
  Right which is why you don't actually do much of anything in the signal handler except save that it happened somewhere to be processed by your application later. https://docs.python.org/3/library/signal.html#execution-of-python-signal-handlers https://docs.python.org/3/library/signal.html#execution-of-p... Here's how Python does it.
  - skitter 3y ago
    Uh, that Examples section calls print inside the signal handler. Python's IO stack is non-reentrant. I hope nobody follows these examples and gets exceptions at runtime because of it.
    - eesmith 3y ago
      "A Python signal handler does not get executed inside the low-level (C) signal handler. Instead, the low-level signal handler sets a flag which tells the virtual machine to execute the corresponding Python signal handler at a later point(for example at the next bytecode instruction)" The low-level C handler set the flag then returns. Only some time later does the Python run-time call the associated Python handler that you see in the Examples section.

claytonjy 3y ago

The most important thing I know about signals is that in dockerized applications, your ENTRYPOINT or CMD must use the exec form (`["command", "arg1", "arg2"]`) in order for your application to receive signals. A lot of applications, like web/API frameworks, will gracefully shutdown if they receive SIGTERM or SIGINT, but if that signal doesn't propagate to the app you'll need to SIGKILL instead. Kubernetes will do this for you, waiting for your process to shutdown and then ungracefully killing it when it doesn't do so in time. But you can't do env var substitution with the exec form...

corytheboyd 3y ago
Isn’t this something that Tini solves? Really easy to add to compose services with “init: true”
- Spivak 3y ago
  It really should be the default, I can't think of any containerized application that actually upholds the responsibilities of PID1.
  - corytheboyd 3y ago
    Yeah I don’t disagree at all. I’ve seen so many people just put up with waiting for their containers to get SIGKILLed by the docker timeout, when they could instead use tini to forward SIGINT to trigger their graceful shutdowns. Drives me nuts haha
- claytonjy 3y ago
  I think it's supposed to? I'm generally running on k8s with base images that don't include tini, so I've settled for exec-style CMD instead
  - corytheboyd 3y ago
    Ah gotcha. I believe it can be baked into images as well, per the entrypoint example in the readme: https://github.com/krallin/tini https://github.com/krallin/tini Not sure how this will fare IRL in k8s as I haven’t much experience there. It’s still silly that this is the default behavior where you need something like Tini, but I digress.

pjmlp 3y ago

It depends on how much one cares to write portable UNIX code, that depends on how signals work across implementations.

ravi-delia 3y ago

If anyone is curious about how signals are used for garbage collection (specifically SIGSEGV) check out this article: https://medium.com/@MartinCracauer/generational-garbage-collection-write-barriers-write-protection-and-userfaultfd-2-8b0e796b8f7f https://medium.com/@MartinCracauer/generational-garbage-coll... It's handy for knowing which areas of the heap have been untouched (and thus don't need to be scavenged in a generational gc).

hayley-patton 3y ago
SBCL now uses the "write-bitmap"/"card marking" scheme on some platforms, which is a tiny bit faster (1-2% mentioned on sbcl-devel). More interesting is that doing the touch-detection in software allows for finer grained precision (e.g. #+mark-region-gc uses 128 byte cards) than hardware (e.g. 4kiB pages on x86-64, 16kiB on M1) which can drastically affect scavenging time [0]. The precision is also really nice for non-moving generational schemes: if old and new objects exist on the same card, writes to new objects (which are more common too!) will cause old objects to needlessly be scanned by GC, which is called "card pollution" by Demers et al [1], so reducing the card size reduces the likelihood of that happening. [0] https://tschatzl.github.io/2022/02/15/card-table-card-size.html https://tschatzl.github.io/2022/02/15/card-table-card-size.h... [1] https://dl.acm.org/doi/pdf/10.1145/96709.96735 https://dl.acm.org/doi/pdf/10.1145/96709.96735
- ravi-delia 3y ago
  Oh neat, I thought it was still unoptimized compared to gengc (which afaik does still use write protection). I'll have to check it out, see if I have a machine it'll run faster on
  - hayley-patton 3y ago
    gencgc uses software protection on some but not all architectures -- I recall x86-64 and MIPS but not ARM though. On x86-64 with SBCL 2.3.8 for example: * (disassemble #'(setf svref)) ; disassembly for (SETF SVREF) ; Size: 39 bytes. Origin: #x5349B00D ; (SETF SVREF) ; 0D: 483B77F9 CMP RSI, [RDI-7] ; 11: 731D JAE L0 ; 13: 488D44B701 LEA RAX, [RDI+RSI*4+1] ↓ card marking here ; 18: 48C1E80A SHR RAX, 10 ; 1C: 25FFFF0F00 AND EAX, 1048575 ; 21: 41C6040400 MOV BYTE PTR [R12+RAX], 0 ↓ pointer store here ; 26: 488954B701 MOV [RDI+RSI*4+1], RDX ; 2B: C9 LEAVE ; 2C: F8 CLC ; 2D: C3 RET ; 2E: CC10 INT3 16 ; Invalid argument count trap ; 30: L0: CC24 INT3 36 ; INVALID-VECTOR-INDEX-ERROR ; 32: 1C BYTE #X1C ; RDI(d) ; 33: 19 BYTE #X19 ; RSI(a)

notbeuller 3y ago

I was talking to someone at a vintage computer meetup this weekend and we ended up talking about "fear" in exactly this context - fear of asking, or fear that we couldn't know something or that it was too hard. From my perspective it was for exactly things like signals that were not intuitive or broke my mental model of linear programming (threads too, when they first became popular). He was an artist and his fear had been around not understanding what other people are doing with their art, or if his art was getting his message across. In both our cases, we benefitted from having mentors that helped explain (like this fine article does) by breaking things down and then... letting us learn from the broken down pieces rather than all at once. A mentor can help us conceptualize difficult concepts by showing us, intuitively where to draw our abstractions. Overcoming the fear for both of us came down to acknowledging it, "owning it", and being ok with it. (Sorry if this is rambling, but it was one of the most rewarding conversations I've had in years, and touched at least tangentially on this article.) (And I'm still have fear of MMUs, but I've never had to write anything at that level, although it's on my programming bucket list...)

helpfulContrib 3y ago
[dead]
convolvatron 3y ago
until I wrote code to manage the page tables, I did too. there's really nothing there that you don't already know.

sriram_malhar 3y ago

In the past I have used a non-blocking pre-allocated ring buffer to hold signal information, so that the sig handler is not held for too long. Using sem_post, it wakes up a separate thread (waiting using sem_wait). This has been quite pain-free, but not entirely sure how to smoke out bugs in the system.

1f60c 3y ago

I think this is also good life advice generally: > don’t be afraid! be careful!

helpfulContrib 3y ago

[dead]

extraduder_ire 3y ago

I've probably learned more about unix signals from using them on a running machine than I did when I had to write c code using them. Both nowadays, from when I have to send a SIGHUP to GNOME to restart it when some extension has a memory leak and it locks up. (Really wish the wayland version had that functionality) And quite a while back when I'd often SIGSTOP firefox so my crummy core2duo machine could play videos without skipping frames.

GuB-42 3y ago

I remember someone telling me that the only thing you should be allowed to do in a signal handler is to set a semaphore (or similar synchronization primitive). The actual logic is to be done in the main application. That's a similar idea as the signalfd function the article mentions. Obviously an oversimplification, but I think, a good rule of thumb.

monocasa 3y ago

So signals make sense once you understand where they came from. The writers of Unix needed a kernel->user downcall mechanism. Back then, user mode code was thought of as being like what we know call unikernels with the kernel thought of a bit more like a hypervisor. The individual user sessions were, early on in Unix's history, only a couple more primitives on top of what we'd later think of as DOS style environments, simply multiplexed on the hardware. From that perspective, mirroring the microcode->kernel downcall mechanism (interrupts) at the kernel->user boundary makes plenty of sense. Once you see them for what they are (interrupts for user mode instead of kernel mode), all of their goofiness makes sense. * SIGSEGV, SIGILL, etc. map to processor exceptions. * SIGIO, SIGWINCH, etc. map to I/O device interrupts, including SIGALRM as a timer interrupt. * SIGSTOP, SIGKILL, etc. map to platform interrupts like NMIs. * SIGUSR, etc. map to IPIs. It's a one shot downcall mechanism that can happen at any time, so all of the same way you think about concurrency during an interrupt applies here as well. You can't take a regular mutex because it might already be held by the context you interrupted. Less of a deal now, but any non-reentrant code shared with the regular user context is off the table. So just like how in interrupt context, large swaths of the kernel are off limits, large swaths of user mode are off limits except very carefully written, very context dependent code. I even ported an RTOS to run in *nix user mode for CI/development purposes, where the interrupt controller abstraction was simply implemented as signals. The code didn't look any different than you'd expect from running under a hypervisor with a paravirtualized interrupt controller.

scottlamb 3y ago
> * SIGSEGV, SIGILL, etc. map to processor exceptions. That's the idea anyway. But you can also `kill -SEGV $SOMEPID`. There are (now, since threading became a thing on Unix) three categories of signals: * async, process-directed, as with the command above, SIGALRM, or the like. * async, thread-directed, as with `pthread_kill`. * sync, thread-directed, as with the processor exceptions you mentioned. The signal numbers generally are good enough, but the `si_code` member of `siginfo_t` will more authoritatively distinguish the source of a signal. Anyway, I agree it's useful to think about the type of signal you're interested in handling. I think most people are primarily interested in async, process-directed signals for graceful shutdown on `SIGTERM`/`SIGINT` and/or config reloading on `SIGHUP`. Fortunately, that's the easiest to handle safely. One of my favorite ways is to not use a signal handler at all. Mask out the signal in main() before creating any threads, and create a thread which loops over sigwaitinfo.

KerrAvon 3y ago

it's mentioned in the article, but it's worth emphasizing that malloc is unsafe to use inside a signal handler. This should indeed scare you.

colonwqbang 3y ago
Now go read the source code for GNU libc printf and grep for "malloc".

matheusmoreira 3y ago

Absolutely. Asynchronous signal safety is among the murkiest waters of systems programming. It's pointless to even try to do anything in a signal handler, it's not safe to do anything more complex than setting a flag. The sanest way to handle signals seems to be signalfd. You just turn off normal signal delivery and handle them by epolling a signals file descriptor instead. Not portable of course, it's a Linux feature.

usefulcat 3y ago
> it's not safe to do anything more complex than setting a flag Though I generally agree with the sentiment, there are other limited but still useful things you can do, for example write() is allowed: const char msg[] = "\ncaught signal, finishing..\n"; write(2, msg, sizeof(msg)); // 1 == stdout, 2 == stderr
- nimish 3y ago
  Which is how the infamous self-pipe trick works: https://cr.yp.to/docs/selfpipe.html https://cr.yp.to/docs/selfpipe.html Nowadays signalfd and pidfd are the better options.
- scottlamb 3y ago
  write is async signal-safe, but you should save errno before the write and restore it afterward...
cmrdporcupine 3y ago
I love that userfaultfd now exists as an alternative to trapping sigsegv. I wish it was a bit more flexible though (I'd like to be able to associate user defined meta-data with the fault depending on address, for example.)
agapon 3y ago
A portable way to do this could be with sigtimedwait or sigwait plus sigpending or multi-threading.
and0 3y ago

pxeger1 3y ago

I think the description of SIGWINCH is subtly wrong. AIUI, it’s sent by the terminal program to the child process (e.g. the shell). It is true if by “terminal program” you mean “program that runs in a terminal” (and arguably there are things which are both, e.g. tmux or emacs)

mark_undoio 3y ago

A weird (at first sight) thing about signal handlers - the kernel doesn't really know, or care, if you're running in one. Which makes sense because signal handlers can nest, so ideally the kernel wouldn't have to maintain some state for each enter/exit of signal handlers to be tracked. Instead, the kernel sets up the necessary conditions for things to act like a signal handler expects and puts some state on your the stack representing your registers, active signal mask, etc, plus some state to ensure a sigreturn syscall will run. If you return from your handler, that'll take effect and your state is restored to where you were before. If you go into a nested signal handler, the kernel pushes another pile of state on the stack to return to where you are now. You're free to change that saved state before returning - or just siglongjmp out of there and discard it, if you know what you're doing. Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers. From the kernel's point of view, there's nothing magic about them at all but usually we're stuck with libc restrictions into the bargain.

10000truths 3y ago
It also makes sense because the kernel has no way of knowing. When you exec a binary, all that the kernel sees is an ELF image with a blob of bytes in the .text section. It doesn't know which part(s) of those blob of bytes is a signal handler. Even if it did know, the kernel has no way of verifying that the function is only ever invoked by the signal (userspace could manually call the signal handler function).
ptsneves 3y ago
> Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers. And this is a huge thing. People do all kinds of operations in signal handlers completely oblivious to the pitfalls. Pitfalls which often do not manifest, making it a great "it works for me" territory. I once raised a ticket on fluentbit[1] about it but they have abused signal handlers so thoroughly that I do not think they can mitigate the issue without a major rewriting of the signal and crash handling. [1] https://github.com/fluent/fluent-bit/issues/4836 https://github.com/fluent/fluent-bit/issues/4836
- 3y ago

wofo 3y ago

As someone who once in a while dabbles in Linux systems programming, is there any reason to use signalfd over timerfd when all you want is a timer? The article suggests you can use signals as timers, which left me wondering...

colonwqbang 3y ago
No, timerfd is a better tool for that now. But it wasn't invented until 2008 or so.

camel-cdr 3y ago

I recently learned about a really cool usecase of signals: https://repo.hca.bsc.es/gitlab/epi-public/risc-v-vector-simulation-environment/-/wikis/vehave-user-guide https://repo.hca.bsc.es/gitlab/epi-public/risc-v-vector-simu... It's a userspace risc-v vector extension emulator that handles SIGILL signals to emulate the missing vector instructions on risc-v cpus without the vector extension. It can just be inserted in to any binary with LD_PRELOAD.

nayuki 3y ago
I used the Intel Software Development Emulator (SDE) about a decade ago. It lets you develop and test software on a CPU that doesn't have the necessary vector extensions (AVX, etc.). I presume it uses the SIGILL trapping technique. In practice, I found it to be about 1000× slower than native - but that's why it's intended to be a development tool. https://www.intel.com/content/www/us/en/developer/articles/tool/software-development-emulator.html https://www.intel.com/content/www/us/en/developer/articles/t...

leoc 3y ago

I found the "Unix signals" section of the 2010 LWN article "Ghosts of Unix past, part 3: Unfixable designs" by Neil Brown https://lwn.net/Articles/414618/ https://lwn.net/Articles/414618/ very interesting, though I don't know enough to recommend it firmly. (Previous HN discussions: https://news.ycombinator.com/item?id=27183784 https://news.ycombinator.com/item?id=27183784 on part 3, https://news.ycombinator.com/item?id=1941189 https://news.ycombinator.com/item?id=1941189 and https://news.ycombinator.com/item?id=1971106 https://news.ycombinator.com/item?id=1971106 on the whole series.)

er0k 3y ago

(2016)

cinntaile 3y ago

If you want portability, keep in mind that the man pages specify that you should use sigaction instead of signal when using a function as a signal handler rather than SIG_IGN or SIG_DFL. [0] [0] https://man7.org/linux/man-pages/man2/signal.2.html https://man7.org/linux/man-pages/man2/signal.2.html

lmm 3y ago

IMO it's fine to be scared of these, and fine to be scared of threads (and I have similar feelings about floating-point). You don't actually need threads; threads can't do anything that processes can't, indeed in the Linux 2.4 days threads and processes were the same thing to the scheduler. The fact that there are programs in common use that handle signals does not convince me that it's possible to handle signals correctly in all cases; lots of programs get used for decades while being subtly broken (see the famous "you are not expected to understand this").

cryptonector 3y ago

Should you be scared of Unix signals? Yes, so turn them all into self-pipes and handle them in an async I/O event loop if at all possible, and stop thinking about blocking them.

deleted 3y ago

[deleted]

bjourne 3y ago

> do something with garbage collection (but what? I’m confused about this still.) It's an almost hilarious technique. In a garbage collected, multi-threaded virtual machine the runtime may at some point decide it is time for garbage collection. However, all threads must first be stopped because allocating memory while the garbage collector runs leads to badness and/or the garbage collector moves objects referenced by threads. There is no direct way to signal to a thread that it should stop and having the thread poll some flag would hurt performance. Instead the compiler inserts dummy instructions (safepoints) at the top of every loop body or function entry point which writes to a page the runtime controls: mov [addr], eax Then when it's time to stop all threads, it just write protects the page! Every thread eventually stumbles upon the bomb. Boom! SIGSEGV! Boom! SIGSEGV! Boom! When every thread has exploded the garbage collector can run in peace. Then the runtime removes the write protection and resumes executing all threads. Something very similar is used on Windows (though I think it's called events and not signals) and sometimes you use read-protected pages rather than write-protected ones, but the principle is the same.

hayley-patton 3y ago
You probably want to read and not write, as writing will cause a lot of cache ping-pong and slow the mutator down when it doesn't SEGV on the dummy access <https://users.cecs.anu.edu.au/~steveb/pubs/papers/yieldpoint-ismm-2015.pdf#page=8 https://users.cecs.anu.edu.au/~steveb/pubs/papers/yieldpoint...>
tubs 3y ago
Have you got any links to how they continue the threads after the gc has run? I was looking for something similar but couldn't find much.
- themoonisachees 3y ago
  Naïve approach is to have each segv handler poll something to tell when GC is done, and simply return when it can. State is then restored by the kernel.
- jlokier 3y ago

dekhn 3y ago

Signals scare the hell out of me- that was the first point in learning about UNIX programming that made me question everything I'd learned about what is available to you as a function implementor/UNIX user in user space. It got even worse when I moved to threads and sudddenly I couldn't reconcile threads with signals (which are very process-oriented). Reentrant code gives me the same willies- or more correctly, it was a big surprise that people wrote code that depended on static values to store state between invocations.

c-cube 3y ago

I'm surprised that SIGPIPE is not mentioned (and barely present in comments). It's 90% of why I'm scared of signals and every time I write something mixing IO and threads it eventually rears its disgusting head. Can someone shine a light on what the heck it's designed to do?

convolvatron 3y ago
if you have a piped series of processes at the command line, and the middle process dies, how does the earlier part of the pipe get cleaned up? more tangentially, the older inetd model of spawning a process per connection used it for cleanup as well.
- c-cube 3y ago
  The earlier part tries to write, gets `0` or `-1` with an error, concludes the pipe is closed, and exits? The doc for `write` even specifies the error code `EPIPE` that is set alongside with the `SIGPIPE`. Why the signal, when it's redundant?

doubloon 3y ago

i built a robot with two processes. one process is the brain, and all the brain does it wait for SIGUSR1. when it stops getting SIGUSR1 for like 2 tenths of a second in a row, it "ceases all motor function". the other process, which takes input and game controller stuff and sends camera feeds and turns on lights and measures temperature and blah blah blah, it continuously sends a SIGUSR1 to the brain process. it's like a dead man's switch for a robot. doesn't matter what goes wrong, basically when it stops sending SIGUSR1 then it means bad things have happened and the robot stops itself. now what if it gets stuck in a loop continuously sending SIGUSR1 erroneously? yes, that could be a problem, but it's never happened. unlike before i implemented this where, say, interference with the wifi or dead batteries or whatever could leave the robot running away in 'move forward mode' forever.

zX41ZdbW 3y ago

We are using a fuzzing method in ClickHouse that involves sending signals to random threads: https://github.com/ClickHouse/ClickHouse/blob/master/src/Common/ThreadFuzzer.h https://github.com/ClickHouse/ClickHouse/blob/master/src/Com... We immediately found bugs in many dependent libraries of one of three types: 1. A library does not check for EINTR. 2. A library that waits for something does check for EINTR but does not update a timeout, therefore stopping sooner than needed. 3. A library that waits for something does check for EINTR but incorrectly updates a timeout and waiting indefinitely.