9 ms·
Unix Structured Concurrency
- gpderetta 3y agoAlso relevant: "The Unix process API is unreliable and unsafe" (https://catern.com/process.html https://catern.com/process.html)
- josephcsible 3y agoIf you're on Linux, you can use PR_SET_PDEATHSIG to accomplish exactly this goal.
- adrusi 3y agoI had a hell of a time a few months ago debugging why my child processes were dying, before learning that PDEATHSIG=9 (don't ask) kills child processes when the thread that created them in the parent process exits. My debugging was not aided by the fact that disabling the code where I set PDEATHSIG had no effect, since someone else's code was invisibly setting it regardless.
- gwright 3y agoI'm a little rusty here, but I think you are talking about the fact that the exit status of the child process has to remain available until the parent process can reap it. Until that happens the child process is in the zombie state. You indicated that the creating thread needed to exit, but that seemed a bit too specific to me, I think that any thread can reap the exit status of a child process. Related to this is the double-fork pattern to avoid zombie processes (and a couple other issues) when initiating a daemon process.
- adrusi 3y agoThe surprising behavior was that the child process received the configured signal not when the process that created it exited, but rather when the specific thread that called fork(2) exited. The parent process was an event loop based python program whose main function was to manage the creation and deletion of these child processes, and the simplest way to spawn child processes without blocking the event loop is to call fork(2) on a thread pool. My thread pool was triaging the number of worker threads based on demand, so occasionally it would decide a worker was no longer needed, and all the child processes that happened to have been created on that thread would get SIGKILL'd — something you rarely want when using a thread pool! I didn't want the child processes to die unless the parent process's business logic decided they were no longer needed, or if the parent was itself killed (this latter reason being the motivation for setting PDEATHSIG). Once I understood why my processes were dying, the solution was simple: make sure the worker threads never exit.
- jcranmer 3y agoNo, what they're talking about is the fact that in Linux, sometimes "process" in the documentation actually means "thread", and this is particularly true if you're doing mildly funky process management APIs.
- epaulson 3y agoIt is kinda bonkers that after 30 years Linux still doesn't some kind of inescapable and recursive process group mechanism. How many tens of thousands of hours of programming time would have been saved over the years if there was a way to: 1. Your process can create a new process group ID (and write it out to stable storage if it wants) 2. add an argument to fork so the same process can pass that ID into fork and the child of the fork wakes up permanently in that group, and any child processes later created by that child will also be in that group 3. Any new groups created by any process in the group are always contained in the original group, you can never leave a group but you might be in multiple groups. 4. There's a way to kill an entire group and verify that everything for an identifier is dead. I think Solaris and some of the BSDs had something like this, so I wish Linux would add one too (though, I guess Linux has managed without one for so long that maybe that's proof enough that it's not really needed, and worst case you can always just reboot the box - nuking from orbit is the only way to be sure)
- gpderetta 3y agoNo need for nuking from orbit. Boiling the ocean (i.e. spawning a VM) is also an option.
- throwaway46281 3y agoWhat about PID namespaces? (You can use those without using a full-on container solution like docker or podman)
- thayne 3y agoRequires root, or use of user namespaces as well. You also need a process that can act as pid 1, with the special semantics that involves.
- dpc_01234 3y agoIt does? Cgroups. The problem is that Posix doesn't have them, so it's not portable.
- 3y ago
- gavinhoward 3y agoI have solved this problem, except for grandchildren processes. (Use some non-portable construct if you must.) This is what I did: I have one thread whose sole purpose is to wait on children. It's activated by a semaphore. But it will wait and then send a message through a pipe to the thread that started that child when the child exits. (As a side benefit, the launching thread can then use poll() or other fd waiting.) The killer is killing children; I found that it is best to kill all children that haven't been waited on when killing is necessary. The problem is that child exit messages can block the pipe, so you need to take them out, but they may arrive in a different order than you expect. If you're just killing all children launched by that thread, no problem, just read and ignore every time you kill a child. There are a lot of details (that code has the single longest code comment I've written in my life), but most of them are in that classic "Gotchas" document matklad references.
- PeterWhittaker 3y agoBut as the original article mentions, what about the case where your parent process is killed uncatchably, e.g., with kill -9? Your children may persist, because the parent was not afforded an opportunity to kill them.
- gavinhoward 3y agoNotice that I said I didn't solve the grandchild problem.
- mksybr 3y agoDoesn't kill -9 impact the ability to kill even children, due to not being given time to clean up?
- gavinhoward 3y agoYes, but at that point, it is the problem of the grandparent, which isn't dead.
- alsuren 3y agoThis sounds like a really good pattern, and something that deserves to become a modern standard (like https://no-color.org/ https://no-color.org/). I feel like this should be made opt in somehow. If you are a noninteractive process and stdin is a tty then you probably shouldn't be swallowing input. I frequently blind-type the next command while a long running command is active, because well behaved noninteractive programs don't swallow input from stdin.
- pjfin123 3y agoIs having a "process manager process" a good pattern? This process would have a record of all currently running processes and their children. When you want to create a new process you make a request to the process manager process. When a process is killed the process manager could then kill its children.
- cgh 3y agoWhat happens if/when the process manager process is abruptly killed?
- ajross 3y agoThis looks entirely wrong to me? The stdin file descriptor isn't "owned" by or associated with the parent process at all. It's whatever the parent decided to set up at fork time, and by default it's just "whatever the parent's parent set up for IT". That is absolutely not guaranteed to be closed when the parent exits. It's true that if you spawn something interactively from the shell and kill it, the shell will clean up the file descriptors before returning to read the next command. And that will look like a close to any children that might have launched. But redirect stdin from a real file and no such close is going to happen. Likewise launcher tools like systemd that want to parse the process output and control it have behavior that tends not to be shell-like.
- tedunangst 3y agoI think the article is unclear. The pattern is to create a pipe and hook that up to stdin.
- tedunangst 3y agoIf you do this, and you should, you should probably set up stdin in the child to be a pipe, not whatever stdin of the parent is, implied by the comment about /dev/null.
- xenadu02 3y agoNormally if the process group leader dies the system will send SIGHUP to all the members of the process group. The default behavior is to exit. So on Unix systems the first process calls setpgrp() and becomes the leader of a new process group. All of its children will, by default, be members of the group, and their children, and so on. When the leader dies they'll all get SIGHUP and die as well. They have the option (if needed) to ignore that signal or spawn their own process group. Session leaders are similar in some ways. When a session leader dies SIGHUP is sent to all members of that session's foreground process group and its controlling tty is released but IIRC it doesn't do anything about background process groups. (Session leaders control this with tcsetpgrp()).
- petters 3y agoIf this works, why does the featured article exist?
- tedunangst 3y agoNot every process wants to be the leader of its own process group.
- Animats 3y agoI'm puzzled, too. The example is about spawning threads in Rust, not processes under UNIX. In Rust, if the main thread dies, all other threads get killed. So child threads outliving the main thread is not a problem. A harder problem is making sure that the program shuts down if some thread panics. There are several ways to do that, but it's not automatic. Incidentally, regular mutexes are "poisoned" if the thread holding them panics. Anyone trying to lock the mutex gets an error. However, parking_lot mutexes do not do this. If the thread holding one of those panics, the mutex unlocks. This may be unsafe.
- neandrake 3y agoI think the example is the code for a child process which must use a separate thread to block on stdin for the whole process. As soon as the parent process dies (no example code?), the connection to the child process’ stdin will be unblock, causing the child monitor thread to detect and panic.
- Permik 3y agoThe comment about '/dev/reads-and-writes-block-forever' intrigues me, as I think there could actually be lots of different use cases for this hypothetical '/dev/blocking' device. For instance, it could be used to intentionally plug the stdin of a process like described in the blog post. Another way that it could be beneficial would be debugging async yielding of blocking operations. Can't think of any better examples, am I totally off the hook?
- chubot 3y agoHm interesting, although I've never come across the need for this. I think it arises in a fairly specific set of circumstances - You're running unit tests for a compiled language. Broken code may crash - You also need to start processes from the same process. In contrast, I basically never have unit test processes that themselves start processes. Instead all unit tests are a single process, and started with a shell script. The shell is responsible for waiting, and handles concurrency. Although I guess the third consideration is - the tests need to work on multiple platforms I can sorta see that, though I still think it's fairly niche, because I think that starting both processes and threads should be parameterized. Library code generally shouldn't start processes or threads -- that should be a policy that you pass in. --- I have come across cases where I'm specifically testing code that starts processes (a shell!), and you get orphans if either the test code or the tested code isn't quite correct. However I'm pretty sure the pattern in the OP would not be enough in those cases -- as mentioned, it requires some cooperation from children and parents. And BTW the general problem of containing adversarial processes is impossible in Unix, and would require Linux cgroups (mentioned in the articled). There's an inherent race between enumerating children and killing processes. A malicious process can always escape your attempts to kill its children. To be honest, what I started doing more is running tests in CI, so there is another layer of cleanup. I think the bottom line is that tests that start processes are very much special cases, and should be separate from most of your test suite.
- gpderetta 3y agoI don't know about unit tests, but I often write regressions test in python that start and orchestrate a dozen or so of subprocesses that communicate with each other. Making sure everything cleans-up properly no matter what is a pain.
- Izkata 3y ago> Sadly, there’s no > /dev/reads-and-writes-block-forever Maybe not in /dev/, but if you make two named pipes ("mkfifo") you can read from one and write to the other, and they'll block since nothing is attached to the other end. Not perfect, but possible.
- hawski 3y agoFor read yes, but for writes it will not block until the pipe buffer is filled.
- shmerl 3y agocgroups should be fine.
- malkia 3y agoEasy on Windows - Job Objects.