10 ms·
The Unix process API is unreliable and unsafe (2021)
- dataangel 4y ago> Shell scripts make starting processes trivial, but it's almost unthinkable that, say, bash, would integrate functionality for starting containers, so that every process is started in a container. Doooooooo it
- edgyquant 4y agoIs it me or does this not make sense? Bash glues and pipes together commands, has network access etc. every process being a container would require either knowing all commands and being able to ensure containers have proper access (even across pipes) or that containers were so open as to defeat the purpose.
- lozenge 4y agoThe problem is every executable can impersonate the user, it has access to do anything the user can do, including deleting or encrypting all their files, reading ssh private keys etc. Network access is rarely concerning unless the program has access to credentials.
- nyrikki 4y agoNothing is stopping you from using namespaces, and containers are just namespaces with cgroups etc But containers aren't jails, pid and uid remapping is just remapping. A huge problem container has to drop capabilities on the honor system. In the default docker mode, running as root, anyone who can launch a container can read from any block device if they don't drop the mknod capability as an example. Actually a privileged container can update the bios or even load arbitrary kernel modules in the host context or change kernel parameters as it is a shared kernel. I tried to get the docker folks to add a conf option disallow privileged container but they refused. You can run in user mode now but most people want persistence and other features that don't allow for that. The important point is if you assume containers are a security feature you are going to have a bad time. Jails were bad enough and containers are just one step up from chroots as far as security go. namespace isolation is the main benefit of containers. Selinux and apparmor are far more appropriate than containers for the security concerns. While I don't personally like selinux, apparmor profiles are pretty easy to write.
- nyrikki 4y agoPlus the 'leaks' in the Linux process API is even worse as each container may run its own tiny-init Containers make the first point of the OP far worse by adding way more pid namespaces.
- Karellen 4y ago> The problem is every executable can impersonate the user, Um, what? What do you mean by "impersonate" here? What does a process that does not impersonate the user look like? Do you just mean "executables that run as the user"? When you log in, and a shell is started that runs as you, is that shell impersonating the user? When you execute commands, as yourself, those commands run with your credentials. Because you ran them. Isn't that, like, the point?
- dllthomas 4y agoTypically, any program I run has the totality of my (regular user) authority, which may let it do things I did not intend. Related: https://en.wikipedia.org/wiki/Ambient_authority https://en.wikipedia.org/wiki/Ambient_authority https://en.wikipedia.org/wiki/Confused_deputy_problem https://en.wikipedia.org/wiki/Confused_deputy_problem https://en.wikipedia.org/wiki/Object-capability_model https://en.wikipedia.org/wiki/Object-capability_model
- derefr 4y ago> What does a process that does not impersonate the user look like? A command running inside a virtual machine, maybe?
- eru 4y agoOr check out how iOS and Android use permissions.
- traverseda 4y agoFirejail
- bombolo 4y agoUse firejail then
- klooney 4y agoBubblewrap is a good tool in this space https://github.com/containers/bubblewrap https://github.com/containers/bubblewrap
- wmf 4y agoMaybe cgroups would be better than full containers here.
- GauntletWizard 4y agoWhich cgroups? Containers are not actually a thing in kernel-land. They're a combination of Process, Network, User, and other namespacing.
- wmf 4y agoNo, cgroups are a separate API from namespaces. https://man7.org/linux/man-pages/man7/cgroups.7.html https://man7.org/linux/man-pages/man7/cgroups.7.html
- GauntletWizard 4y agoYou're not wrong, but the point remains - Are you going to limit their CPUs? Are you going to limit their RAM? Network Performance? The collections of Cgroups and Namespaces (and for all that they are different APIs, you almost never use one without the other, so perhaps it's bet to refer to the whole group of them as "Containers" or "Containment" to differentiate it from Docker-style containers) is complex and flexible for a reason, even if an absurd proportion of the common cases can be solved with a reasonable set of defaults of them.
- ryukafalz 4y agoBash may need a high level of authority as it's spawning processes and wiring them together, but those processes don't necessarily need to have as much access as you do themselves. Take the venerable `cowsay` for example. Currently, running `cowsay` (as with any other program) can cryptolocker your hard drive, delete all your files, reach out to arbitrary servers on the internet, etc. But how much access to your system does it actually need to do its job? Well, mostly... STDIN, and STDOUT, really. Yes, actually doing this is complicated. Another reply has linked to some broad info about object-capability security, but here's a good introduction to the subject: http://habitatchronicles.com/2017/05/what-are-capabilities/ http://habitatchronicles.com/2017/05/what-are-capabilities/ ...and this paper is an excellent deep dive into a capability system: https://mumble.net/~jar/pubs/secureos/secureos.html https://mumble.net/~jar/pubs/secureos/secureos.html
- mattpallissard 4y agoDone. https://pallissard.net/2022/06/27/limiting_application_resources/ https://pallissard.net/2022/06/27/limiting_application_resou... Tl'dr two functions "dispatch" that calls systemd-run and "wrap" that takes a command, a memory limit, and a cpu limit.
- nine_k 4y agosystemd is not bash. Otherwise indeed true.
- not_enoch_wise 4y agoyou're exciting me
- userbinator 4y agoIs "unreliable and unsafe" the new "considered harmful"? Because it sure feels like that.
- calt 4y agoI think it's quite a bit more descriptive and objective than "considered harmful."
- jamesdutc 4y agoI recently wrote an autorunner[1] (like Entr[2] and Watchexec[3]) so I have some recent exposure to this problem. (I will be releasing it on Github shortly.) My autorunner allows running interactive programmes, so it is very sensitive to lingering child processes. For the purposes of the autorunner, I use approach 1.1.3 (“always write down the pid of every process you start, or otherwise coordinate between A and B”) and leave it to the user to figure out what happens if the child process misbehaves with relation to any processes it starts. However, I want to point out that approach 1.1.4 (“A should run B inside a container”) is easier to do than one might expect, and I'd like to plug one of my favourite utilities—Bubblewrap[4]. The Bubblewrap documentation says “[y]ou are unlikely to use it directly from the commandline, although that is possible” but I have built some amazing little tools from it. Try the following invocation: bwrap --ro-bind / / --proc /proc --unshare-pid ps This launches `ps` in a PID namespace with a new `/proc` (since `ps` will read from the host proc otherwise) and the root filesystem mounted readonly. Any procesesses within the PID namespace should have been created by the immediate command that `bwrap` launched. There are also flags `--die-with-parent` and `--as-pid-1` which can further reduce runtime overhead. If you really need a supervisor process, this can be as simple as a `/bin/sh` script that `kill TERM --timeout 1000 KILL` in a loop on everything it sees in `ps`.) As you can see, there's a lot you can do with this tool with significantly lower overhead than using Docker. It has been my goal for some time to extract some of the functionality of Bubblewrap into a Zsh extension to allow accessing these mechanisms with even lower overhead. I think the creation of namespaces is a missing primitive in Linux shells, and being able to quickly construct namespaced environments allows for a style of safe, robust, simple shell scripting. e.g., if you create a mount namespace to run your script, you can actually be looser about parameterising file locations (since the namespace can ensure everything is exactly where you want it to be.) [1] https://fosstodon.org/@dontusethsicode/110019380909461936 https://fosstodon.org/@dontusethsicode/110019380909461936 [2] http://eradman.com/entrproject/ http://eradman.com/entrproject/ [3] https://watchexec.github.io/ https://watchexec.github.io/ [4] https://github.com/containers/bubblewrap https://github.com/containers/bubblewrap
- jrootabega 4y agoLooks interesting. Have you needed or found any good ways to detach the wrapped code from the terminal where you first launch the wrapper? (for security mostly) I haven't found a good way to do that with bwrap other than using sudo or su and their pty feature. bwrap's --new-session flag didn't play nice with interactive programs in my attempts.
- kccqzy 4y agoNote that this article is really talking about the general case, but in practice a lot of techniques can work if you have narrower requirements or if you have more control over what you run. For example, in 1.1.4 the author talks about why containers are not a solution giving three distinct reasons. But if we change our perspective a little bit, none of the three reasons are blocking. The first is that it's not easy; but `docker run` or `podman run` is easy. Even systemd units start with separate control groups to allow you to terminate everything at once. The second reason was about gdb; when was the last time you used gdb in production? If you are using gdb someone is interactively using the computer and can be relied upon to clean up processes manually. The third reason is that containers are more heavyweight, but there's no need to make every subprocess a separate container: if multiple processes should be managed as a single unit (including the case when we'd want to terminate a whole group of processes) they should run in the same container. So with a slight change of perspective we find the problem easily solved. It had trade offs but it works well enough in practice that only very few purists have a problem with it. Not to diss on the author—I think this type of perfectionist thinking is illuminating in terms of API design—but pragmatically it's a solved problem.
- wahern 4y agoThe same is true for the process group/session + controlling terminal solution: the solution doesn't work recursively (can't do process management downstream), and it also requires child processes to abstain from changing SIGHUP handler or mask, but in the vast majority most cases none of those limitations are a problem. Combined with POSIX fcntl locks[1] on a PID file, this is my go to generic solution for Unix-portable[2], multiprocess daemons. The amount of code required in the supervisor component is quite trivial, yet covers almost all of your bases. [1] fcntl locks permit querying the PID of the lock holder, so you don't need to write the PID to the file, providing a solution to the PID file race and loaded gun dilemmas. (There's still a race, but the same race exists with Linux containers, and both can be resolved in similar manner--query PID, send SIGSTOP, verify PID association, send SIGKILL or SIGCONT.) [2] One of the crucial behaviors, that the kernel atomically sends SIGHUP to all processes in the group if the controlling process terminates, isn't guaranteed by POSIX, but it's the behavior on all Unix I've tried--AIX, FreeBSD, macOS, Linux, NetBSD, OpenBSD, and Solaris.
- kwhitefoot 4y agoThat was interesting and clearly written, I wish all such articles were as clear.
- nickdothutton 4y agoIt is after reading pieces like these that I'm reminded of how fortunate I am to have had experience of other "serious" Operating Systems, used at scale, in complex and sometimes unfriendly environments. Namely VAX/VMS. Although some might feel the title was a little clickbaity, I enjoyed the article.
- DeathArrow 4y agoVMS was released for x86, so if you miss it you can give it a spin. https://vmssoftware.com/about/news/2022-07-14-openvms-v92-for-x86-announced/ https://vmssoftware.com/about/news/2022-07-14-openvms-v92-fo...
- skissane 4y agoThus far the x86 port is only available to paying customers. x86 hobbyist program is expected very soon now (within the next few days/weeks). Until then, the best x86 option for hobbyist use is probably running the Alpha version under an emulator. (I don't know if any Itanium emulators are available.) Or emulated VAX–OpenVMS for VAX is no longer legally available to hobbyists, but not hard to find if you don't care about the legalities of it.
- icedchai 4y agoAny idea how much a license is? I'd legit buy one for personal use, but if it's thousands of dollars I'm better off sticking with my old Alpha.
- skissane 4y agoAFAIK, VMS Software doesn’t publish a price list. But I expect it would be several thousand dollars. I doubt they really want hobbyists as paying customers because the cost of servicing them is likely to greatly outweigh the revenue potential.
- sitkack 4y agoExcellent article! Thanks for posting it. It outlines all the problems and then offers a solution with this tool (by the author) https://github.com/catern/supervise https://github.com/catern/supervise
- wmf 4y agoI kept expecting Capsicum to step from behind the curtain but no.
- loeg 4y agoCapsicum is about sandboxing code in the same process, not really related to the problem the article is talking about. FreeBSD's somewhat related mechanism to Linux pidfd is pdfork / pdkill: https://man.freebsd.org/cgi/man.cgi?query=pdfork&sektion=2&n=1 https://man.freebsd.org/cgi/man.cgi?query=pdfork&sektion=2&n...
- bolangi 4y agoWhere process supervision is required under unix, you can use systemd, the linux-only solution pushed by redhat, or one of the small supervision suites such as s6 developed by skarnet.org.
- slondr 4y agoWhat happens when s6 crashes, then?
- aidenn0 4y agoDo cgroups solve any of these problems? I was mildly surprised to not see them mentioned.
- jwilk 4y agoThey were mentioned in 2.1.1.
- wmf 4y agoWhere the author talks about containers you can mentally substitute cgroups since Linux containers are cgroups + namespaces.
- rcoveson 4y agoThat's how I look at it too, but lots of people don't look at it that way, hence all the handwaving about "too heavyweight" and "seems like overkill" etc. Largely because of Docker and Kubernetes, many think of a container as all of the following: 1. A cgroup + [all or nearly all of the] unshare-able namespaces 2. A writable, disposable overlay on top of an immutable "image", which may be lazily downloaded and extracted 3. A resource managed by a userspace daemon managed by a userspace utility over a socket 4. Optionally, a seccomp-bpf filter or apparmor profile or something But there's a whole useful spectrum between a vanilla process and a Docker container like that. Lots of points on that spectrum still feel highly container-ized but aren't really much more heavyweight than a vanilla process. Beyond that, in the point about PID namespaces, the author should mention that there are ultra-light-weight init implementations that are barely a factor in overhead.
- remram 4y agoCan you create them at all, as an unprivileged process, or does it need something else to set it up for you like systemd? If you have to rely on systemd then arguably is a systemd solution, not a Unix or Linux one.
- deathanatos 4y ago> 1.1.4 A should run B inside a container I think the author knows this, but you don't have to start a full-blown container if all you want is to solve the article's stated problem of process leaks. Become a new pid NS: point 1, the subprocess.run criticism is fixed (it just works); point 2, I don't believe a pid NS requires either root or a user NS; and all that remains is point 3. It doesn't require you to start a separate init, you can be the init, i.e., whatever your top-level service is. IIRC, the only two requirements is handling SIGTERM (which you should probably already be doing) and reaping reparented orphans who then die. But also dumb-init is available? The article notes using a separate init, too: "This init process will do nothing but increase the load on the system, and it will prevent us from directly monitoring the started processes." and … no? dumb-init, in a container I have here that's run for >2 weeks, has used < 20 ms of CPU time. RSS of 522 KiB. You'll be fine. I'm not sure how it "will prevent us from directly monitoring the started processes" — it would live above you in the process tree. You'd monitor the started process the same way you would any started process. Edit: ah, crap, I've got it wrong. A new PID NS requires root (or user NSes); being a subreaper, I think, maybe does not. But I'm not sure being a subreaper is sufficient; you want the subtree reaped on the subtree root's death. (I'm also not sure that the subreaper approach is sufficient: if the subreaper itself dies, the processes leak.)
- mike_hock 4y agoThe subreaper is also gonna have the same footprint as the pidns init, and is more complicated. It's just as flawed a solution as the other flawed solutions. We can accept the subreaper being bug-free as a requirement for this workaround to be working, but we can't prevent it from being sigkilled.
- 1vuio0pswjnm7 4y ago"I only know one existing solution that fixes all these problems without sacrificing flexibility or generality. Use the C utility supervise to start your processes; for Python, you can use its associated Python library." C utility written in 1999. Last updated in 2001. I'm still using it everyday, not always with multilog and svscan.
- cryptonector 4y agoYup. PIDs are racy unless they are direct children processes' and you've not reaped them yet. And it goes on. Windows has a much better process API, except for CreateProcess() (the less said about which the better). One thing I generally do when I have a multi-process program (one that starts multiple worker processes, say), is to have a pipe with the write end only in the parent process and whose read end the children include in their I/O event loops. That way when the parent exits the children find out and then they too exit. The parent will still try to signal them, but say the parent gets `SIGKILL`ed: the children find out and they exit.
- monocasa 4y agopidfds solve some of those problems.
- cryptonector 4y agoIndeed, they do. One can approximate pidfd in multi-processed programs on OSes that lack it, but that's about it. pidfd needs to be first-class.
- rand_flip_bit 4y agoCurious why you think CreateProcess is worse than fork/exec. Sure it takes about a dozen parameters but is that really the end of the world?!? It’s much much easier to use correctly and doesn’t have nearly as many of the pitfalls as fork/exec. Especially in large processes with lots of memory allocated. I genuinely don’t understand why people dislike it so much.
- jborean93 4y agoMost of the complaints I've seen are about the number of args and the complexity of calling it vs something simple like fork. There are a lot of knobs to turn which you need to be explicit about. That's not even getting into the whole ProcThreadAttributeList and the myriad of options it exposes. In saying all that I do prefer the `CreateProcess*` APIs on Windows vs the POSIX ones but that might be because I understand the former better.
- dataflow 4y agoIt seems there isn't even anything written about FD_CLOEXEC and its associated race conditions either, as far as I can tell. Basically it's impossible to portably spawn a subprocess in a safe manner if you don't have sufficient control over all the code running in your process, because you might duplicate file descriptors into the child that you might not have intended, and that can break things in the parent.
- rwmj 4y agoAIUI the problem there is not FD_CLOEXEC/SOCK_CLOEXEC but the possibility that some library might not be using it? (That is to say, *_CLOEXEC if used does not have race conditions) However we usually cope with that by closing all unknown/unexpected file descriptors after fork and before exec. Linux even has a system call to make that easier: https://man7.org/linux/man-pages/man2/close_range.2.html https://man7.org/linux/man-pages/man2/close_range.2.html
- dataflow 4y ago> AIUI the problem there is not FD_CLOEXEC/SOCK_CLOEXEC but the possibility that some library might not be using it? Not exactly. The problem is that you have to be able to set it atomically from the creation of the file descriptor. Setting it after creation is subject to a race condition where a fork occurs in the interim. There's no portable way to do that, and people often ignore O_CLOEXEC even when there's a platform-dependent way to pass it. (How often do you see dup3() called, for example? And how often do you see higher-level languages and libraries expose this and force callers to make a conscious decision?) > However we usually cope with that by closing all unknown/unexpected file descriptors after fork and before exec. You can't really do that portably (well, maybe unless you want to call close() billions of times). And even if/when you can do that, you run into the reverse problem, where you might close descriptors that were supposed to be duplicated into the subprocess but that you didn't know about. (One example is when a user performs redirect inside a shell like 2>&3 and wants it to work inside a descendant process - you don't want to just randomly close FDs you don't recognize.)
- 4y ago
- ary 4y agoThis links one of my favorite critiques of API design: 'A fork() in the road' https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0... It's very much worth a read.
- evilotto 4y agoIs basic fork/exec from a large process still slow or have newer apis fixed that?
- pengaru 4y agovfork() / CLONE_VFORK / posix_spawn()
- anfilt 4y agoOn linux fork is pretty fast since it's copy on write. Exec or anything similar is gonna be slower. For example you want to exec some program in the elf format. Well exec is likely to lead these things: One is likely got to load the file from non-volatile storage. Then parse the elf file and place each section in the correct place in memory, setup other memory maps, and stack ect.. Not just that though then run time linker probably has to run loading and linking each .so file, also patching code with the correct relocations as needed. Loading a new process is a lot more expensive where as fork can reuse a lot of that work since it's just a clone of the process at the point fork was called and was already done.
- evilotto 4y agoUnless something has changed in the past decade, fork on linux is really slow when you're forking a large multi-GB process because even though the memory is COW all the kernel page tables still need to be copied. This is one reason why large servers will have a separate "launcher" process that is forked early on rather than forking themselves. (the other reason of course is threads).
- anfilt 4y agoWell that is needed to support isolated address spaces. It might be possible to share some the of the page tables safely. Still copying page tables is a lot faster than copying many gigs of data around, but for most that don't have massive page tables this is pretty fast. You can also use things like MADV_HUGEPAGE to make the page tables smaller. Now if this is a problem and the application does not need address space isolation. clone() with the CLONE_VM flag will get rid of this problem, and clone() is nice in the fact you can get more granularity/control. This control basically ranging between most of the continuum between thread and separate process depending on your needs.
- deleted 4y ago[deleted]
- wang_li 4y agoThis reads like they have a set of requirements and since the Unix model doesn’t meet their requirements, the Unix model is bad. As opposed to it’s fine for those who have different requirements.
- Arch-TK 4y agoI think the article is pretty clear on that, if your requirements do not include "reliable" and "safe" then yes, the unix APIs meet your requirements. But if your requirements include "reliable" and "safe" then they definitely don't.
- thayne 4y agoThat's overly general. The unix APIs don't meet requirements if you need to reliably and safely reap child processes, and all of their descendants. And on linux, there are non-posix APIs that help in certain situations (pid namespaces and pidfds). And for many, maybe most, applications, leaking a child process isn't really a problem.
- Arch-TK 4y agoLeaking child processes is inherently unreliable, it can lead to lots of issues in any non-trivial program, but even in the most trivial of programs it can lead to basic problems like accidentally fork-bombing yourself because you made a small programming error. There's no guaranteed way to avoid that, therefore to a lesser or greater extent, depending on the circumstances, it is not reliable. The APIs are inherently unsafe, this is because once you are in a situation where you have leaked a process, you can't safely kill it without risking other processes. In some situations this could even be abused by an attacker to cause other problems. So yes, in the most boring of applications where you have a user in front of a computer, there's no need for the reliability of processes having lifetimes bounded by the references held to those processes over their life. Likewise, there's probably no massive need for safety because if a process gets killed unintentionally, the user can fix it. But just because there exist circumstances where neither reliability nor safety are important, doesn't mean that the APIs aren't inherently unreliable and unsafe. Furthermore, yes, linux has solutions for half of these problems, FreeBSD has solutions for other parts of these problems, Solaris has solution for other problems. The article calls out "the UNIX APIs" for a reason. The original APIs are as bad as the article says, and they have to be worked around with OS specific non-portable APIs. Which are, by definition, not "standard UNIX APIs" anymore.
- jiveturkey 4y agoToo bad the article doesn't discuss contracts, the Solaris solution. As the article is very linux focused, I imagine the author is blissfully unaware.
- Arch-TK 4y agoThe article mentions non-linux solutions to these problems, such as those in FreeBSD, but realistically this is a Unix wide problem as irrespective of what solution any particular flavour of Unix gives for these problems, there's no standard solution.
- jiveturkey 4y ago100% agree on the breadth. However each flavor should learn from what the others have done. Especially if you write about it, as an educational piece vs just documentation, I'm just expressing my disappointment that known excellent solutions aren't part of the discussion.
- gtirloni 4y agoIs Fuchsia any better for what the article is concerned about? https://fuchsia.dev https://fuchsia.dev
- Arch-TK 4y agoWhen quickly glancing at https://fuchsia.dev/fuchsia-src/reference/syscalls/process_create https://fuchsia.dev/fuchsia-src/reference/syscalls/process_c... , It looks like it does not have the same issues.
- shanemhansen 4y agoThe author said something that is technically correct but I feel gives the wrong impression to us folks who may be trying to solve the "kill all child processes" problem. There is a simple bash one-liner to ensure child processes are killed that doesn't require root and it's using one of their examples of a process leak: unshare -U --map-user=$(id -u) --map-group=$(id -g) -f -p sh -c '{ sleep inf &} &' The author doesn't share this example because "user namespaces introduce a number of quirks, such as breaking gdb (by breaking ptrace), so they also can't be used by most users". I disagree that the container/unshare approach can't be used by most users. So strace works just fine on my machine and gdb is able to attach and print a backtrace. Now it's true that gdb itself prints out a warning when I do this: warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable. Connect to gdbserver inside the container. So: - user namespaces do solve the "kill all child processes" problem - strace still works - gdb from outside the namespace isn't fully supported So if you wanted to debug a child subprocess with gdb you'd presumably have to invoke gdb via unshare so it shared the same pid namespace (not tested).
- user3939382 4y ago> There is a simple bash one-liner… > unshare -U --map-user=$(id -u) --map-group=$(id -g) -f -p sh -c '{ sleep inf &} &'
- deleted 4y ago[deleted]
- nh2 4y agoThis does not work for me, at least not in the way most people would expect it to. When I run: > unshare -U "--map-user=$(id -u)" "--map-group=$(id -g)" -f -p sh -c 'command sleep infinity' then Ctrl+C has no effect. Sending SIGTERM to `unshare` has no effect. Sending SIGKILL to `unshare` reparents the inner process `sleep infinity` to my computer's top-level systemd, where it continues to run (process leak). So not sure how that ensures "child processes are killed". You need at least the `--kill-child` flag: > unshare -Ufp --kill-child -- bash -c "command sleep infinity" See https://unix.stackexchange.com/questions/393210/why-does-unshare-based-killing-only-work-reliably-with-fork/393279#393279 https://unix.stackexchange.com/questions/393210/why-does-uns... I added the `--kill-child` flag to unshare because Linux did not offer a reliable way to kill child processes when pressing the "Cancel build" button in my CI pipeline. With the above, SIGKILL against `unshare` will reliably tear it down and everything below it. But Ctrl+C still has no effect, and SIGTERM against `unshare` still has no effect. So I agree with the post author that the Linux process API is unreliable. This stuff should be easy.
- jclulow 4y agoMany of these problems are at least partially solved on other UNIX systems like illumos. To run down the top-level list: 1. It's easy for processes to leak illumos has contracts[1][2][3], which were developed as part of the Service Management Facility[4]. They are another process grouping abstraction that allows SMF to track trees of processes, whether they daemonise or not. They allow for tracking or ignoring certain events (e.g., a fatal signal sent from outside the contract, a process within the contract that aborts and dumps core, if a particular process, or all processes, terminate in some way) and for doing certain kinds of automatic cleanup (e.g., terminate all processes in the contract when the process that owns the contract is terminated). These are managed by the kernel, so they are effectively inescapable even when not held correctly by a user process. 2. It's impossible to prevent malicious processes leaks This is not really true for us either. Between contracts, and resource controls[5], and privileges[6], I expect one would be able to limit the malicious or accidental escape of processes from supervision or any run-away resource consumption caused by, say, a fork bomb. 3. Processes have global, reusable IDs This is true on some level, but in practice I think that's just part of UNIX and when you have solved 1-2 and 4 in other ways, it's not actually that bad. If you want to kill everything in a contract you own, even without knowing the full list of pids, you can do that with ct_ctl_abandon(3CONTRACT)[7] which takes a contract file descriptor. The termination action (which could be to tear down all of the processes) will take effect then. 4. Process exit is communicated through signals This is not entirely true, in the sense that they are by default for classic UNIX applications -- but they need not be. We have forkx(2)[8], which has the FORK_NOSIGCHLD and FORK_WAITPID flags. These request that SIGCHLD is not posted for process termination, and that the classic UNIX wait(2) family of calls will not receive notification or reap children. You can use these from a library, and then manage your own waitid(2) or waitpid(3C) calls on the specific process IDs you are responsible for reaping. You can also use contracts to receive notifications of events about processes within the contract coming and going. [1]: https://illumos.org/man/5/contract https://illumos.org/man/5/contract [2]: https://illumos.org/man/3LIB/libcontract https://illumos.org/man/3LIB/libcontract [3]: https://illumos.org/man/3CONTRACT/ https://illumos.org/man/3CONTRACT/ [4]: https://illumos.org/man/7/smf https://illumos.org/man/7/smf [5]: https://illumos.org/man/7/resource_controls https://illumos.org/man/7/resource_controls [6]: https://illumos.org/man/7/privileges https://illumos.org/man/7/privileges [7]: https://illumos.org/man/3CONTRACT/ct_ctl_abandon https://illumos.org/man/3CONTRACT/ct_ctl_abandon [8]: https://illumos.org/man/2/forkx https://illumos.org/man/2/forkx
- chubot 4y agoDJB's self pipe trick solves the awkwarness of #4 -- waiting for a process exits plus other events non-deterministically: https://cr.yp.to/docs/selfpipe.html https://cr.yp.to/docs/selfpipe.html FWIW a shell is basically two alternating, non-overlapping event loops: - a select() loop on the input terminal FD for getting keystrokes (e.g. GNU readline) - the waitpid(-1) loop for running code, i.e. get the next process that exited It never actually does both at the same time -- it doesn't wait for processes and stream input simultaneously, which is awkward without the self-pipe trick. --- Regarding adversarial processes, yes you need something like Linux cgroups to solve that problem. In traditional Unix, a process that can run arbitrary code can always escape your attempts to kill it. IIRC you can start a Linux process in a freezer cgroup, and stop everything in the cgroup. I recall reading the docs for an HPC platform that does that, and I'm sure Docker does it in some way too. --- I'd be interested in where `supervise` is used in production ... it seems like there is a bigger story behind this article! (copy of lobste.rs comment)
- teddyh 4y ago> It never actually does both at the same time -- it doesn't wait for processes and stream input simultaneously Couldn’t you use a signalfd to receive the SIGCHLD, and listen to both the terminal and signalfd at the same time?
- chubot 4y agoSo there was a correction here, zsh and fish do wait for processes in the interactive part, but apparently not bash/dash/mksh: https://lobste.rs/s/om32da/unix_process_api_is_unreliable_unsafe#c_amz4g4 https://lobste.rs/s/om32da/unix_process_api_is_unreliable_un... As far as your question, shells don't use signalfd() because it's Linux-only! Most shells predate not just signalfd(), but Linux itself :) I wouldn't use it in a newer shell for portability. If it ever came up, I probably I would use the self-pipe trick. BTW I think there was some argument awhile ago that signalfd() is actually quite bad in the presence of threads, but a shell doesn't have threads so that doesn't matter.
- 4y ago
- jmmv 4y agoI went down this particular rabbit hole of trying to terminate processes reliably a few years ago. The context was termination of tasks and tests by Bazel. You can find the text in https://jmmv.dev/2019/11/wait-for-process-group.html https://jmmv.dev/2019/11/wait-for-process-group.html and the two extra linked posts in the last paragraph, one providing a solution for Linux and the other for macOS. It isn’t pretty nor easy, but it’s doable to various degrees of correctness.
- ezekiel68 4y agoThere is zero new information in this article that has not been known since the early days of what became Unix. It seems to me that a reasonable summary of the content might be: "It is difficult to keep track of important things properly and no one should be asked to do it."
- eru 4y agoComputers should help us keep track of things.
- ezekiel68 4y agoAnd sure enough, they do. When I fork() a process, I get a return code which may be a child pid. I can hold it as a variable, put it in a hashmap, save it to a file, output it to a logging stream, kill it, or send a signal to it. The API ain't perfect but the world has not come crashing down upon us due to this in the past half-century as far as I have seen.
- eru 4y agoAren't you proving too much here? The argument you bring here can also be advanced in favour of Windows 95 running stable enough, can't it?
- ben0x539 4y agoSimply use systemd/cgroups they tell me and now I'm trying to figure out how my systemd service somehow ended up controlling processes for another service.
- teo_zero 4y agoI don't understand why the article lists 4 issues, when 2 of them were solved in 2019: > pidfd is a great solution to the third and fourth problems So why talking about them? Is it just for proving the superiority of the author's own utility "supervise"?
- kajaktum 4y agoAside from having to run a short command line program to get an stdout/stderr from; I have never had to use processes. It has always been simple threads. Why and when would I want to spawn a process instead?
- adrianN 4y agoFor example because you want better isolation guarantees than threads provide. Say you spawn workers running requests for different users with different permissions.
- IAmPaigeAT 4y ago> unreliable and unsafe Just like the information you’re providing if you can’t be bothered to setup an ssl certificate for a page