9 ms·
Killing processes that don't want to die
- anon49124 8y agoMost old-school shutdown init scripts called the following at some point before shutting down or rebooting: kill -TERM -1 # sending all processes the TERM signal... # ... sleep $SOME_SECS # ... kill -9 -1 # sending all processes the KILL signal... # ... sleep $SOME_MORE_SECS Also old-school local DoS from root: # used to be very bad rm -rf / ; kill -9 -1 Another classic: # bash fork bomb :(){ :|:& };: # creates a function : and calls it # spawns a background process that pipes to itself, which creates another process... effectively two recursive calls per call.
- adrianratnapala 8y agoWhy are the pipes useful for the fork bomb? Or is it just a visual thing?
- hayden592 8y agoArticles like this give me a better understanding of computers and how they fundamentally work. Thank you.
- NullPrefix 8y agoCan it kill sync process?
- jacobparker 8y agoFun article! A modern way to do this on Linux (specifically) is to use PID-namespaces for isolation. You can pass CLONE_NEWPID to the flags argument of clone(2) so that the new process is PID 1 in a new PID-namespace. One of the special properties of PID 1 is that when it dies all of other processes in the PID-namespace will receive SIGKILL. So, create a new "shim" process in a new PID-namespace and have that process fork&exec the rogue program (it's probably better to fork&exec than just exec because of the special properties of PID 1 that the rogue programming you're running may not be built to handle.) If you want to kill the rogue program, just kill your shim. You need to be root (have CAP_SYS_ADMIN) to create a new PID-namespace. If you're not (or more likely don't want to be) you can first create a new user-namespace with the CLONE_NEWUSER flag. You'll be root in that user namespace and can create a new PID-namespace. You'll probably also want to launch the rogue program as a non-root user though, so make sure to figure that out. Docker does this sort of stuff. man pages: * http://man7.org/linux/man-pages/man7/pid_namespaces.7.html http://man7.org/linux/man-pages/man7/pid_namespaces.7.html * http://man7.org/linux/man-pages/man7/user_namespaces.7.html http://man7.org/linux/man-pages/man7/user_namespaces.7.html * http://man7.org/linux/man-pages/man7/namespaces.7.html http://man7.org/linux/man-pages/man7/namespaces.7.html * http://man7.org/linux/man-pages/man2/clone.2.html http://man7.org/linux/man-pages/man2/clone.2.html
- mehrdadn 8y agoWindows has job objects that work nicely for this purpose. :)
- lostmsu 8y agoIsn't their purpose basically the same as process groups?
- mehrdadn 8y agoI think they're similar, yeah, though they seem more general/orthogonal... e.g. they allow a process to be associated with multiple jobs (or so I think -- I've never needed to associate 1 process with multiple jobs), and this is independent of whether the process is detached and starts a new hierarchy or not.
- CamperBob2 8y agoHow is that not a fork bomb? ("The author disagrees with this characterization, which was added by an editor late in the publication process.")
- nneonneo 8y agoThe parent always exits (fork() returns the nonzero child pid in the parent, and 0 in the child), meaning that at any point there’s only one process that is actively forking. A typical fork bomb, on the other hand, floods the system with processes that are all attempting to call fork, resulting in an exponentially growing set of processes until the system runs out of resources.
- CGamesPlay 8y agoBecause the number of live processes never exceeds 2, and fork bombs typically refer to DoS attacks by exhausting the PID space.
- deleted 8y ago[deleted]
- userbinator 8y agoI propose the term "forkworm", or perhaps "forxolotl"[1] for these processes that continuously reproduce but don't grow exponentially. [1] https://en.wikipedia.org/wiki/Axolotl#Regeneration https://en.wikipedia.org/wiki/Axolotl#Regeneration
- caf 8y agoIt already has a name - processes like this were historically called "comets".
- OskarS 8y agoThis is a fork bomb: while (1) { fork(); } This will fork forever, creating children that fork forever, exhausting the system's resources. The example in the article only ever has at most 2 processes, and they don't exhaust anything.
- daleon 8y agoIs there really no cross-platform way to do this that works o both *nix and windows?
- dagenix 8y agoI dunno what you are going for - processes function pretty differently on those systems, so, no, there is not a cross-platform way to handle that case since process handling is inherently system specific. If you are asking if there is some library to do this type of thing - maybe? I dunno. But, even if there is, the articles is talking about how such functionality can be implemented on Linux - which, if such a library existed, would be something that said library would implement.
- boomskats 8y agoI'm not sure you fully appreciate the context of the article is around the linux kernel, on LWN.net (a linux news site). The article has no reason to talk about Windows. If you're really that keen on Windows and need to implement process management functionality in a crossplatform tool, take a look at the Python psutil library, or the all- but-obsoleted SIGAR. That's not really what this article is about though.
- lozenge 8y agoThere isn't even a cross platform way to start a process, never mind killing one. In Windows you can launch the process in a paused state then assign it a job object which all sub processes will inherit. Then the job object has a function to kill all processes in it.
- johannkokos 8y agoI cannot get the first simple example working. int main() { while (1) { if (fork()) _exit(0); } return 0; } If I run `htop` immediately after typing `./a.out`, it shows a error, Could not create child process - exiting fork: Resource temporarily unavailable However, if I wait a second or more, I cannot find `a.out` in the process list.
- nytopop 8y agoYou can apply a filter in htop before running your binary, it'll only show matching processes. I think the hotkey is F4.
- jakeogh 8y agoI increasingly find myself looking at pledge... https://man.openbsd.org/pledge.2 https://man.openbsd.org/pledge.2 It's objectively the right model, if my assumptions are wrong, the program fails. I like that. Maybe it could be taken further... to lie to the app and report back about it's sandbox... or halt and log it's image.
- cyphar 8y agoAs a complete aside, I'm happy that they added "execpromises" to pledge. It was one of the main things that really concerned me about its usage as a sandboxing system (seccomp I think is still more powerful but I will be the first to admit it's ridiculously complicated to get right).
- tux1968 8y agoWonder how hard it would be to provide a Pledge API which simply passes through to seccomp.
- ori_b 8y agoIt's possible, but it's fairly hard. You'd need a second process to supervise the seccomp'ed process, watching the paths that it accesses with ptrace, and allows/disallows the actions based on inspection of the process state. The problem is that seccomp, as far as I'm aware, does not provide you with easy access to the actual arguments of system calls if they are not pointers. The actual pointed-at values of the arguments are required for emulating pledge.
- Ua2Reemo 8y agoYeah, the seccomp limitation that it can't deref pointer arguments makes things a lot less elegant than pledge. But at least unveil can be implemented via mount namespaces.
- cyphar 8y agoThe need for ptrace is soon going to change with the new seccomp userspace notification API (still undergoing review) which allows a userspace process to get notifications when seccomp is tripped and decide what the return code of the syscall is (you can also give it an fd which is copied to the process unix socket style). This would allow you to use process_vm_{read,write}v without the need to mess around with PTRACE_SEIZE. The main usecase for this is container runtimes (emulating privileged operations like auto-loading of kernel modules), but it could also be used for this an a whole host of other really neat features. (Huge shout-outs to Tycho Anderson for working on this for quite a while!)
- mjevans 8y agoA namespace freeze option for halting app process might be a helpful feature in these cases. However this still doesn't solve the Zombie process problem where something has blocked on IO/another resource. Failure should absolutely always be an option.
- mnarayan01 8y agohttps://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-s...
- fake-name 8y agoStep 1: Install a kernel driver. Step 2: Do blocking IO in the kernel that will never finish. Step 3: Your process is now literally impossible to kill in any way. This works in windows too (for similar drivers). This happens fairly regularly with devices with crappy drivers. If something gets wedged in kernel IO, there is literally nothing you can do to kill it, bar rebooting. IIRC, the kernel won't do anything until the IO finishes. When shitty <hardware driver tool> craps out this way, it's extremely annoying. I have semi-fond memories of trying all sorts of interesting tools that attach to a process and inject faults to kill a wedged TV-Tuner application when I was in high school. I never found a option that works. Literally zero of the examples here for preventing unkillable processes would work, too. If you can load kernel drivers, you're hoseable.
- pjc50 8y agoYes, this is the most common way of accidentally getting into this state. Certain combinations of NFS options can also cause this, or even worse produce a filesystem where any process that looks at it gets permanently stuck in "D" state.
- vesinisa 8y agoHaha, had exactly this happen to me, which is why I switched to CIFS / Samba. I don't know what I was doing wrong, but each time the LAN connection to my file server broke, NFS would enter a state where any attempt to touch the file system - including umount - would just freeze the process and make it uninterruptable. This made even turning the computer off safely impossible, since init would try to unmount the NFS file system and hang. Samba seems to have saner defaults, and does not hang if the connection is interrupted. I guess NFS was written with C and A but no P of the CAP theorem in mind, or I had it just misconfigured?
- pjc50 8y agoNah, NFS cares neither for C nor A nor P; it's almost the simplest possible protocol, with primitives of "read/write the block at offset X from inode N". There are all sorts of hacks due to the only reliable atomic primitive on NFS being "rename". Technically it makes no effort to solve either A or P - all the data is stored in one place, with maybe a small local dirent and block cache. There's only one server (so no "P") and no failover (no "A").
- the_ed 8y ago* 'Kill Dash Nine' by Monzy starts playing
- emilfihlman 8y agoIt's absolutely infuriating that you can, with a simple NFS, cause an unkillable, unremovable, always there process to exist. There is simply no reason to require this.
- giovannibajo1 8y agoFor non vital filesystems, I use "-o intr,soft,timeo=5" at NFS mount time. This causes syscalls to return EINTR and timeout without hard locking. It can sometimes confuse userspace, but at least you don't get a stuck process.
- pjc50 8y agoI always preferred "intr,hard", because having filesystem operations fail really stresses out a lot of userspace applications. Most developers haven't thoroughly tested their handling of "what if this disk operation fails?", because it's a pain to do and a rare case.
- MaxBarraclough 8y agoCan't say I took much away from that article. Next time my RabbitMQ broker goes nuts and refuses to die (thank you Erlang), I'm still going to go ahead and reboot the whole machine.
- known 8y agosudo dmesg -w -H --level=err displays the exact problem
- taejo 8y agoOur product occasionally does this at the moment. We've set up asserts in our application to launch a "phone-home" process (on the theory that launching a new process is more likely to succeed than trying to phone home from the borked process). However, the phone-home program uses the same assert library, so the rare assertion failure in the phone-home will launch a new phone-home.
- emmelaich 8y agoA comment in the article has it right -- use SIGSTOP not SIGKILL. In a tight loop.
- RickJWagner 8y agoHacker News paydirt! I love an interesting programming article, this is a nice read. Kudos to the author.
- VLM 8y agoDuring "its bad, but not that bad" situations over the last couple decades I've had positive results with dropping the init level to single user mode "1" and then going back to normal multiuser mode which historically could be any number "2", "3", or "5". Obviously this doensn't help with processes stuck in a kernel call. But for other sorts of malfunctions, or simply testing init scripts and their dependency tree, there's really no reason to reinit the kernel and check the filesystem and all that, and it can be very fast compared to a complete boot process. This is not exactly a "gui-generation" solution for problems, but it has occasionally helped very quickly clean things up.