9 ms·

Step 1: Install a kernel driver. Step 2: Do blocking IO in the kernel that will never finish. Step 3: Your process is now literally impossible to kill i

Step 1: Install a kernel driver. Step 2: Do blocking IO in the kernel that will never finish. Step 3: Your process is now literally impossible to kill in any way. This works in windows too (for similar drivers). This happens fairly regularly with devices with crappy drivers. If something gets wedged in kernel IO, there is literally nothing you can do to kill it, bar rebooting. IIRC, the kernel won't do anything until the IO finishes. When shitty <hardware driver tool> craps out this way, it's extremely annoying. I have semi-fond memories of trying all sorts of interesting tools that attach to a process and inject faults to kill a wedged TV-Tuner application when I was in high school. I never found a option that works. Literally zero of the examples here for preventing unkillable processes would work, too. If you can load kernel drivers, you're hoseable.

pjc50 8y ago
Yes, this is the most common way of accidentally getting into this state. Certain combinations of NFS options can also cause this, or even worse produce a filesystem where any process that looks at it gets permanently stuck in "D" state.
- vesinisa 8y ago
  Haha, had exactly this happen to me, which is why I switched to CIFS / Samba. I don't know what I was doing wrong, but each time the LAN connection to my file server broke, NFS would enter a state where any attempt to touch the file system - including umount - would just freeze the process and make it uninterruptable. This made even turning the computer off safely impossible, since init would try to unmount the NFS file system and hang. Samba seems to have saner defaults, and does not hang if the connection is interrupted. I guess NFS was written with C and A but no P of the CAP theorem in mind, or I had it just misconfigured?
  - pjc50 8y ago
    Nah, NFS cares neither for C nor A nor P; it's almost the simplest possible protocol, with primitives of "read/write the block at offset X from inode N". There are all sorts of hacks due to the only reliable atomic primitive on NFS being "rename". Technically it makes no effort to solve either A or P - all the data is stored in one place, with maybe a small local dirent and block cache. There's only one server (so no "P") and no failover (no "A").
  - paol 8y ago
    Nope, you didn't misconfigure it, NFS behaves like that by default. In fact it's really hard to configure it not to that.
    - jandrese 8y ago
      It can be annoyingly difficult to even configure NFS to enable soft-interrupts that actually work. On one hand it is nice because if your fileserver reboots for some reason you can be reasonably sure that you won't lose data or have truncated/corrupted files, but it's annoying if your fileserver goes down hard then it brings down everything else on the network with it.
  - Someone 8y ago
    To learn the basics about NFS, read chapter 14 of the Unix haters handbook” (http://simson.net/ref/ugh.pdf http://simson.net/ref/ugh.pdf) ”By design, NFS is connectionless and stateless. In practice, it is neither. This conflict between design and implementation is at the root of most NFS problems. File systems, by their very nature, have state. You can only delete a file once, and then it’s gone. That’s why, if you look inside the NFS code, you’ll see lots of hacks and kludges—all designed to impose state on a stateless protocol.” It goes on to mention some issues that NFS wouldn’t handle if it were truly stateless and connectionless: ”NFS is stateless, but many programs designed for Unix systems require record locking in order to guarantee database consistency. […] NFS is based on UDP; if a client request isn’t answered, the client resends the request until it gets an answer. If the server is doing something time-consuming for one client, all of the other clients who want file service will continue to hammer away at the server with duplicate and triplicate NFS requests, rather than patiently putting them into a queue and waiting for the reply. […] If you delete a file in Unix that is still open, the file’s name is removed from its directory, but the disk blocks associated with the file are not deleted until the file is closed. This gross hack allows programs to create temporary files that can’t be accessed by other programs. (This is the second way that Unix uses to create temporary files; the other technique is to use the mktmp() function and create a temporary file in the /tmp directory that has the process ID in the filename. Deciding which method is the grosser of the two is an exercise left to the reader.) But this hack doesn’t work over NFS. The stateless protocol doesn't know that the file is “opened” — as soon as the file is deleted, it's gone.”
bArray 8y ago
I was just about to suggest, get your process into `TASK_UNINTERRUPTIBLE` state and watch people squirm [1] [2]. Only reboot to recover. You may think this is an edge case, but I mounted an FTP drive, ran a normal `cp` into it and the internet connection broke temporarily. You literally cannot kill the process without rebooting. Remove the drive, `kill -9`, `sudo` everything - nothing. Caused by a normal user no less! [1] https://stackoverflow.com/questions/20423521/process-permanently-stuck-on-d-state https://stackoverflow.com/questions/20423521/process-permane... [2] https://web.archive.org/web/20131229000010/http://linuxgazette.net//issue83/tag/6.html https://web.archive.org/web/20131229000010/http://linuxgazet...
- AnIdiotOnTheNet 8y ago
  I agree with a comment from the SO thread: "And thus a simple, solvable hardware or locking problem will escalate to a major problem, needing reboot. And the kernel people doesn't even understand, that it is a… a… suboptimal handling of the things" This is an example of the kind of thing that really irks me about developers, where they insist that they know better than the user and artificially constrain them "for their own good". It's one thing to make something difficult to do accidentally, it's another to prevent it entirely. I've run into unkillable task problems many times, and every time it happens on Linux I think back to the 90s when annoying linux evangelists would joke about how Windows required a reboot to fix things.
  - IronBacon 8y ago
    Well, in the 90s Windos usually required a reboot after modify a config setting, like the host IP address...
    - jiveturkey 8y ago
      i’m told it still does if it’s a domain controller
  - wyldfire 8y ago
    I disagree. The defect is not that you can't kill these tasks, the defect is that the I/O wasn't terminated by a timeout somewhere in the kernel/driver. > where they insist that they know better than the user and artificially constrain them "for their own good". I'm not sure where this comes from but if it's the intended design of linux (to not be able to kill tasks in the uninterruptible state), then that's their design. You're free to design your own operating system without this limitation -- though it likely means just taking the other side of the tradeoffs involved here. It's not as if linux designers think "tasks holding resources that can't be killed by a sysadmin are a good thing" rather instead "letting tasks currently executing kernel code to be interrupted is a bad thing." And you could imagine other solutions that might liberate you from the dilemma entirely, but then that wouldn't be linux anymore.
kpcyrd 8y ago
If you are able to load kernel drivers you could just nop out the kill syscall though.
- oblio 8y ago
  For the mere mortals amongst us, how would you actually do that, practically? I.e. what commands?
  - TheDong 8y ago
    There are many options. One new way would be to install a kprobe. You can read more about kprobes here: https://www.kernel.org/doc/Documentation/kprobes.txt https://www.kernel.org/doc/Documentation/kprobes.txt (note, they're not meant for modifying syscall arguments in general, but in the case of kill it will be possible :) Effectively, they let you write code that will run before syscalls and can mutate the state of the kernel and registers and such. You could write a kprobe which changed "kill"s pid argument to some random pid that didn't exist each time the given pid was being killed. You could also just patch the kernel in memory yourself easily enough (which is even easier for a kernel module; look up modfying the sys_call_table; that table exists for a reason). Modifying the sys_call_table is probably what the parent poster was thinking of, and is definitely the "right" way to do this. You could compile a new kernel with the "kill" syscall modified by editing the c code, and then do something akin to what ksplice did to patch the running kernel with that. kexec/kpatch are the modern tools to do that, though neither of them are as good as ksplice is... not that ksplice is actually something you can use without working at oracle on that team. You could also modify the kill binary under the assumption the user is running "/usr/bin/kill", not making the syscall in some other way, and modifying that binary is pretty trivial. These last few are more like what rootkits do and are harder to detect. kprobes and kernel modules are both meant to be visible to users in general (though they can ofc hide themselves with work), but patching binaries on disk and the kernel memory live is a bit less transparent. You could do various other strategies rootkits do, though I'd rather not get too deep into those since those can get relatively complicated (with the ideas presented in the Blue Pill rootkit paper still being some of my favorites). > what commands Asking it like that is less productive. The ideas I mention above should be enough for you to research and do it yourself. The exact details are complicated because this isn't a simple "out of the box" sort of thing to do. Modifying one call in one or two processes with gdb / LD_PRELOAD hacks is easy, but still isn't just "out of the box" since you'll have to write some gdb scripts / a preload shim. Doing it system wide in the kernel is even more complicated and even more resilient to listing commands instead of concepts, and asking "what commands" doesn't really make the answer more meaningful. Me saying "sys_call_table + gcc + insmod" isn't really any more helpful than just saying "sys_call_table". Hopefully the above helped! I recommend doing more research, especially on the sys_call_table, kprobes and, ksplice as those are the most relevant techniques, though realistically only the first two are things you could reasonably do without becoming an expert in patching kernels in memory.
ahartmetz 8y ago
This happened to me once in ~2006 on Linux with a scratched CD. I hope it's been fixed.
- filomeno 8y ago
  I think you're out of luck. I once accidentally scratched a CD and ten years later it only got worse.
  - jfroma 8y ago
    You made me laugh a lot. Thank you
- imtringued 8y ago
  It can also happen if you run Linux on a crappy flash drive.
caf 8y ago
If you have a kernel driver that is giving you this problem, it should be fairly easy in most cases to replace its use of TASK_UNINTERRUPTIBLE with TASK_KILLABLE.
- jakeogh 8y ago
  Why does TASK_UNINTERRUPTIBLE exist? 'I'm sorry Dave'?
  - ovi256 8y ago
    Obviously for the app developpers that write rocket flight computers. They also don't use free, their garbage collection is implemented in a physical mechanism.
  - caf 8y ago
    It's because for a long time, there was only "interruptible" and "uninterruptible", where "interruptible" meant that any signal could interrupt it. Historically, many operations were specified not to be interrupted by signals - reads and writes on disk files (as opposed to "slow devices" like ttys and sockets), mkdir(2), fsync(2), etc. They're not allowed to return EINTR. This meant using an uninterruptible sleep, which has the consequence that SIGKILL can't interrupt it either. That's why the code in top or ps for uninterruptible sleep is 'D' - historically it meant "Disk Wait". TASK_KILLABLE is a more recent invention, which can be interrupted only by fatal signals. The code doing the wait still has to unwind as if it were going to exit from the syscall with an error, but userspace won't see that error because it will die from the signal instead.
  - notriddle 8y ago
    Because the call doesn't support rolling back partial progress. Imagine if you interrupted `mount`, and it had written to the directory table but hadn't yet initialized the underlying filesystem and now the kernel reads uninitialized memory whenever you look at that directory (or, less disastrously, the mutex that protects that table is left locked and you can never mount or unmount anything ever again).
deleted 8y ago
[deleted]
- sitkack 8y ago
  You need to build your own hardware.
ajross 8y ago
Step 2 needs some clarification. It is possible to implement "blocking I/O" in the kernel with the wait_interruptible() family of functions, and many drivers and subsystems do. These have the fairly straightforward behavior that they can return early with an exit code instead of when the given waitq ueue has been tickled. In the kernel, in C, this is pretty obvious stuff. But the side effect of not using the _interruptible variants is that there's no way for the kernel to return except from the intended event, which means that other things that you'd want to produce a return from the kernel context can't happen either, including signal delivery. So to userspace the process becomes unkillable and it looks like voodoo. But at the bottom level all it is in a C API that made promises that weren't kept at runtime.
mrguyorama 8y ago
Interestingly enough, Windows has gotten a little smart about kernel drivers misbehaving. I found out on my Windows 7 machine that if the AMD video driver becomes unhappy in certain ways, Windows will be able to recover it with no blue screen or loss of functionality other than the program who's access to the direct3D device caused the problem no longer having access until it attempts to re-establish a connection or is restarted. An incredibly limited set of circumstances, but I was awestruck to see Windows catch itself hiccuping in kernel space and not explode.
- rusk 8y ago
  and yet, user-space applications can still bring down the whole thing; and just try bringing up Task Manager to kill a task when you need to.
  - mrguyorama 8y ago
    Killing the "task" hasn't worked since Windows 95, but killing the process itself still unwedges nearly anything userspace in my experience
    - rusk 8y ago
      Still need task manager though

pjc50 8y ago

Yes, this is the most common way of accidentally getting into this state. Certain combinations of NFS options can also cause this, or even worse produce a filesystem where any process that looks at it gets permanently stuck in "D" state.

vesinisa 8y ago
Haha, had exactly this happen to me, which is why I switched to CIFS / Samba. I don't know what I was doing wrong, but each time the LAN connection to my file server broke, NFS would enter a state where any attempt to touch the file system - including umount - would just freeze the process and make it uninterruptable. This made even turning the computer off safely impossible, since init would try to unmount the NFS file system and hang. Samba seems to have saner defaults, and does not hang if the connection is interrupted. I guess NFS was written with C and A but no P of the CAP theorem in mind, or I had it just misconfigured?
- pjc50 8y ago
  Nah, NFS cares neither for C nor A nor P; it's almost the simplest possible protocol, with primitives of "read/write the block at offset X from inode N". There are all sorts of hacks due to the only reliable atomic primitive on NFS being "rename". Technically it makes no effort to solve either A or P - all the data is stored in one place, with maybe a small local dirent and block cache. There's only one server (so no "P") and no failover (no "A").
- paol 8y ago
  Nope, you didn't misconfigure it, NFS behaves like that by default. In fact it's really hard to configure it not to that.
  - jandrese 8y ago
    It can be annoyingly difficult to even configure NFS to enable soft-interrupts that actually work. On one hand it is nice because if your fileserver reboots for some reason you can be reasonably sure that you won't lose data or have truncated/corrupted files, but it's annoying if your fileserver goes down hard then it brings down everything else on the network with it.
- Someone 8y ago
  To learn the basics about NFS, read chapter 14 of the Unix haters handbook” (http://simson.net/ref/ugh.pdf http://simson.net/ref/ugh.pdf) ”By design, NFS is connectionless and stateless. In practice, it is neither. This conflict between design and implementation is at the root of most NFS problems. File systems, by their very nature, have state. You can only delete a file once, and then it’s gone. That’s why, if you look inside the NFS code, you’ll see lots of hacks and kludges—all designed to impose state on a stateless protocol.” It goes on to mention some issues that NFS wouldn’t handle if it were truly stateless and connectionless: ”NFS is stateless, but many programs designed for Unix systems require record locking in order to guarantee database consistency. […] NFS is based on UDP; if a client request isn’t answered, the client resends the request until it gets an answer. If the server is doing something time-consuming for one client, all of the other clients who want file service will continue to hammer away at the server with duplicate and triplicate NFS requests, rather than patiently putting them into a queue and waiting for the reply. […] If you delete a file in Unix that is still open, the file’s name is removed from its directory, but the disk blocks associated with the file are not deleted until the file is closed. This gross hack allows programs to create temporary files that can’t be accessed by other programs. (This is the second way that Unix uses to create temporary files; the other technique is to use the mktmp() function and create a temporary file in the /tmp directory that has the process ID in the filename. Deciding which method is the grosser of the two is an exercise left to the reader.) But this hack doesn’t work over NFS. The stateless protocol doesn't know that the file is “opened” — as soon as the file is deleted, it's gone.”

bArray 8y ago

I was just about to suggest, get your process into `TASK_UNINTERRUPTIBLE` state and watch people squirm [1] [2]. Only reboot to recover. You may think this is an edge case, but I mounted an FTP drive, ran a normal `cp` into it and the internet connection broke temporarily. You literally cannot kill the process without rebooting. Remove the drive, `kill -9`, `sudo` everything - nothing. Caused by a normal user no less! [1] https://stackoverflow.com/questions/20423521/process-permanently-stuck-on-d-state https://stackoverflow.com/questions/20423521/process-permane... [2] https://web.archive.org/web/20131229000010/http://linuxgazette.net//issue83/tag/6.html https://web.archive.org/web/20131229000010/http://linuxgazet...

AnIdiotOnTheNet 8y ago
I agree with a comment from the SO thread: "And thus a simple, solvable hardware or locking problem will escalate to a major problem, needing reboot. And the kernel people doesn't even understand, that it is a… a… suboptimal handling of the things" This is an example of the kind of thing that really irks me about developers, where they insist that they know better than the user and artificially constrain them "for their own good". It's one thing to make something difficult to do accidentally, it's another to prevent it entirely. I've run into unkillable task problems many times, and every time it happens on Linux I think back to the 90s when annoying linux evangelists would joke about how Windows required a reboot to fix things.
- IronBacon 8y ago
  Well, in the 90s Windos usually required a reboot after modify a config setting, like the host IP address...
  - jiveturkey 8y ago
    i’m told it still does if it’s a domain controller
- wyldfire 8y ago
  I disagree. The defect is not that you can't kill these tasks, the defect is that the I/O wasn't terminated by a timeout somewhere in the kernel/driver. > where they insist that they know better than the user and artificially constrain them "for their own good". I'm not sure where this comes from but if it's the intended design of linux (to not be able to kill tasks in the uninterruptible state), then that's their design. You're free to design your own operating system without this limitation -- though it likely means just taking the other side of the tradeoffs involved here. It's not as if linux designers think "tasks holding resources that can't be killed by a sysadmin are a good thing" rather instead "letting tasks currently executing kernel code to be interrupted is a bad thing." And you could imagine other solutions that might liberate you from the dilemma entirely, but then that wouldn't be linux anymore.

kpcyrd 8y ago

If you are able to load kernel drivers you could just nop out the kill syscall though.

oblio 8y ago
For the mere mortals amongst us, how would you actually do that, practically? I.e. what commands?
- TheDong 8y ago
  There are many options. One new way would be to install a kprobe. You can read more about kprobes here: https://www.kernel.org/doc/Documentation/kprobes.txt https://www.kernel.org/doc/Documentation/kprobes.txt (note, they're not meant for modifying syscall arguments in general, but in the case of kill it will be possible :) Effectively, they let you write code that will run before syscalls and can mutate the state of the kernel and registers and such. You could write a kprobe which changed "kill"s pid argument to some random pid that didn't exist each time the given pid was being killed. You could also just patch the kernel in memory yourself easily enough (which is even easier for a kernel module; look up modfying the sys_call_table; that table exists for a reason). Modifying the sys_call_table is probably what the parent poster was thinking of, and is definitely the "right" way to do this. You could compile a new kernel with the "kill" syscall modified by editing the c code, and then do something akin to what ksplice did to patch the running kernel with that. kexec/kpatch are the modern tools to do that, though neither of them are as good as ksplice is... not that ksplice is actually something you can use without working at oracle on that team. You could also modify the kill binary under the assumption the user is running "/usr/bin/kill", not making the syscall in some other way, and modifying that binary is pretty trivial. These last few are more like what rootkits do and are harder to detect. kprobes and kernel modules are both meant to be visible to users in general (though they can ofc hide themselves with work), but patching binaries on disk and the kernel memory live is a bit less transparent. You could do various other strategies rootkits do, though I'd rather not get too deep into those since those can get relatively complicated (with the ideas presented in the Blue Pill rootkit paper still being some of my favorites). > what commands Asking it like that is less productive. The ideas I mention above should be enough for you to research and do it yourself. The exact details are complicated because this isn't a simple "out of the box" sort of thing to do. Modifying one call in one or two processes with gdb / LD_PRELOAD hacks is easy, but still isn't just "out of the box" since you'll have to write some gdb scripts / a preload shim. Doing it system wide in the kernel is even more complicated and even more resilient to listing commands instead of concepts, and asking "what commands" doesn't really make the answer more meaningful. Me saying "sys_call_table + gcc + insmod" isn't really any more helpful than just saying "sys_call_table". Hopefully the above helped! I recommend doing more research, especially on the sys_call_table, kprobes and, ksplice as those are the most relevant techniques, though realistically only the first two are things you could reasonably do without becoming an expert in patching kernels in memory.

ahartmetz 8y ago

This happened to me once in ~2006 on Linux with a scratched CD. I hope it's been fixed.

filomeno 8y ago
I think you're out of luck. I once accidentally scratched a CD and ten years later it only got worse.
- jfroma 8y ago
  You made me laugh a lot. Thank you
imtringued 8y ago
It can also happen if you run Linux on a crappy flash drive.

caf 8y ago

If you have a kernel driver that is giving you this problem, it should be fairly easy in most cases to replace its use of TASK_UNINTERRUPTIBLE with TASK_KILLABLE.

jakeogh 8y ago
Why does TASK_UNINTERRUPTIBLE exist? 'I'm sorry Dave'?
- ovi256 8y ago
  Obviously for the app developpers that write rocket flight computers. They also don't use free, their garbage collection is implemented in a physical mechanism.
- caf 8y ago
  It's because for a long time, there was only "interruptible" and "uninterruptible", where "interruptible" meant that any signal could interrupt it. Historically, many operations were specified not to be interrupted by signals - reads and writes on disk files (as opposed to "slow devices" like ttys and sockets), mkdir(2), fsync(2), etc. They're not allowed to return EINTR. This meant using an uninterruptible sleep, which has the consequence that SIGKILL can't interrupt it either. That's why the code in top or ps for uninterruptible sleep is 'D' - historically it meant "Disk Wait". TASK_KILLABLE is a more recent invention, which can be interrupted only by fatal signals. The code doing the wait still has to unwind as if it were going to exit from the syscall with an error, but userspace won't see that error because it will die from the signal instead.
- notriddle 8y ago
  Because the call doesn't support rolling back partial progress. Imagine if you interrupted `mount`, and it had written to the directory table but hadn't yet initialized the underlying filesystem and now the kernel reads uninitialized memory whenever you look at that directory (or, less disastrously, the mutex that protects that table is left locked and you can never mount or unmount anything ever again).

deleted 8y ago

[deleted]

sitkack 8y ago
You need to build your own hardware.

ajross 8y ago

Step 2 needs some clarification. It is possible to implement "blocking I/O" in the kernel with the wait_interruptible() family of functions, and many drivers and subsystems do. These have the fairly straightforward behavior that they can return early with an exit code instead of when the given waitq ueue has been tickled. In the kernel, in C, this is pretty obvious stuff. But the side effect of not using the _interruptible variants is that there's no way for the kernel to return except from the intended event, which means that other things that you'd want to produce a return from the kernel context can't happen either, including signal delivery. So to userspace the process becomes unkillable and it looks like voodoo. But at the bottom level all it is in a C API that made promises that weren't kept at runtime.

mrguyorama 8y ago

Interestingly enough, Windows has gotten a little smart about kernel drivers misbehaving. I found out on my Windows 7 machine that if the AMD video driver becomes unhappy in certain ways, Windows will be able to recover it with no blue screen or loss of functionality other than the program who's access to the direct3D device caused the problem no longer having access until it attempts to re-establish a connection or is restarted. An incredibly limited set of circumstances, but I was awestruck to see Windows catch itself hiccuping in kernel space and not explode.

rusk 8y ago
and yet, user-space applications can still bring down the whole thing; and just try bringing up Task Manager to kill a task when you need to.
- mrguyorama 8y ago
  Killing the "task" hasn't worked since Windows 95, but killing the process itself still unwedges nearly anything userspace in my experience
  - rusk 8y ago
    Still need task manager though