5 ms·

I've now seen a similar case multiple times in the wild - if a process has a thread in uninterruptible sleep (e.g., blocked on a bad disk or a stuck network fil

by geofft 7y ago

I've now seen a similar case multiple times in the wild - if a process has a thread in uninterruptible sleep (e.g., blocked on a bad disk or a stuck network filesystem) and you kill it, the process dies, but the kernel waits forever for that thread before informing the parent process. The parent doesn't get SIGCHLD, and wait() doesn't return. (So, for example, your favorite init/supervisor won't restart the process or even realize the process has died and raise an alert.) If the thread wasn't the first thread, then the process looks like it's gone and stops having most information in /proc/$pid, but it's still got the thread in /proc/$pid/tasks/. I've taken to calling such a process a "lich," because it's not quite a zombie - the parent can reap a zombie by calling wait(), but the lich has used magic to avoid true death.

btown 7y ago
Can/do supervisors poll for this situation? Seems like that should be possible.
- xenadu02 7y ago
  Also curious why the kernel can't abandon the thread since the process is dead anyway; set a bit in kernel space so on the walk back from the syscall it bails out instead of trying to return to userspace. That doesn't solve the kernel thread being stuck forever problem. I'm not sure what the fix is there... is it bad drivers with no timeout mechanism? I don't know how they'd do that unless the kernel IO is all async anyway. Just tearing down the kernel thread seems likely to leave the locks abandoned in bad states.
  - cryptonector 7y ago
    The kernel can't abandon that thread entirely, but it could "detach" it from the process it was part of and arrange for it to clean itself up and disappear if ever it wakes. And so the parent process could get told of the child's death.
- jerf 7y ago
  This becomes a tarpit. Whatever metric you think to measure for this may itself become a source of false positives in the field. Whatever things you choose to try to avoid that becomes another source of falseness in the system. This tends to cascade as people apply patch upon patch until you have a system that basically randomly fails and nobody can tell you why. This is not a theoretical report; in theory this approach can work, if you get it perfect enough. This is a practical report, both of systems I've built myself and of many other such cases I've witnessed, including some things around process management and supervision. In the end, you really have little choice but to trust what the OS is telling you, and if the OS is lying for whatever reason, to fix the OS. As hard as that last bit may be as a solution, it still turns out to be the easiest, most-effective fix. And it may not be a perfect fix, either... but it'll still be the best.
- phkahler 7y ago
  >> Can/do supervisors poll for this situation? Sounds like a hack.
  - jrockway 7y ago
    Typically supervisors are monitoring readiness/liveness of the managed process for other reasons; you don't have the number of replicas you've promised if one of the replicas is un-live (perhaps hitting this bug, perhaps internally deadlocked, perhaps internally waiting forever on something the OS doesn't care about, like a certain network request), and you shouldn't be routing network traffic to a service that isn't ready for it. So while I agree with the other posts that a supervisor can't recover from this state, it can be aware that it's happened. And it's quite common for supervisors to check for readiness and liveness beyond "well the OS says it's running, sounds good to me."
- sigjuice 7y ago
  Sure supervisors can be made to poll. But a supervisor won’t be able to do anything to recover from this.
Lich 7y ago
I think lich fits well :)
gue5t 7y ago
The solution to this situation, as I understand it (on Linux), is TASK_KILLABLE, which effectively supercedes uninterruptible sleep as a concept: https://lwn.net/Articles/288056/ https://lwn.net/Articles/288056/ But TASK_KILLABLE is not used in most places it should be. Patching your (least) favorite driver to use TASK_KILLABLE could be a good entry point to contributing to the kernel. This older, tangential HN discussion and the comments on LWN have a bit more info: https://news.ycombinator.com/item?id=18056946 https://news.ycombinator.com/item?id=18056946
amelius 7y ago
For anyone who was wondering, I found a post explaining why threads can be in "uninterruptible sleep" in the first place: https://eklitzke.org/uninterruptible-sleep https://eklitzke.org/uninterruptible-sleep
- gravitas 7y ago
  On this article, one of the tricks you can do to avoid the hung NFS client reboots is to quickly alias the NFS server IP onto the local device (ip addr add ...). This basically causes the networking layer to act like the NFS server went away but also return a RST (reset) network packet to that effect, allowing the failure to propagate upwards to the rest of the stack and unhang.
  - jmiserez 7y ago
    Unplugging the ethernet cable usually also works.
  - the8472 7y ago
    you could also mount nfs with soft timeouts which won't wait forever and instead return an error to userspace.
    - Spivak 7y ago
      Which is a very good practice if you're in control of the software receiving the error. But so much freaking software doesn't handle this correctly it's maddening and so sleep everything that touches it is the only safe thing to do.
      - GauntletWizard 7y ago
        For some network filesystems, I want to mount them "Kill every process with a filehandle open to this on error". Is there a way to do that?
        7y ago
lonelappde 7y ago
> So, for example, your favorite init/supervisor won't restart the process or even realize the process has died and raise an alert. That's why a process that wants to stay running has to provide some form of heartbeat API.
emmelaich 7y ago
I saw my first lich on SPARCstation 10 (or 20?). It was a java program that I killed -9. It died but was never reaped. It prevented a software shutdown so I had to use the ol' STOP-A
rat87 7y ago
what happens to the other threads? Are they killed? Can a malicious process use this to avoid being killed
- geofft 7y ago
  All the other threads are killed. A malicious process can't do useful work - as soon as the kernel gets unstuck, the signal will get delivered to the thread and kill it.

by geofft 7y ago

btown 7y ago

Can/do supervisors poll for this situation? Seems like that should be possible.

xenadu02 7y ago
Also curious why the kernel can't abandon the thread since the process is dead anyway; set a bit in kernel space so on the walk back from the syscall it bails out instead of trying to return to userspace. That doesn't solve the kernel thread being stuck forever problem. I'm not sure what the fix is there... is it bad drivers with no timeout mechanism? I don't know how they'd do that unless the kernel IO is all async anyway. Just tearing down the kernel thread seems likely to leave the locks abandoned in bad states.
- cryptonector 7y ago
  The kernel can't abandon that thread entirely, but it could "detach" it from the process it was part of and arrange for it to clean itself up and disappear if ever it wakes. And so the parent process could get told of the child's death.
jerf 7y ago
This becomes a tarpit. Whatever metric you think to measure for this may itself become a source of false positives in the field. Whatever things you choose to try to avoid that becomes another source of falseness in the system. This tends to cascade as people apply patch upon patch until you have a system that basically randomly fails and nobody can tell you why. This is not a theoretical report; in theory this approach can work, if you get it perfect enough. This is a practical report, both of systems I've built myself and of many other such cases I've witnessed, including some things around process management and supervision. In the end, you really have little choice but to trust what the OS is telling you, and if the OS is lying for whatever reason, to fix the OS. As hard as that last bit may be as a solution, it still turns out to be the easiest, most-effective fix. And it may not be a perfect fix, either... but it'll still be the best.
phkahler 7y ago
>> Can/do supervisors poll for this situation? Sounds like a hack.
- jrockway 7y ago
  Typically supervisors are monitoring readiness/liveness of the managed process for other reasons; you don't have the number of replicas you've promised if one of the replicas is un-live (perhaps hitting this bug, perhaps internally deadlocked, perhaps internally waiting forever on something the OS doesn't care about, like a certain network request), and you shouldn't be routing network traffic to a service that isn't ready for it. So while I agree with the other posts that a supervisor can't recover from this state, it can be aware that it's happened. And it's quite common for supervisors to check for readiness and liveness beyond "well the OS says it's running, sounds good to me."
sigjuice 7y ago
Sure supervisors can be made to poll. But a supervisor won’t be able to do anything to recover from this.

Lich 7y ago

I think lich fits well :)

gue5t 7y ago

The solution to this situation, as I understand it (on Linux), is TASK_KILLABLE, which effectively supercedes uninterruptible sleep as a concept: https://lwn.net/Articles/288056/ https://lwn.net/Articles/288056/ But TASK_KILLABLE is not used in most places it should be. Patching your (least) favorite driver to use TASK_KILLABLE could be a good entry point to contributing to the kernel. This older, tangential HN discussion and the comments on LWN have a bit more info: https://news.ycombinator.com/item?id=18056946 https://news.ycombinator.com/item?id=18056946

amelius 7y ago

For anyone who was wondering, I found a post explaining why threads can be in "uninterruptible sleep" in the first place: https://eklitzke.org/uninterruptible-sleep https://eklitzke.org/uninterruptible-sleep

gravitas 7y ago
On this article, one of the tricks you can do to avoid the hung NFS client reboots is to quickly alias the NFS server IP onto the local device (ip addr add ...). This basically causes the networking layer to act like the NFS server went away but also return a RST (reset) network packet to that effect, allowing the failure to propagate upwards to the rest of the stack and unhang.
- jmiserez 7y ago
  Unplugging the ethernet cable usually also works.
- the8472 7y ago
  you could also mount nfs with soft timeouts which won't wait forever and instead return an error to userspace.
  - Spivak 7y ago
    Which is a very good practice if you're in control of the software receiving the error. But so much freaking software doesn't handle this correctly it's maddening and so sleep everything that touches it is the only safe thing to do.
    - GauntletWizard 7y ago
      For some network filesystems, I want to mount them "Kill every process with a filehandle open to this on error". Is there a way to do that?
      - 7y ago

lonelappde 7y ago

> So, for example, your favorite init/supervisor won't restart the process or even realize the process has died and raise an alert. That's why a process that wants to stay running has to provide some form of heartbeat API.

emmelaich 7y ago

I saw my first lich on SPARCstation 10 (or 20?). It was a java program that I killed -9. It died but was never reaped. It prevented a software shutdown so I had to use the ol' STOP-A

rat87 7y ago

what happens to the other threads? Are they killed? Can a malicious process use this to avoid being killed

geofft 7y ago
All the other threads are killed. A malicious process can't do useful work - as soon as the kernel gets unstuck, the signal will get delivered to the thread and kill it.