4 ms·

This is a total side question but I noticed this line: > The process was stuck in the kernel waiting for an IOCTL, so killing the process did nothing – even ki

This is a total side question but I noticed this line: > The process was stuck in the kernel waiting for an IOCTL, so killing the process did nothing – even kill -9. It was going to stay that way until EBS was back to normal. I did not know that can happen! Can anyone provide a further explanation/material for why this is the case?

1SaltwaterC 14y ago
IIRC is the only case where you can not kill a process under unices. It goes into the D state (uninterruptible sleep). Usually waiting for I/O is the reason for a process going into a D state. From my own experience, had this issue with faulty hardware and network filesystems. Here's something easy for reproducing a process going into uninterruptible sleep: GlusterFS 3.0.4 (Ubuntu 10.04 packages that version, a single machine setup is sufficient). A bug in Gluster turns into a deadlock and the process that triggers that bug does into the D state. Cause: using file locks without explicit unlock when releasing the file descriptor. The process can't be killed as it waits for the FS while the FS waits for the process to release the lock (which never happens). Had fun debugging that on a live deployment where each process got stuck one by one till the process pool was exhausted, while monit could not restart the daemon that managed the process pool.
matthewlucid 14y ago
Three links for you: https://jira.mongodb.org/browse/SERVER-6178 https://jira.mongodb.org/browse/SERVER-6178 http://www.noah.org/wiki/Kill_-9_does_not_work http://www.noah.org/wiki/Kill_-9_does_not_work http://stackoverflow.com/questions/223644/what-is-an-uninterruptable-process http://stackoverflow.com/questions/223644/what-is-an-uninter... The first is the bug I reported to 10gen about their MongoDB setup. Contained in the bug report is the set of commands I used to reproduce the issue so that you can see it yourself. The second and third links are about processes in uninterruptible sleep, which is the state my MongoDB processes were in. It should give more detail than I'm about to give. Linux processes all have a state (running, sleeping, etc). One of those states is "Uninterruptible Sleep". Only the kernel can change a process's state. Uninterruptible sleep ("D" in `top`) is a state in which the process will receive no interrupts (signals). Since the `kill` command is just a glorified signal sender, it ceases to work for those processes.
ajross 14y ago
Processes blocked in system calls can be unkillable. In the Linux kernel, what typically happens is that the process is blocked on a wait queue (what userspace usually call a "condition variable" or "monitor") in one of the wait_event() family of functions. Because the kernel doesn't implement any kind of exception handling, there is no way to automatically break out of this; you have to wait for some other task to call wake_up(). It's possible to wait in a "interruptible" way, and many subsystems do. But that's hard, because it means that the resources you would otherwise be responsible for clean up need to be detected and cleaned up by someone else (or alternatively that someone else needs to pick up and finish what you started, or you need to be able to roll back a partially completed action, etc...). Userspace usually ignoes this problem (because the task got killed, right?), but in the kernel a leak is a huge bug. So the default is that waiting in the kernel means that the kernel can't allow the process to be terminated (or otherwise be delivered a signal that would invalidate the stack). And the side effect is that userspace sees an unkillable process in the "D" state.

1SaltwaterC 14y ago

IIRC is the only case where you can not kill a process under unices. It goes into the D state (uninterruptible sleep). Usually waiting for I/O is the reason for a process going into a D state. From my own experience, had this issue with faulty hardware and network filesystems. Here's something easy for reproducing a process going into uninterruptible sleep: GlusterFS 3.0.4 (Ubuntu 10.04 packages that version, a single machine setup is sufficient). A bug in Gluster turns into a deadlock and the process that triggers that bug does into the D state. Cause: using file locks without explicit unlock when releasing the file descriptor. The process can't be killed as it waits for the FS while the FS waits for the process to release the lock (which never happens). Had fun debugging that on a live deployment where each process got stuck one by one till the process pool was exhausted, while monit could not restart the daemon that managed the process pool.

matthewlucid 14y ago

Three links for you: https://jira.mongodb.org/browse/SERVER-6178 https://jira.mongodb.org/browse/SERVER-6178 http://www.noah.org/wiki/Kill_-9_does_not_work http://www.noah.org/wiki/Kill_-9_does_not_work http://stackoverflow.com/questions/223644/what-is-an-uninterruptable-process http://stackoverflow.com/questions/223644/what-is-an-uninter... The first is the bug I reported to 10gen about their MongoDB setup. Contained in the bug report is the set of commands I used to reproduce the issue so that you can see it yourself. The second and third links are about processes in uninterruptible sleep, which is the state my MongoDB processes were in. It should give more detail than I'm about to give. Linux processes all have a state (running, sleeping, etc). One of those states is "Uninterruptible Sleep". Only the kernel can change a process's state. Uninterruptible sleep ("D" in `top`) is a state in which the process will receive no interrupts (signals). Since the `kill` command is just a glorified signal sender, it ceases to work for those processes.

ajross 14y ago

Processes blocked in system calls can be unkillable. In the Linux kernel, what typically happens is that the process is blocked on a wait queue (what userspace usually call a "condition variable" or "monitor") in one of the wait_event() family of functions. Because the kernel doesn't implement any kind of exception handling, there is no way to automatically break out of this; you have to wait for some other task to call wake_up(). It's possible to wait in a "interruptible" way, and many subsystems do. But that's hard, because it means that the resources you would otherwise be responsible for clean up need to be detected and cleaned up by someone else (or alternatively that someone else needs to pick up and finish what you started, or you need to be able to roll back a partially completed action, etc...). Userspace usually ignoes this problem (because the task got killed, right?), but in the kernel a leak is a huge bug. So the default is that waiting in the kernel means that the kernel can't allow the process to be terminated (or otherwise be delivered a signal that would invalidate the stack). And the side effect is that userspace sees an unkillable process in the "D" state.