5 ms·

If you are able to load kernel drivers you could just nop out the kill syscall though.

by kpcyrd 8y ago

oblio 8y ago
For the mere mortals amongst us, how would you actually do that, practically? I.e. what commands?
- TheDong 8y ago
  There are many options. One new way would be to install a kprobe. You can read more about kprobes here: https://www.kernel.org/doc/Documentation/kprobes.txt https://www.kernel.org/doc/Documentation/kprobes.txt (note, they're not meant for modifying syscall arguments in general, but in the case of kill it will be possible :) Effectively, they let you write code that will run before syscalls and can mutate the state of the kernel and registers and such. You could write a kprobe which changed "kill"s pid argument to some random pid that didn't exist each time the given pid was being killed. You could also just patch the kernel in memory yourself easily enough (which is even easier for a kernel module; look up modfying the sys_call_table; that table exists for a reason). Modifying the sys_call_table is probably what the parent poster was thinking of, and is definitely the "right" way to do this. You could compile a new kernel with the "kill" syscall modified by editing the c code, and then do something akin to what ksplice did to patch the running kernel with that. kexec/kpatch are the modern tools to do that, though neither of them are as good as ksplice is... not that ksplice is actually something you can use without working at oracle on that team. You could also modify the kill binary under the assumption the user is running "/usr/bin/kill", not making the syscall in some other way, and modifying that binary is pretty trivial. These last few are more like what rootkits do and are harder to detect. kprobes and kernel modules are both meant to be visible to users in general (though they can ofc hide themselves with work), but patching binaries on disk and the kernel memory live is a bit less transparent. You could do various other strategies rootkits do, though I'd rather not get too deep into those since those can get relatively complicated (with the ideas presented in the Blue Pill rootkit paper still being some of my favorites). > what commands Asking it like that is less productive. The ideas I mention above should be enough for you to research and do it yourself. The exact details are complicated because this isn't a simple "out of the box" sort of thing to do. Modifying one call in one or two processes with gdb / LD_PRELOAD hacks is easy, but still isn't just "out of the box" since you'll have to write some gdb scripts / a preload shim. Doing it system wide in the kernel is even more complicated and even more resilient to listing commands instead of concepts, and asking "what commands" doesn't really make the answer more meaningful. Me saying "sys_call_table + gcc + insmod" isn't really any more helpful than just saying "sys_call_table". Hopefully the above helped! I recommend doing more research, especially on the sys_call_table, kprobes and, ksplice as those are the most relevant techniques, though realistically only the first two are things you could reasonably do without becoming an expert in patching kernels in memory.
  - oblio 8y ago
    > Asking it like that is less productive. Less productive only if there's no command :) Thanks a lot for the info, it should provide quite a few hours of hacking around!
  - jakeogh 8y ago
    A syscall zombie is pretty easy to find, I often run into them when making NFS mistakes (like leaving the local net without properly ending some dep)... is there a general way to see what syscall the undead process is waiting for? Is there a simple way to make it return ERRx? Can I get a list of pending syscalls by age?
    - TheDong 8y ago
      It's very easy to find the current stack of running syscalls and how long they take using perf [0]. In fact, a flame graph will give you a wonderful way to visualize this [1]. I could go on and on about perf, but there's no reason since Brendan Gregg's page is so darned good. Just read that, play around with the perf cli, find the answers to all your questions... Well, all of them except how to cause that syscall to error out. There's no simple way to force an in-progress syscall to error out as far as I know. If any one else knows of a simple way, I'd be curious to hear of it. [0]: http://www.brendangregg.com/perf.html http://www.brendangregg.com/perf.html [1]: http://www.brendangregg.com/perf.html#FlameGraphs http://www.brendangregg.com/perf.html#FlameGraphs
      - jakeogh 8y ago
        Very cool. I'm trying to start with: perf stat -e 'syscalls' but haven't figured out how to specify all events... if that's even the right route. The examples that are close dont work for me: https://bpaste.net/raw/455ebe2dd400 https://bpaste.net/raw/455ebe2dd400
    - cyphar 8y ago
      If you have the process that is currently blocked (note that "zombie" doesn't apply here -- zombies are processes that are dead but are waiting for someone to collect their exit code) you can look in /proc/$pid/stack to see what the current kernel stack is (the bottom-most entries tell you what syscall the process started by doing). This is the easiest way to start debugging a kernel problem (what is the function it's blocked in -- then go read that function and see whether the the pathology is obvious).

If you are able to load kernel drivers you could just nop out the kill syscall though.

by kpcyrd 8y ago

If you are able to load kernel drivers you could just nop out the kill syscall though.

oblio 8y ago

For the mere mortals amongst us, how would you actually do that, practically? I.e. what commands?

TheDong 8y ago
There are many options. One new way would be to install a kprobe. You can read more about kprobes here: https://www.kernel.org/doc/Documentation/kprobes.txt https://www.kernel.org/doc/Documentation/kprobes.txt (note, they're not meant for modifying syscall arguments in general, but in the case of kill it will be possible :) Effectively, they let you write code that will run before syscalls and can mutate the state of the kernel and registers and such. You could write a kprobe which changed "kill"s pid argument to some random pid that didn't exist each time the given pid was being killed. You could also just patch the kernel in memory yourself easily enough (which is even easier for a kernel module; look up modfying the sys_call_table; that table exists for a reason). Modifying the sys_call_table is probably what the parent poster was thinking of, and is definitely the "right" way to do this. You could compile a new kernel with the "kill" syscall modified by editing the c code, and then do something akin to what ksplice did to patch the running kernel with that. kexec/kpatch are the modern tools to do that, though neither of them are as good as ksplice is... not that ksplice is actually something you can use without working at oracle on that team. You could also modify the kill binary under the assumption the user is running "/usr/bin/kill", not making the syscall in some other way, and modifying that binary is pretty trivial. These last few are more like what rootkits do and are harder to detect. kprobes and kernel modules are both meant to be visible to users in general (though they can ofc hide themselves with work), but patching binaries on disk and the kernel memory live is a bit less transparent. You could do various other strategies rootkits do, though I'd rather not get too deep into those since those can get relatively complicated (with the ideas presented in the Blue Pill rootkit paper still being some of my favorites). > what commands Asking it like that is less productive. The ideas I mention above should be enough for you to research and do it yourself. The exact details are complicated because this isn't a simple "out of the box" sort of thing to do. Modifying one call in one or two processes with gdb / LD_PRELOAD hacks is easy, but still isn't just "out of the box" since you'll have to write some gdb scripts / a preload shim. Doing it system wide in the kernel is even more complicated and even more resilient to listing commands instead of concepts, and asking "what commands" doesn't really make the answer more meaningful. Me saying "sys_call_table + gcc + insmod" isn't really any more helpful than just saying "sys_call_table". Hopefully the above helped! I recommend doing more research, especially on the sys_call_table, kprobes and, ksplice as those are the most relevant techniques, though realistically only the first two are things you could reasonably do without becoming an expert in patching kernels in memory.
- oblio 8y ago
  > Asking it like that is less productive. Less productive only if there's no command :) Thanks a lot for the info, it should provide quite a few hours of hacking around!
- jakeogh 8y ago
  A syscall zombie is pretty easy to find, I often run into them when making NFS mistakes (like leaving the local net without properly ending some dep)... is there a general way to see what syscall the undead process is waiting for? Is there a simple way to make it return ERRx? Can I get a list of pending syscalls by age?
  - TheDong 8y ago
    It's very easy to find the current stack of running syscalls and how long they take using perf [0]. In fact, a flame graph will give you a wonderful way to visualize this [1]. I could go on and on about perf, but there's no reason since Brendan Gregg's page is so darned good. Just read that, play around with the perf cli, find the answers to all your questions... Well, all of them except how to cause that syscall to error out. There's no simple way to force an in-progress syscall to error out as far as I know. If any one else knows of a simple way, I'd be curious to hear of it. [0]: http://www.brendangregg.com/perf.html http://www.brendangregg.com/perf.html [1]: http://www.brendangregg.com/perf.html#FlameGraphs http://www.brendangregg.com/perf.html#FlameGraphs
    - jakeogh 8y ago
      Very cool. I'm trying to start with: perf stat -e 'syscalls' but haven't figured out how to specify all events... if that's even the right route. The examples that are close dont work for me: https://bpaste.net/raw/455ebe2dd400 https://bpaste.net/raw/455ebe2dd400
  - cyphar 8y ago
    If you have the process that is currently blocked (note that "zombie" doesn't apply here -- zombies are processes that are dead but are waiting for someone to collect their exit code) you can look in /proc/$pid/stack to see what the current kernel stack is (the bottom-most entries tell you what syscall the process started by doing). This is the easiest way to start debugging a kernel problem (what is the function it's blocked in -- then go read that function and see whether the the pathology is obvious).