3 ms·

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understa

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understand the gdb-output, nor what that has to do with the rest of the text.

jsnell 14y ago
So that whole section doesn't make a lot of sense to me. Running strace on an unkillable process tends to produce no output (if the process was actually making new system calls, it wouldn't be wedged). And worse, my experience is that attaching to a wedged process with ptrace() usually does nothing at all except also hang the attaching the process. This also applies to gdb. And finally, even if attaching worked, getting a disassembly of the current PC (in this case the syscall trampoline) would tell nothing useful about what's going on.
- agwa 14y ago
  I don't understand it either. The only syscalls going on there are reads from /dev/random and writes to /dev/null - both of those can be interrupted (in fact writes to /dev/null should be instantaneous). I think the author may be conflating applications blocked in system calls (since reads from /dev/random will block if the system lacks entropy) with applications blocked in uninterruptible system calls.
  - syncsynchalt 14y ago
    There are no reads from /dev/random successfully happening in the example (not after the first few blocks, anyway). /dev/random reads from an entropy pool that is quickly exhausted and slowly filled. The kernel will lock a process in D state while waiting for entropy. If you're expecting a stream of pseudo-random data then you can get that by directing /dev/urandom to /dev/null.
    - agwa 14y ago
      > The kernel will lock a process in D state while waiting for entropy. No it doesn't, at least not in Linux 3.0.57 or any other kernel I can remember for the last many years. It blocks if it needs entropy, but it's interruptible meaning it's in the S state, not the D state, and can be killed.
nonane 14y ago
It think one of the reasons the kernel can't kill a process is because one of the process's threads is blocked inside a kernel call (not completely sure about this). Hes using ps's 'wchan' option to get the address of the kernel function that the process is currently blocked or sleeping on. After he gets the wchan address, he uses gdb to map the address to function name. Taken from: http://unixhelp.ed.ac.uk/CGI/man-cgi?ps http://unixhelp.ed.ac.uk/CGI/man-cgi?ps nwchan WCHAN address of the kernel function where the process is sleeping (use wchan if you want the kernel function name). Running tasks will display a dash ('-') in this column.
- mbell 14y ago
  /proc/{pid}/wchan contains the function name, not its address. In this example he's just printing the syscall that his example process is running. The calls to strace and gdb are disjoint, he's cut out a lot of stuff here, probably why its confusing. In the commands he's used strace would dump all the syscall details to the console and GDB would just attach to the process and enter its normal console (presumably hes determined hes stuck in a loop so hes just using disassemble).
mbell 14y ago
I believe the author is using that as an example of a command that will 'wedge' if you tried to kill -9 it. The rest of it is an example of how to dump the cause of a process that is wedged. "/proc/{pid}/wchan" contains the name of the current syscall a process is executing. Basically you should be able to use everything but the first line of that section as a shell script(with some modifications) to determine the cause of a wedged process.
dllthomas 14y ago
Not sure just how much you do/don't understand, or how much others will/won't understand, so I'll run through line by line: PID=$! grabs the PID of the process you just spawned (in the background, with &) into shell variable PID CMDLINE="!-2" grabs the full line you just ran (before the line storing PID) with shell history expansion CMD=${CMDLINE%% *} expands the CMDLINE variable, replacing everything after the first space (so CMD now has "cat") with bash trickery WCHAN=$(cat /proc/${PID}/wchan) grabs the name of the currently executing syscall for the process (at least, according to http://www.lindevdoc.org/wiki/proc/pid/wchan http://www.lindevdoc.org/wiki/proc/pid/wchan) echo "command: ${CMD}, pid: ${PID}, wchan: ${WCHAN}" prints the info we've grabbed strace -p ${PID} connects a trace to the process to see what it's doing gdb ${CMD} ${PID} connects to the process (gdb needs program name and can be given a pid to connect to) (gdb) disassemble prints the actual (assembler) code being run. In this case, I think all we get from the output is that it's in fact in the middle of some syscall - you'd have to check registers and syscall tables to determine which. As others have mentioned, much of this is less useful than implied in the face of an actual wedged process. Tangentially, using gdb to attach to running processes is a very powerful technique - I've been able to get line numbers out of running bash scripts.
- a_bonobo 14y ago
  Thank you for that! Makes it clear, I guess I have to properly learn gdb's output... I work in bioinformatics, we never go that low.

jsnell 14y ago

So that whole section doesn't make a lot of sense to me. Running strace on an unkillable process tends to produce no output (if the process was actually making new system calls, it wouldn't be wedged). And worse, my experience is that attaching to a wedged process with ptrace() usually does nothing at all except also hang the attaching the process. This also applies to gdb. And finally, even if attaching worked, getting a disassembly of the current PC (in this case the syscall trampoline) would tell nothing useful about what's going on.

agwa 14y ago
I don't understand it either. The only syscalls going on there are reads from /dev/random and writes to /dev/null - both of those can be interrupted (in fact writes to /dev/null should be instantaneous). I think the author may be conflating applications blocked in system calls (since reads from /dev/random will block if the system lacks entropy) with applications blocked in uninterruptible system calls.
- syncsynchalt 14y ago
  There are no reads from /dev/random successfully happening in the example (not after the first few blocks, anyway). /dev/random reads from an entropy pool that is quickly exhausted and slowly filled. The kernel will lock a process in D state while waiting for entropy. If you're expecting a stream of pseudo-random data then you can get that by directing /dev/urandom to /dev/null.
  - agwa 14y ago
    > The kernel will lock a process in D state while waiting for entropy. No it doesn't, at least not in Linux 3.0.57 or any other kernel I can remember for the last many years. It blocks if it needs entropy, but it's interruptible meaning it's in the S state, not the D state, and can be killed.

nonane 14y ago

It think one of the reasons the kernel can't kill a process is because one of the process's threads is blocked inside a kernel call (not completely sure about this). Hes using ps's 'wchan' option to get the address of the kernel function that the process is currently blocked or sleeping on. After he gets the wchan address, he uses gdb to map the address to function name. Taken from: http://unixhelp.ed.ac.uk/CGI/man-cgi?ps http://unixhelp.ed.ac.uk/CGI/man-cgi?ps nwchan WCHAN address of the kernel function where the process is sleeping (use wchan if you want the kernel function name). Running tasks will display a dash ('-') in this column.

mbell 14y ago
/proc/{pid}/wchan contains the function name, not its address. In this example he's just printing the syscall that his example process is running. The calls to strace and gdb are disjoint, he's cut out a lot of stuff here, probably why its confusing. In the commands he's used strace would dump all the syscall details to the console and GDB would just attach to the process and enter its normal console (presumably hes determined hes stuck in a loop so hes just using disassemble).

mbell 14y ago

I believe the author is using that as an example of a command that will 'wedge' if you tried to kill -9 it. The rest of it is an example of how to dump the cause of a process that is wedged. "/proc/{pid}/wchan" contains the name of the current syscall a process is executing. Basically you should be able to use everything but the first line of that section as a shell script(with some modifications) to determine the cause of a wedged process.

dllthomas 14y ago

Not sure just how much you do/don't understand, or how much others will/won't understand, so I'll run through line by line: PID=$! grabs the PID of the process you just spawned (in the background, with &) into shell variable PID CMDLINE="!-2" grabs the full line you just ran (before the line storing PID) with shell history expansion CMD=${CMDLINE%% *} expands the CMDLINE variable, replacing everything after the first space (so CMD now has "cat") with bash trickery WCHAN=$(cat /proc/${PID}/wchan) grabs the name of the currently executing syscall for the process (at least, according to http://www.lindevdoc.org/wiki/proc/pid/wchan http://www.lindevdoc.org/wiki/proc/pid/wchan) echo "command: ${CMD}, pid: ${PID}, wchan: ${WCHAN}" prints the info we've grabbed strace -p ${PID} connects a trace to the process to see what it's doing gdb ${CMD} ${PID} connects to the process (gdb needs program name and can be given a pid to connect to) (gdb) disassemble prints the actual (assembler) code being run. In this case, I think all we get from the output is that it's in fact in the middle of some syscall - you'd have to check registers and syscall tables to determine which. As others have mentioned, much of this is less useful than implied in the face of an actual wedged process. Tangentially, using gdb to attach to running processes is a very powerful technique - I've been able to get line numbers out of running bash scripts.

a_bonobo 14y ago
Thank you for that! Makes it clear, I guess I have to properly learn gdb's output... I work in bioinformatics, we never go that low.