6 ms·

Understanding how killall works using strace

65 points by jvns 13y ago

fsniper 13y ago
I think this article is great for learning about strace but it's not a good advice on learning about open source software. Why not to look into the source? I used strace for different scenarios and always it saved the day. But this would not be my play for learning about killall.
- midas007 13y ago
  Linux: https://github.com/acg/psmisc/blob/master/src/killall.c https://github.com/acg/psmisc/blob/master/src/killall.c FreeBSD: http://svnweb.freebsd.org/base/stable/10/usr.bin/killall/killall.c?view=markup http://svnweb.freebsd.org/base/stable/10/usr.bin/killall/kil... OSX: http://www.publicsource.apple.com/source/shell_cmds/shell_cmds-34/killall/killall.c http://www.publicsource.apple.com/source/shell_cmds/shell_cm...
- davidw 13y ago
  Strace gives you a quick, higher level view of what is actually happening, which is sometimes much more useful than slogging through the source code of something.
losethos 13y ago
software interrupt is 500 cycles. CALL instruction is 10. Ring-0-only rules!
vezzy-fnord 13y ago
strace is a pretty neat tool. You might also be interested in the Windows NT equivalent: http://www.intellectualheaven.com/default.asp?BH=projects&H=strace.htm http://www.intellectualheaven.com/default.asp?BH=projects&H=...
- windowsworkstoo 13y ago
  The supported Windows equivalent is actually Event Tracing for Windows (ETW): http://msdn.microsoft.com/en-us/library/windows/desktop/bb968803(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/bb96...
0x0 13y ago
ltrace is also pretty neat; it shows library calls (as opposed to system calls) - so you should be able to see the missing "strcmp()"s alluded to in the article.
philh 13y ago
strace is one of my favourite tools. I don't need it that often, but it's comforting to know that it's there. Documentation can lie or just not answer my question, the source code might be unavailable or unreadable, but if I have a question that strace can answer, I can rely on it to answer. It's one of the things I miss on OS X. dtruss isn't as powerful, and dtrace seems like it might be more powerful but I have no idea how to use it.
mhurron 13y ago
Anyone know of a good strace tutorial or how-to to start getting the best out of it?
- spudlyo 13y ago
  I made a screencast showing a practical example of how to use strace to debug PHP/Apache issues. http://www.youtube.com/watch?v=eF-p--AH37E http://www.youtube.com/watch?v=eF-p--AH37E
- jzwinck 13y ago
  I get the best out of it by knowing about its -f option, and the fact that the program htop is like top but crazier, and there you can press "s" to jump to a paged strace view of any process you can control.
cstavish 13y ago
Great article! There's a slight typo--"stdout" should be "stderr", as 2 is the file descriptor instead of 1.
waeyn 13y ago
I learned that on UNIX System V machines such as Solaris, killall ignores parameters and just tries to kill everything [1]. Maybe it's better to use pkill on GNU/Linux instead. 1: http://docs.oracle.com/cd/E19253-01/816-5166/killall-1m/index.html http://docs.oracle.com/cd/E19253-01/816-5166/killall-1m/inde...
- k3oni 13y ago
  Yep pkill is recommended on UNIX based OSs like Openindiana/Solaris etc. On these killall will actually do what it's name says :)
- vacri 13y ago
  What's the use case of that?
  - moyix 13y ago
    It survives in Linux as killall5 (the 5 being a reference to System V Unix). It's used during shutdown to send SIGTERM to all processes, then SIGKILL to kill off any stragglers.
  - midas007 13y ago
    Cleaning up after a user's process that has something that's effectively a fork-bombing. Also, terminating an employee.
cperciva 13y ago
Random trivial apropos killall: The killall utility in FreeBSD had one of my all-time "favourite" bugs, prompting me to write this commit message in 2004: Serial murderers shouldn't commit suicide. (killall should avoid killing itself.)
- deleted 13y ago
  [deleted]
derleth 13y ago
ltrace never seems to get as much attention as strace: http://en.wikipedia.org/wiki/Ltrace http://en.wikipedia.org/wiki/Ltrace > ltrace is a debugging utility in Linux, used to display the calls a userland application makes to shared libraries. This is useful because strace can completely ignore the information you really want. OTOH, ltrace doesn't provide information that's as pretty as what strace shows, because ltrace doesn't know as much about the functions it's showing you. This means arguments passed as pointers will be shown as pointer values, functions will occasionally be shown as taking the wrong number of arguments, and enum values will be shown as ints. ltrace also can't show function calls to functions in statically-linked libraries, because it hooks into the dynamic loading mechanism.
belluchan 13y ago
You could use strace or ... you could just look at the code: http://src.gnu-darwin.org/src/usr.bin/killall/killall.c.html http://src.gnu-darwin.org/src/usr.bin/killall/killall.c.html
- alayne 13y ago
  Step 1: find the correct source code. Reading code has its place and tracing code has its place. I have difficulty understanding nontrivial codebases without some amount of tracing. In this case you chose to present the source for a FreeBSD/OS X version of killall when OP is running Linux.
  - belluchan 13y ago
    Finding the right source for killall for whatever platform you want is trivial.
    - FeeTinesAMady 13y ago
      You didn't.
  - nly 13y ago
    On my Linux system it's provided by psmisc: http://sourceforge.net/p/psmisc/code/ci/master/tree/src/killall.c http://sourceforge.net/p/psmisc/code/ci/master/tree/src/kill...
axylone 13y ago
What are the anonymous 4096 byte allocations before every read? I would think that killall would read /proc/{pid}/stat directly into a stack buffer, especially if the reads are only 1024 bytes.
- nly 13y ago
  I had a quick look. There are 6 calls to malloc in the killall program and several calls to functions like asprintf which will allocate. See my other post for source.
jlgaddis 13y ago
If you're a Linux guy/girl and you someday end up in charge of a Solaris box, just remember that killall acts very differently! I almost think this is one of those "rights of passage" -- that everybody must do this at least once. You'll learn it the hard way but then you'll never forget... Kinda like many years ago when I was (new to Cisco gear and) making changes to a firewall ruleset -- remotely -- and decided to take a shortcut because I was supposed to go meet friends and so I was in a hurry.
- fsniper 13y ago
  Very bad incompatibility between oses indeed. It hurts when you learn but you can never ever forget. I feel for you. Actually I did not learned the hard way. But I was about to..
- jdswain 13y ago
  Never done this myself, but have done worse really. I was working at Sun a long time ago and we had a big shared development server. Being familiar with linux killall I suggested to my boss that he use killall instead of killing processes one by one. So he tried it out. Whole team had to wait quite a while for the server to restart. :-)
frik 13y ago
A bit offtopic: Julia Evans finished Hacker School, so I wonder if she continues with her Rust based operating system? https://github.com/jvns/puddle https://github.com/jvns/puddle featured on HN 3 days ago: https://news.ycombinator.com/item?id=6935763 https://news.ycombinator.com/item?id=6935763

65 points by jvns 13y ago

I think this article is great for learning about strace but it's not a good advice on learning about open source software. Why not to look into the source? I used strace for different scenarios and always it saved the day. But this would not be my play for learning about killall.

midas007 13y ago
Linux: https://github.com/acg/psmisc/blob/master/src/killall.c https://github.com/acg/psmisc/blob/master/src/killall.c FreeBSD: http://svnweb.freebsd.org/base/stable/10/usr.bin/killall/killall.c?view=markup http://svnweb.freebsd.org/base/stable/10/usr.bin/killall/kil... OSX: http://www.publicsource.apple.com/source/shell_cmds/shell_cmds-34/killall/killall.c http://www.publicsource.apple.com/source/shell_cmds/shell_cm...
davidw 13y ago
Strace gives you a quick, higher level view of what is actually happening, which is sometimes much more useful than slogging through the source code of something.

losethos 13y ago

software interrupt is 500 cycles. CALL instruction is 10. Ring-0-only rules!

vezzy-fnord 13y ago

strace is a pretty neat tool. You might also be interested in the Windows NT equivalent: http://www.intellectualheaven.com/default.asp?BH=projects&H=strace.htm http://www.intellectualheaven.com/default.asp?BH=projects&H=...

windowsworkstoo 13y ago
The supported Windows equivalent is actually Event Tracing for Windows (ETW): http://msdn.microsoft.com/en-us/library/windows/desktop/bb968803(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/bb96...

0x0 13y ago

ltrace is also pretty neat; it shows library calls (as opposed to system calls) - so you should be able to see the missing "strcmp()"s alluded to in the article.

philh 13y ago

strace is one of my favourite tools. I don't need it that often, but it's comforting to know that it's there. Documentation can lie or just not answer my question, the source code might be unavailable or unreadable, but if I have a question that strace can answer, I can rely on it to answer. It's one of the things I miss on OS X. dtruss isn't as powerful, and dtrace seems like it might be more powerful but I have no idea how to use it.

mhurron 13y ago

Anyone know of a good strace tutorial or how-to to start getting the best out of it?

spudlyo 13y ago
I made a screencast showing a practical example of how to use strace to debug PHP/Apache issues. http://www.youtube.com/watch?v=eF-p--AH37E http://www.youtube.com/watch?v=eF-p--AH37E
jzwinck 13y ago
I get the best out of it by knowing about its -f option, and the fact that the program htop is like top but crazier, and there you can press "s" to jump to a paged strace view of any process you can control.

cstavish 13y ago

Great article! There's a slight typo--"stdout" should be "stderr", as 2 is the file descriptor instead of 1.

waeyn 13y ago

I learned that on UNIX System V machines such as Solaris, killall ignores parameters and just tries to kill everything [1]. Maybe it's better to use pkill on GNU/Linux instead. 1: http://docs.oracle.com/cd/E19253-01/816-5166/killall-1m/index.html http://docs.oracle.com/cd/E19253-01/816-5166/killall-1m/inde...

k3oni 13y ago
Yep pkill is recommended on UNIX based OSs like Openindiana/Solaris etc. On these killall will actually do what it's name says :)
vacri 13y ago
What's the use case of that?
- moyix 13y ago
  It survives in Linux as killall5 (the 5 being a reference to System V Unix). It's used during shutdown to send SIGTERM to all processes, then SIGKILL to kill off any stragglers.
- midas007 13y ago
  Cleaning up after a user's process that has something that's effectively a fork-bombing. Also, terminating an employee.

cperciva 13y ago

Random trivial apropos killall: The killall utility in FreeBSD had one of my all-time "favourite" bugs, prompting me to write this commit message in 2004: Serial murderers shouldn't commit suicide. (killall should avoid killing itself.)

deleted 13y ago
[deleted]

derleth 13y ago

ltrace never seems to get as much attention as strace: http://en.wikipedia.org/wiki/Ltrace http://en.wikipedia.org/wiki/Ltrace > ltrace is a debugging utility in Linux, used to display the calls a userland application makes to shared libraries. This is useful because strace can completely ignore the information you really want. OTOH, ltrace doesn't provide information that's as pretty as what strace shows, because ltrace doesn't know as much about the functions it's showing you. This means arguments passed as pointers will be shown as pointer values, functions will occasionally be shown as taking the wrong number of arguments, and enum values will be shown as ints. ltrace also can't show function calls to functions in statically-linked libraries, because it hooks into the dynamic loading mechanism.

belluchan 13y ago

You could use strace or ... you could just look at the code: http://src.gnu-darwin.org/src/usr.bin/killall/killall.c.html http://src.gnu-darwin.org/src/usr.bin/killall/killall.c.html

alayne 13y ago
Step 1: find the correct source code. Reading code has its place and tracing code has its place. I have difficulty understanding nontrivial codebases without some amount of tracing. In this case you chose to present the source for a FreeBSD/OS X version of killall when OP is running Linux.
- belluchan 13y ago
  Finding the right source for killall for whatever platform you want is trivial.
  - FeeTinesAMady 13y ago
    You didn't.
- nly 13y ago
  On my Linux system it's provided by psmisc: http://sourceforge.net/p/psmisc/code/ci/master/tree/src/killall.c http://sourceforge.net/p/psmisc/code/ci/master/tree/src/kill...

axylone 13y ago

What are the anonymous 4096 byte allocations before every read? I would think that killall would read /proc/{pid}/stat directly into a stack buffer, especially if the reads are only 1024 bytes.

nly 13y ago
I had a quick look. There are 6 calls to malloc in the killall program and several calls to functions like asprintf which will allocate. See my other post for source.

jlgaddis 13y ago

If you're a Linux guy/girl and you someday end up in charge of a Solaris box, just remember that killall acts very differently! I almost think this is one of those "rights of passage" -- that everybody must do this at least once. You'll learn it the hard way but then you'll never forget... Kinda like many years ago when I was (new to Cisco gear and) making changes to a firewall ruleset -- remotely -- and decided to take a shortcut because I was supposed to go meet friends and so I was in a hurry.

fsniper 13y ago
Very bad incompatibility between oses indeed. It hurts when you learn but you can never ever forget. I feel for you. Actually I did not learned the hard way. But I was about to..
jdswain 13y ago
Never done this myself, but have done worse really. I was working at Sun a long time ago and we had a big shared development server. Being familiar with linux killall I suggested to my boss that he use killall instead of killing processes one by one. So he tried it out. Whole team had to wait quite a while for the server to restart. :-)

frik 13y ago

A bit offtopic: Julia Evans finished Hacker School, so I wonder if she continues with her Rust based operating system? https://github.com/jvns/puddle https://github.com/jvns/puddle featured on HN 3 days ago: https://news.ycombinator.com/item?id=6935763 https://news.ycombinator.com/item?id=6935763