5 ms·

Linux Performance Analysis and Tools

127 points by jswanson 13y ago

nailer 13y ago
Awesome talk, though the SystemTap coverage was oddly weak. Saying 'SystemTap was was created when there weren't Solaris for Linux ports' is a little disingenuous. Linux had live kernel instrumentation with kprobes way before Sun debuted DTrace, and DTrace was deliberately licensed in a way to be incompatible with the Linux kernel. If you want to see some actual useful production taps you can use right now on your Red Hat / CentOS box, check out http://sourceware.org/systemtap/wiki/WarStories http://sourceware.org/systemtap/wiki/WarStories
- insaneirish 13y ago
  DTrace was deliberately licensed in a way to be incompatible with the Linux kernel. This was commented on directly by Bryan Cantrill on HN [1]. You should also watch his "Fork Yeah!" [2] and "Corporate Open Source Anti-Patterns" [3] talks for full context. [1]: https://news.ycombinator.com/item?id=4357507 https://news.ycombinator.com/item?id=4357507 [2]: http://www.youtube.com/watch?v=-zRN7XLCRhc http://www.youtube.com/watch?v=-zRN7XLCRhc [3]: http://www.youtube.com/watch?v=NhgXQFk9noI http://www.youtube.com/watch?v=NhgXQFk9noI
  - alanmackenzie 13y ago
    Great links.
  - ehsanu1 13y ago
    I want to point out a comment [1] apparently by the former chief open source evangelist, Danese Cooper [2], on the "Fork Yeah!" video contradicting Bryan Cantrill on this point. Here it is reproduced: Lovely except it really was decided to explicitly make OpenSolaris incompatible with GPL. That was one of the design points of the CDDL. I was in that room, Bryan and you were not, but I know its fun to re-write history to suit your current politics. I pleaded with Sun to use a BSD family license or the GPL itself and they would consider neither because that would have allowed D-Trace to end up in Linux. You can claim otherwise all you want...this was the truth in 2005 [1] http://www.youtube.com/watch?v=-zRN7XLCRhc&lc=UEj5uH4rMafUnXNekr8VLEFBfO2UpbRVyNaLbGRcDzs http://www.youtube.com/watch?v=-zRN7XLCRhc&lc=UEj5uH4rMafUnX... [2] http://en.wikipedia.org/wiki/Danese_Cooper http://en.wikipedia.org/wiki/Danese_Cooper
    - gillianseed 13y ago
      I fully trust in Danese's description of the events in this case. Not only because she (Danese) was the person who actually created the licence (and thus is the best authority on the prerequisites set by the Sun management), but also because it makes perfect sense. Sun, a company whose Solaris product was suffering greatly under competition from Linux, would NOT want to offer up their systems technical advantages under a open source licence which would allow Linux to use said technical advantages. It seems purely logical to me. Listening further to Danese, she describes the Sun management as wanting a copyleft style licence for the code, and that they were eyeing GPLv3, but not GPLv2 which was already available (again, GPLv3 code would not be compatible with the Linux kernel, while GPLv2 obviously would), however GPLv3 was taking to long to be finalized so they set her (Danese) on the task of creating a new licence.
      - brendangregg 13y ago
        I've recently been using DTrace on Ubuntu. I filed two bugs on the project page (https://github.com/dtrace4linux/linux https://github.com/dtrace4linux/linux), and Paul Fox fixed them quickly. I have more bugs to file, but I'm doing my part in making DTrace on Linux a reality. It's very tempting to join these licensing discussions (I was at Sun, too, and many of us DID want to see DTrace on Linux), but I think that's a distraction from what's happening right now. There are two projects porting DTrace to Linux, which is really exciting. I'm helping out.
    - justin66 13y ago
      > Lovely except it really was decided to explicitly make OpenSolaris incompatible with GPL. In fairness to both sides, pretty much everything that isn't GPL can be said to be incompatible with GPL, and almost anyone who decides to use something different than the GPL can be said to be deliberately incompatible with the GPL. That's just the nature of the GPL.
      - _delirium 13y ago
        Permissive open-source licenses (MIT, BSD, etc.) are GPL-compatible. But I think the question here is more one of motivation: was a decision made for reason X which had the side-effect of making DTrace's license incompatible with Linux, or was the choice of license made specifically to cause that effect? There appear to be conflicting accounts from people who were at Sun at the time.
evmar 13y ago
I recently dove into this area in an attempt to find something like strace that didn't cost as much (as the slides mention, it's hundreds of times slower to run something under strace). It seems like dtrace can likely do what I want, but it doesn't sound reliable under linux yet. Does anyone know if the native linux profiling tools can be used to answer questions like "which files is my process accessing"? As far as I found, various tools can trace syscalls and count them, but the critical thing strace adds is that it decodes the pointer parameters to syscalls into strings, something that e.g. "perf trace" doesn't seem to do.
- noerps 13y ago
  Try lsof for files accessed by process.
  - 1amzave 13y ago
    Correct me if I'm wrong here, but I believe lsof can only display files held open at a given point in time (when you run lsof), no? So if a process opens a file, does something with it, and then immediately closes it, you're unlikely to be able to catch that with lsof unless you happen to run it at exactly the right time.
- joshbaptiste 13y ago
  Ktap was recently announced also https://github.com/ktap/ktap https://github.com/ktap/ktap
- justincormack 13y ago
  You could link your library to a passthrough libc that logs calls, if it is dynamically linked. There are various library hijacking solutions to make it easier. ptrace, that strace uses is necessarily slow (ktrace in eg NetBSD is much faster, the kernel does the tracing not userspace).

127 points by jswanson 13y ago

Awesome talk, though the SystemTap coverage was oddly weak. Saying 'SystemTap was was created when there weren't Solaris for Linux ports' is a little disingenuous. Linux had live kernel instrumentation with kprobes way before Sun debuted DTrace, and DTrace was deliberately licensed in a way to be incompatible with the Linux kernel. If you want to see some actual useful production taps you can use right now on your Red Hat / CentOS box, check out http://sourceware.org/systemtap/wiki/WarStories http://sourceware.org/systemtap/wiki/WarStories

insaneirish 13y ago
DTrace was deliberately licensed in a way to be incompatible with the Linux kernel. This was commented on directly by Bryan Cantrill on HN [1]. You should also watch his "Fork Yeah!" [2] and "Corporate Open Source Anti-Patterns" [3] talks for full context. [1]: https://news.ycombinator.com/item?id=4357507 https://news.ycombinator.com/item?id=4357507 [2]: http://www.youtube.com/watch?v=-zRN7XLCRhc http://www.youtube.com/watch?v=-zRN7XLCRhc [3]: http://www.youtube.com/watch?v=NhgXQFk9noI http://www.youtube.com/watch?v=NhgXQFk9noI
- alanmackenzie 13y ago
  Great links.
- ehsanu1 13y ago
  I want to point out a comment [1] apparently by the former chief open source evangelist, Danese Cooper [2], on the "Fork Yeah!" video contradicting Bryan Cantrill on this point. Here it is reproduced: Lovely except it really was decided to explicitly make OpenSolaris incompatible with GPL. That was one of the design points of the CDDL. I was in that room, Bryan and you were not, but I know its fun to re-write history to suit your current politics. I pleaded with Sun to use a BSD family license or the GPL itself and they would consider neither because that would have allowed D-Trace to end up in Linux. You can claim otherwise all you want...this was the truth in 2005 [1] http://www.youtube.com/watch?v=-zRN7XLCRhc&lc=UEj5uH4rMafUnXNekr8VLEFBfO2UpbRVyNaLbGRcDzs http://www.youtube.com/watch?v=-zRN7XLCRhc&lc=UEj5uH4rMafUnX... [2] http://en.wikipedia.org/wiki/Danese_Cooper http://en.wikipedia.org/wiki/Danese_Cooper
  - gillianseed 13y ago
    I fully trust in Danese's description of the events in this case. Not only because she (Danese) was the person who actually created the licence (and thus is the best authority on the prerequisites set by the Sun management), but also because it makes perfect sense. Sun, a company whose Solaris product was suffering greatly under competition from Linux, would NOT want to offer up their systems technical advantages under a open source licence which would allow Linux to use said technical advantages. It seems purely logical to me. Listening further to Danese, she describes the Sun management as wanting a copyleft style licence for the code, and that they were eyeing GPLv3, but not GPLv2 which was already available (again, GPLv3 code would not be compatible with the Linux kernel, while GPLv2 obviously would), however GPLv3 was taking to long to be finalized so they set her (Danese) on the task of creating a new licence.
    - brendangregg 13y ago
      I've recently been using DTrace on Ubuntu. I filed two bugs on the project page (https://github.com/dtrace4linux/linux https://github.com/dtrace4linux/linux), and Paul Fox fixed them quickly. I have more bugs to file, but I'm doing my part in making DTrace on Linux a reality. It's very tempting to join these licensing discussions (I was at Sun, too, and many of us DID want to see DTrace on Linux), but I think that's a distraction from what's happening right now. There are two projects porting DTrace to Linux, which is really exciting. I'm helping out.
  - justin66 13y ago
    > Lovely except it really was decided to explicitly make OpenSolaris incompatible with GPL. In fairness to both sides, pretty much everything that isn't GPL can be said to be incompatible with GPL, and almost anyone who decides to use something different than the GPL can be said to be deliberately incompatible with the GPL. That's just the nature of the GPL.
    - _delirium 13y ago
      Permissive open-source licenses (MIT, BSD, etc.) are GPL-compatible. But I think the question here is more one of motivation: was a decision made for reason X which had the side-effect of making DTrace's license incompatible with Linux, or was the choice of license made specifically to cause that effect? There appear to be conflicting accounts from people who were at Sun at the time.

evmar 13y ago

I recently dove into this area in an attempt to find something like strace that didn't cost as much (as the slides mention, it's hundreds of times slower to run something under strace). It seems like dtrace can likely do what I want, but it doesn't sound reliable under linux yet. Does anyone know if the native linux profiling tools can be used to answer questions like "which files is my process accessing"? As far as I found, various tools can trace syscalls and count them, but the critical thing strace adds is that it decodes the pointer parameters to syscalls into strings, something that e.g. "perf trace" doesn't seem to do.

noerps 13y ago
Try lsof for files accessed by process.
- 1amzave 13y ago
  Correct me if I'm wrong here, but I believe lsof can only display files held open at a given point in time (when you run lsof), no? So if a process opens a file, does something with it, and then immediately closes it, you're unlikely to be able to catch that with lsof unless you happen to run it at exactly the right time.
joshbaptiste 13y ago
Ktap was recently announced also https://github.com/ktap/ktap https://github.com/ktap/ktap
justincormack 13y ago
You could link your library to a passthrough libc that logs calls, if it is dynamically linked. There are various library hijacking solutions to make it easier. ptrace, that strace uses is necessarily slow (ktrace in eg NetBSD is much faster, the kernel does the tracing not userspace).