4 ms·

Linux Perf_events Off-CPU Time Flame Graph

55 points by hepha1979 12y ago

rodgerd 12y ago
As a general observation, and coming from a background spending a lot of time on performance troubleshooting, Brendan spending time doing performance work on Linux is a huge win for Linux bods everywhere, and a big loss to the Solaris community.
bentcorner 12y ago
Can anyone who has dealt with Linux perf and Windows perf comment? I've spent a fair amount of time dealing with xperf/WPA and spelunking ETLs and find that it's a really nice environment. Does something similar exist for the Linux world? This seems similar to wait chain analysis, although it's something that I'm still trying to get better at, so I can't really draw any parallels here.
- brendangregg 12y ago
  I've done some Xperf/ETW, and saw many similarities to Linux perf: various events to trace, in groups, and a trace->dump->post-process workflow. Linux perf has good support for profiling performance monitoring counters, from instructions per cycle to uops events. I haven't seen them instrumented on Windows, other than by Intel VTune. Linux perf can also do kernel & user-level dynamic tracing; I've heard such support exists on Windows, but not seen it done yet. The event documentation with Windows ETW looked better. On Linux, some of the events (tracepoints) have no documentation other than the kernel source. (But, at least you have kernel source. :) Symbols is also often more difficult to get going on Linux. Bruce Dawson (who can probably answer this Q better than anyone) has written about it in the past: https://randomascii.wordpress.com/2013/02/20/symbols-on-linux-part-three-linux-versus-windows/ https://randomascii.wordpress.com/2013/02/20/symbols-on-linu... perf, and Xperf, are pretty good, but what I think they really need is more kernel level programability, to do (more) custom in-kernel filtering and summaries. DTrace has shown the value that this can bring. It should lower the overhead involved, and allow these tools to be used more often. Linux perf may get there with eBPF (although it does currently have in-kernel counts and custom filters)...
  - brucedawson 12y ago
    You called? I didn't know you did Windows profiling. I do some profiling on Linux, although nowhere near as much as on Windows. My observations of the differences are: I love how easy it is to measure CPU performance counters on Linux, whereas this requires VTune or similar on Windows and is generally painful. I've heard that ETW does support these performance counters but that Microsoft has yet to expose them, which is sad. I love how easy it is to get counters like context-switch counts on Linux. Just run your command under perf and the results are available the instant it finishes. perf top is also a wonderful thing for real-time investigations. So, Linux windows for quick access, access to real-time information, and CPU performance counters. xperf/ETW/WPA wins a lot of points for the UI. This allows exploring details. For instance, on Linux I can trivially see how many context switches there are, but zooming in on an individual context switch to find out when and why it happened... I wouldn't know how to begin. Brendan said: > The event documentation with Windows ETW looked better. I thought that was curious because I found the documentation of the WPA columns so atrociously non-existent that I wrote a series of blog posts to summarize what I had figured out: https://randomascii.wordpress.com/2012/05/11/the-lost-xperf-documentationcpu-scheduling/ https://randomascii.wordpress.com/2012/05/11/the-lost-xperf-... At least they have tooltips now -- that's progress. So, Windows needs access to CPU performance counters, kernel level programmability, and ability to quickly get back statistics. Linux needs a UI for exploring the data from "perf record", and Linux needs a better story for retrieving related symbols and source automatically. In short, both platforms should try to copy the best features of the other because they are both missing vital items. BTW, nice job on the idle-time flame graphs, but I'd definitely still give xperf/ETW/WPA the advantage, especially when tracing wait-chains that span processes.
    - brucedawson 12y ago
      Update: did somebody say they wanted a better UI for Linux perf? I just saw this: http://www.rotateright.com/zoom-release-notes/ http://www.rotateright.com/zoom-release-notes/ In particular: Removed licensing restriction: Zoom is transitioning into a free software product and is no longer available for sale I used Zoom as a commercial product on Linux and found it helpful, although licensing bugs eventually made it work poorly and I have not used it since changing jobs. It looks like I may be using it again.

55 points by hepha1979 12y ago

As a general observation, and coming from a background spending a lot of time on performance troubleshooting, Brendan spending time doing performance work on Linux is a huge win for Linux bods everywhere, and a big loss to the Solaris community.

bentcorner 12y ago

Can anyone who has dealt with Linux perf and Windows perf comment? I've spent a fair amount of time dealing with xperf/WPA and spelunking ETLs and find that it's a really nice environment. Does something similar exist for the Linux world? This seems similar to wait chain analysis, although it's something that I'm still trying to get better at, so I can't really draw any parallels here.

brendangregg 12y ago
I've done some Xperf/ETW, and saw many similarities to Linux perf: various events to trace, in groups, and a trace->dump->post-process workflow. Linux perf has good support for profiling performance monitoring counters, from instructions per cycle to uops events. I haven't seen them instrumented on Windows, other than by Intel VTune. Linux perf can also do kernel & user-level dynamic tracing; I've heard such support exists on Windows, but not seen it done yet. The event documentation with Windows ETW looked better. On Linux, some of the events (tracepoints) have no documentation other than the kernel source. (But, at least you have kernel source. :) Symbols is also often more difficult to get going on Linux. Bruce Dawson (who can probably answer this Q better than anyone) has written about it in the past: https://randomascii.wordpress.com/2013/02/20/symbols-on-linux-part-three-linux-versus-windows/ https://randomascii.wordpress.com/2013/02/20/symbols-on-linu... perf, and Xperf, are pretty good, but what I think they really need is more kernel level programability, to do (more) custom in-kernel filtering and summaries. DTrace has shown the value that this can bring. It should lower the overhead involved, and allow these tools to be used more often. Linux perf may get there with eBPF (although it does currently have in-kernel counts and custom filters)...
- brucedawson 12y ago
  You called? I didn't know you did Windows profiling. I do some profiling on Linux, although nowhere near as much as on Windows. My observations of the differences are: I love how easy it is to measure CPU performance counters on Linux, whereas this requires VTune or similar on Windows and is generally painful. I've heard that ETW does support these performance counters but that Microsoft has yet to expose them, which is sad. I love how easy it is to get counters like context-switch counts on Linux. Just run your command under perf and the results are available the instant it finishes. perf top is also a wonderful thing for real-time investigations. So, Linux windows for quick access, access to real-time information, and CPU performance counters. xperf/ETW/WPA wins a lot of points for the UI. This allows exploring details. For instance, on Linux I can trivially see how many context switches there are, but zooming in on an individual context switch to find out when and why it happened... I wouldn't know how to begin. Brendan said: > The event documentation with Windows ETW looked better. I thought that was curious because I found the documentation of the WPA columns so atrociously non-existent that I wrote a series of blog posts to summarize what I had figured out: https://randomascii.wordpress.com/2012/05/11/the-lost-xperf-documentationcpu-scheduling/ https://randomascii.wordpress.com/2012/05/11/the-lost-xperf-... At least they have tooltips now -- that's progress. So, Windows needs access to CPU performance counters, kernel level programmability, and ability to quickly get back statistics. Linux needs a UI for exploring the data from "perf record", and Linux needs a better story for retrieving related symbols and source automatically. In short, both platforms should try to copy the best features of the other because they are both missing vital items. BTW, nice job on the idle-time flame graphs, but I'd definitely still give xperf/ETW/WPA the advantage, especially when tracing wait-chains that span processes.
  - brucedawson 12y ago
    Update: did somebody say they wanted a better UI for Linux perf? I just saw this: http://www.rotateright.com/zoom-release-notes/ http://www.rotateright.com/zoom-release-notes/ In particular: Removed licensing restriction: Zoom is transitioning into a free software product and is no longer available for sale I used Zoom as a commercial product on Linux and found it helpful, although licensing bugs eventually made it work poorly and I have not used it since changing jobs. It looks like I may be using it again.