6 ms·

The Linux perf command rocks

188 points by pixelbeat 15y ago

joelthelion 15y ago
Does anyone know how to install it on Arch?
- b6 15y ago
  maybe https://aur.archlinux.org/packages.php?ID=31439 https://aur.archlinux.org/packages.php?ID=31439
dsantos 15y ago
the article mentions perf, valgrind, oprofile, etc. but misses the lttng tool http://lttng.org http://lttng.org
- EventHorizon 15y ago
  Is there a better viewer for lttng than the lttv gui? I got spoiled by QNX's amazing tracing infrastructure and analysis tools and long for something similar on Linux.
  - dsantos 15y ago
    lttng is integrated into eclipse framework, the gui looks better than the gtk lttv gui
bcl 15y ago
A few other tools I use are atop (more detailed top info, with logging), iostat (harddrive io) and vmstat (memory and cpu details).
- ootachi 15y ago
  Also don't forget Cachegrind. Cachegrind is amazing and runs entirely from user space.
  - lutorm 15y ago
    cachegrind is extremely useful, but unfortunately it seems that it's not updated with the current generation processors. Every time I run it, it complains about "unrecognized architecture" or something similar.
- dkarl 15y ago
  Do you know of any manual or book-style treatment about using those tools for investigating the performance of Linux systems? Every once in a while I have to look at performance problems on Linux systems, but I always start from zero. I need a place where I can study the topic and refresh myself when necessary, because I'm apparently not capable of mastering it via occasional piecemeal experience.
  - ghshephard 15y ago
    The closest I've found (and spent 10s, if not hundred+ hours reading) are: System Performance Tuning, 2nd Edition (O'Reilly System Administration) by Gian-Paolo D. Musumeci [1] and the somewhat dated (and Solaris centric) Sun Performance and Tuning: Java and the Internet (2nd Edition) [2] I'd love to hear that there were better book out there - if anyone has read either of those two, and has found a better one, I'll be one-clicking it in 30 seconds. [1] http://www.amazon.com/System-Performance-Tuning-OReilly-Administration/dp/059600284X/ref=sr_1_1?s=books&ie=UTF8&qid=1304012614&sr=1-1 http://www.amazon.com/System-Performance-Tuning-OReilly-Admi... [2] http://www.amazon.com/Sun-Performance-Tuning-Java-Internet/dp/0130952494/ref=sr_1_8?s=books&ie=UTF8&qid=1304012614&sr=1-8 http://www.amazon.com/Sun-Performance-Tuning-Java-Internet/d...
- sciurus 15y ago
  As a sysadmin. I don't love any other tool nearly as much as I love atop. It comes close to making all other interactive general-purpose performance monitoring tools redundant. The logging, too, is fantastic. There's nothing like looking through the graphs of my metrics gathered by collectd, wondering why a certain spike or pattern occurred, and being able to step through previous days per-process resource usage via atop and figure it out. There's a great introduction to it at https://lwn.net/Articles/387202/ https://lwn.net/Articles/387202/
- bostonvaulter2 15y ago
  You should try htop and iotop as well. Although atop is generally better for more sysadmin-type things than htop is.
hexley 15y ago
Is this like the "Sample" button in Activity Monitor on OS X?
- dsantos 15y ago
  its more like instruments and shark tools that you can find in xcode
ssp 15y ago
Also take a look at sysprof: http://sysprof.com http://sysprof.com The 1.1.6 release uses the same kernel API as perf, but displays full call trees. Screenshot: http://sysprof.com/screen-shot-4.png http://sysprof.com/screen-shot-4.png
- xtacy 15y ago
  Cool! The sysprof website says that there's a command line tool included, but I can't find it. Do you have any info on that?
  - ssp 15y ago
    Yes, it's called sysprof-cli. It should be built and installed out of the box.
    - xtacy 15y ago
      Hmmm, I guess it's not available on Debian wheezy. I just installed sysprof and sysprof-cli wasn't included in it. I'll install it separately; thanks!
js2 15y ago
For a recent example of perf in practice, see "git gc: Speed it up by 18% via faster hash comparisons" on the git mailing list http://thread.gmane.org/gmane.comp.version-control.git/172286 http://thread.gmane.org/gmane.comp.version-control.git/17228...
me_again 15y ago
For Windows, xperf is in the systemwide-profiler category: http://msdn.microsoft.com/en-us/performance/cc825801.aspx http://msdn.microsoft.com/en-us/performance/cc825801.aspx
joezydeco 15y ago
Do any of these run on MIPS? Profiling for embedded Linux can really suck sometimes...
- dsantos 15y ago
  lttng currently runs on: x86-32, x86-64, SPARC, SPARC64, ppc, ppc64, sh, sh64, ia64, s390, MIPS 32/64, ARM
loboman 15y ago
Not cool... "sudo perf record -a -g sleep 10" crashed my system badly (not even reisub would close it).
bsergean 15y ago
For C/C++ level user space profiling there's an excellent commercial tool with a nice GUI called Zoom. Works on Linux. To add 2 profiling tools to the long list here: DTrace and htop.
hallmark 15y ago
Are these tools (perf, atop, etc) useful and accurate in virtualized environments? For example, under Xen. Fine, I'll say it: on an EC2 or Linode instance. Would there be any skewing or clock issues due to virtualization? Collecting the comments of HN readers with deep experience in this matter, on this thread, would be preferable to me trying to give a brief and uninformed Google search summary.
- JoshTriplett 15y ago
  Some features of perf will work perfectly in an emulated environment. Perf's kernel tracing uses tracepoints in the kernel (built-in or dynamically patched in), and those will work just fine. However, perf's use of CPU performance counters will only handle those the virtual environment emulates; I don't know if Xen implements those.
vetler 15y ago
Anyone know how to install this on CentOS?
- bsergean 15y ago
  I tried to install it on Rhel5 but the kernel is too old. It compiles fine and run but fails with a (Function not implemented error). To build this I downloaded the latest linux tree, cd tools/perf + make -j install. You might get more luck with CentOS for Rhel6 ...

188 points by pixelbeat 15y ago

joelthelion 15y ago

Does anyone know how to install it on Arch?

b6 15y ago
maybe https://aur.archlinux.org/packages.php?ID=31439 https://aur.archlinux.org/packages.php?ID=31439

dsantos 15y ago

the article mentions perf, valgrind, oprofile, etc. but misses the lttng tool http://lttng.org http://lttng.org

EventHorizon 15y ago
Is there a better viewer for lttng than the lttv gui? I got spoiled by QNX's amazing tracing infrastructure and analysis tools and long for something similar on Linux.
- dsantos 15y ago
  lttng is integrated into eclipse framework, the gui looks better than the gtk lttv gui

bcl 15y ago

A few other tools I use are atop (more detailed top info, with logging), iostat (harddrive io) and vmstat (memory and cpu details).

ootachi 15y ago
Also don't forget Cachegrind. Cachegrind is amazing and runs entirely from user space.
- lutorm 15y ago
  cachegrind is extremely useful, but unfortunately it seems that it's not updated with the current generation processors. Every time I run it, it complains about "unrecognized architecture" or something similar.
dkarl 15y ago
Do you know of any manual or book-style treatment about using those tools for investigating the performance of Linux systems? Every once in a while I have to look at performance problems on Linux systems, but I always start from zero. I need a place where I can study the topic and refresh myself when necessary, because I'm apparently not capable of mastering it via occasional piecemeal experience.
- ghshephard 15y ago
  The closest I've found (and spent 10s, if not hundred+ hours reading) are: System Performance Tuning, 2nd Edition (O'Reilly System Administration) by Gian-Paolo D. Musumeci [1] and the somewhat dated (and Solaris centric) Sun Performance and Tuning: Java and the Internet (2nd Edition) [2] I'd love to hear that there were better book out there - if anyone has read either of those two, and has found a better one, I'll be one-clicking it in 30 seconds. [1] http://www.amazon.com/System-Performance-Tuning-OReilly-Administration/dp/059600284X/ref=sr_1_1?s=books&ie=UTF8&qid=1304012614&sr=1-1 http://www.amazon.com/System-Performance-Tuning-OReilly-Admi... [2] http://www.amazon.com/Sun-Performance-Tuning-Java-Internet/dp/0130952494/ref=sr_1_8?s=books&ie=UTF8&qid=1304012614&sr=1-8 http://www.amazon.com/Sun-Performance-Tuning-Java-Internet/d...
sciurus 15y ago
As a sysadmin. I don't love any other tool nearly as much as I love atop. It comes close to making all other interactive general-purpose performance monitoring tools redundant. The logging, too, is fantastic. There's nothing like looking through the graphs of my metrics gathered by collectd, wondering why a certain spike or pattern occurred, and being able to step through previous days per-process resource usage via atop and figure it out. There's a great introduction to it at https://lwn.net/Articles/387202/ https://lwn.net/Articles/387202/
bostonvaulter2 15y ago
You should try htop and iotop as well. Although atop is generally better for more sysadmin-type things than htop is.

hexley 15y ago

Is this like the "Sample" button in Activity Monitor on OS X?

dsantos 15y ago
its more like instruments and shark tools that you can find in xcode

ssp 15y ago

Also take a look at sysprof: http://sysprof.com http://sysprof.com The 1.1.6 release uses the same kernel API as perf, but displays full call trees. Screenshot: http://sysprof.com/screen-shot-4.png http://sysprof.com/screen-shot-4.png

xtacy 15y ago
Cool! The sysprof website says that there's a command line tool included, but I can't find it. Do you have any info on that?
- ssp 15y ago
  Yes, it's called sysprof-cli. It should be built and installed out of the box.
  - xtacy 15y ago
    Hmmm, I guess it's not available on Debian wheezy. I just installed sysprof and sysprof-cli wasn't included in it. I'll install it separately; thanks!

js2 15y ago

For a recent example of perf in practice, see "git gc: Speed it up by 18% via faster hash comparisons" on the git mailing list http://thread.gmane.org/gmane.comp.version-control.git/172286 http://thread.gmane.org/gmane.comp.version-control.git/17228...

me_again 15y ago

For Windows, xperf is in the systemwide-profiler category: http://msdn.microsoft.com/en-us/performance/cc825801.aspx http://msdn.microsoft.com/en-us/performance/cc825801.aspx

joezydeco 15y ago

Do any of these run on MIPS? Profiling for embedded Linux can really suck sometimes...

dsantos 15y ago
lttng currently runs on: x86-32, x86-64, SPARC, SPARC64, ppc, ppc64, sh, sh64, ia64, s390, MIPS 32/64, ARM

loboman 15y ago

Not cool... "sudo perf record -a -g sleep 10" crashed my system badly (not even reisub would close it).

bsergean 15y ago

For C/C++ level user space profiling there's an excellent commercial tool with a nice GUI called Zoom. Works on Linux. To add 2 profiling tools to the long list here: DTrace and htop.

hallmark 15y ago

Are these tools (perf, atop, etc) useful and accurate in virtualized environments? For example, under Xen. Fine, I'll say it: on an EC2 or Linode instance. Would there be any skewing or clock issues due to virtualization? Collecting the comments of HN readers with deep experience in this matter, on this thread, would be preferable to me trying to give a brief and uninformed Google search summary.

JoshTriplett 15y ago
Some features of perf will work perfectly in an emulated environment. Perf's kernel tracing uses tracepoints in the kernel (built-in or dynamically patched in), and those will work just fine. However, perf's use of CPU performance counters will only handle those the virtual environment emulates; I don't know if Xen implements those.

vetler 15y ago

Anyone know how to install this on CentOS?

bsergean 15y ago
I tried to install it on Rhel5 but the kernel is too old. It compiles fine and run but fails with a (Function not implemented error). To build this I downloaded the latest linux tree, cd tools/perf + make -j install. You might get more luck with CentOS for Rhel6 ...