One problem is just knowing what's out there. My tools diagrams on http://www.brendangregg.com/linuxperf.html http://www.brendangregg.com/linuxperf.html should

One problem is just knowing what's out there. My tools diagrams on http://www.brendangregg.com/linuxperf.html http://www.brendangregg.com/linuxperf.html should help, and my last LinuxCon talk on Linux Performance Tools -- a good overview of all the stuff. As for learning how to do all the stuff... If you're a programmer, a good approach is to write your own simple poor performance programs to anaylze. Eg: - burn CPU in a function - do too much disk I/O in a function - do lots of large size network I/O from a function - do heavy memory allocation from a function Now analyze these using the Linux toolset. Quantify the behavior (how much per second), show its effect on system resources (%utilized, or IOPS, or whatever), then drill down and identify the code path responsible: the function. This sounds easy, since you wrote the program to start with, and already know the answer. But there's often a lot of quirky behavior with perf tools, and assumed knowledge to learn, that you want the target to be as easy as possible. A mistake beginners make is to aim performance tools at complex production workloads, and then get lost and confused. It's like trying to run before knowing how to walk. I think Julia Evans is going to cover this approach in an upcoming talk, so that should become a good reference. You'll also want to develop a good understanding of software, systems, and kernels, so that you can reason about performance. That may take years to pick up, and the study of various text books. I tried to summarize all the content in my last book, systems performance. Edit: formatting.