4 ms·

This is so cool. Netflix seems to have a damn fine eng team. Its too bad they don't have roles for new grads :P Is there an authoritative beginners guide to li

This is so cool. Netflix seems to have a damn fine eng team. Its too bad they don't have roles for new grads :P Is there an authoritative beginners guide to linux performance analysis? A cursory glance at trace-cmd tutorials shows I have a lot to learn. Or maybe it would be more beneficial to start playing with a tool like sysdig?

cthalupa 12y ago
Brendan's performance in the cloud/enterprise book is super solid. Not only does it give information on performance analysis, but it's a very solid primer to the systems involved. It's not specifically Linux - lots of solaris/illumos stuff in it too - but it's definitely a starting point. Not a ton on ftrace in it, though. Maybe we'll see a new book more focused on Linux specifically with some ftrace stuff?
- brendangregg 12y ago
  Thanks, yes, my Systems Performance book focused on the how and why of performance analysis: background, concepts, architecture, methodologies. The implementation specifics (which are an application of the earlier content) were covered last in each chapter, which were Linux and Solari. I had some perf_events but not ftrace. I've been thinking of how best to cover my new ftrace and perf_events content in book form...
brendangregg 12y ago
One problem is just knowing what's out there. My tools diagrams on http://www.brendangregg.com/linuxperf.html http://www.brendangregg.com/linuxperf.html should help, and my last LinuxCon talk on Linux Performance Tools -- a good overview of all the stuff. As for learning how to do all the stuff... If you're a programmer, a good approach is to write your own simple poor performance programs to anaylze. Eg: - burn CPU in a function - do too much disk I/O in a function - do lots of large size network I/O from a function - do heavy memory allocation from a function Now analyze these using the Linux toolset. Quantify the behavior (how much per second), show its effect on system resources (%utilized, or IOPS, or whatever), then drill down and identify the code path responsible: the function. This sounds easy, since you wrote the program to start with, and already know the answer. But there's often a lot of quirky behavior with perf tools, and assumed knowledge to learn, that you want the target to be as easy as possible. A mistake beginners make is to aim performance tools at complex production workloads, and then get lost and confused. It's like trying to run before knowing how to walk. I think Julia Evans is going to cover this approach in an upcoming talk, so that should become a good reference. You'll also want to develop a good understanding of software, systems, and kernels, so that you can reason about performance. That may take years to pick up, and the study of various text books. I tried to summarize all the content in my last book, systems performance. Edit: formatting.

cthalupa 12y ago

Brendan's performance in the cloud/enterprise book is super solid. Not only does it give information on performance analysis, but it's a very solid primer to the systems involved. It's not specifically Linux - lots of solaris/illumos stuff in it too - but it's definitely a starting point. Not a ton on ftrace in it, though. Maybe we'll see a new book more focused on Linux specifically with some ftrace stuff?

brendangregg 12y ago
Thanks, yes, my Systems Performance book focused on the how and why of performance analysis: background, concepts, architecture, methodologies. The implementation specifics (which are an application of the earlier content) were covered last in each chapter, which were Linux and Solari. I had some perf_events but not ftrace. I've been thinking of how best to cover my new ftrace and perf_events content in book form...

brendangregg 12y ago

One problem is just knowing what's out there. My tools diagrams on http://www.brendangregg.com/linuxperf.html http://www.brendangregg.com/linuxperf.html should help, and my last LinuxCon talk on Linux Performance Tools -- a good overview of all the stuff. As for learning how to do all the stuff... If you're a programmer, a good approach is to write your own simple poor performance programs to anaylze. Eg: - burn CPU in a function - do too much disk I/O in a function - do lots of large size network I/O from a function - do heavy memory allocation from a function Now analyze these using the Linux toolset. Quantify the behavior (how much per second), show its effect on system resources (%utilized, or IOPS, or whatever), then drill down and identify the code path responsible: the function. This sounds easy, since you wrote the program to start with, and already know the answer. But there's often a lot of quirky behavior with perf tools, and assumed knowledge to learn, that you want the target to be as easy as possible. A mistake beginners make is to aim performance tools at complex production workloads, and then get lost and confused. It's like trying to run before knowing how to walk. I think Julia Evans is going to cover this approach in an upcoming talk, so that should become a good reference. You'll also want to develop a good understanding of software, systems, and kernels, so that you can reason about performance. That may take years to pick up, and the study of various text books. I tried to summarize all the content in my last book, systems performance. Edit: formatting.