SystemTap (or any other dynamic tracing tool) is very powerful; I suggest to everyone who does system-level or performance-oriented work that you learn how to u

SystemTap (or any other dynamic tracing tool) is very powerful; I suggest to everyone who does system-level or performance-oriented work that you learn how to use it. It's not as difficult as it first looks. If you understand what a syscall is, it's useful. If you understand more about the kernel insides, e.g. the various vfs functions, scheduling, memory mapping and allocation, it gets so much more powerful. It basically lets you run arbitrary code anywhere inside a kernel (or userspace). That is, at any line of code, you can break and jump to your own code and return again. Your code can do pretty much anything - there is a way to embed C in a SystemTap script. Usually, you're just keeping counters, timers, and backtraces; there is a nice little scripting language that SystemTap compiles to C for those cases. I have done a very similar analysis recently. I had an HTTP service with latency spikes. I used SystemTap to discover occasional long-running syscalls; the contents of the syscalls pointed out the exact problem. Once those latency spikes were clear, I still had a few very occasional spikes (as in this article). I created a SystemTap script that collected backtraces anytime our process was rescheduled after being asleep for longer than some time (using [1][2] to start/stop a timer). I then used that to plot a flamegraph. After that, the problem was pretty clear and I was able to intelligently mess with some kernel tuning parameters and eventually discover that it wasn't enough - I just need to deal with the problem on my side, from there the fix was pretty easy. [1] https://sourceware.org/systemtap/tapsets/API-scheduler-cpu-on.html https://sourceware.org/systemtap/tapsets/API-scheduler-cpu-o... [2] https://sourceware.org/systemtap/tapsets/API-scheduler-cpu-off.html https://sourceware.org/systemtap/tapsets/API-scheduler-cpu-o...