4 ms·
KUtrace: Low-overhead Linux kernel tracing facility
- CalChris 2y agoSite's article Benchmarking "Hello, World!" is basically a KUTrace tutorial. https://queue.acm.org/detail.cfm?id=3291278 https://queue.acm.org/detail.cfm?id=3291278 Also, https://www.youtube.com/watch?v=D_qRuKO9qzM https://www.youtube.com/watch?v=D_qRuKO9qzM
- ndesaulniers 2y agoEarly in my career, I reached out to the author and was able to grab lunch with him; he was about to retire! Was insightful to hear his thoughts on system performance, particularly systems involving more than one machine, which is something they studied deeply. It gave me appreciation for the amount of knowledge one accumulates over a career, and what a loss it is to an organization when one so knowledgeable retires.
- c0deR3D 2y agoOut of curious, does BPF now capable of capturing all the context switch events such as CPU trap? Also, if the overhead is negligible, maybe the author can try to merge this into mainline with the use of static key to make the incurred overhead switchable. In spite of the static key, the degree of the accompanied inteferences on cache and branch predictor might be an intriguing topic though.
- tanelpoder 2y agoLots of the low level exception/trap/fault handling functions are blacklisted, probably to avoid lockups and unwanted recursion mayhem: $ sudo wc -l /sys/kernel/debug/kprobes/blacklist 783 /sys/kernel/debug/kprobes/blacklist Edit: Perhaps an alternative approach would be to attach probes to relevant (precise) PMU events. There's also this prototype of adding breakpoint/watchpoint support to eBPF [1]. But actually doing stuff within this context may get complicated very fast, so would need to be severely limited, if feasible at all. [1] https://ebpf.io/summit-2020-slides/eBPF_Summit_2020-Lightning-Manali_Shukla-Hardware_Breakpoint_implementation_in_BCC.pdf https://ebpf.io/summit-2020-slides/eBPF_Summit_2020-Lightnin...
- usr1106 2y agoSounds very interesting. But it works by patching the kernel, not just using eBPF like many performance tools recently. So it needs active maintenance all the time considering the current velocity of internal kernel changes. And I would not be surprised if it didn't build or work correctly if you have a heavily patched and customized kernel. On the positive side at a first glimpse the maintenance to adapt to new kernels looks very active.
- elric 2y agoIs the naming intentional? Or just a weird coincidence? Kut being Dutch for cunt, by an author called dick..?
- crissaegrim 2y agoKUtrace stands for "Kernel-User" trace as (most of) the tracepoints are at the kernel-userspace boundary. So it is a coincidence.
- deleted 2y ago[deleted]
- xoranth 2y agoHow would this interact with `io_uring`, especially the polling modes (IO_SETUP_SQPOLL, IO_SETUP_IOPOLL)?
- rwmj 2y agoYou still have to wait for your cache to reload from main memory, or for disk or network I/O, or for processes to be scheduled to run, so while it's likely more efficient than epoll approaches, I doubt there's any really fundamental difference in the performance problems you would find.
- li4ick 2y agoHis book, "Understanding Software Dynamics", is one of the best technical books I've ever read. Top 3 for me.
- Quizzical4230 2y agoIf you don't mind me asking, what are the other two?
- deleted 2y ago[deleted]
- li4ick 2y ago- Managing Gigabytes - Hacker's Delight
- jeroenvlek 2y agoMuch appreciated. I bought all 3.
- Quizzical4230 2y agoThanks a bunch! Will check them out.
- crissaegrim 2y agoI had the pleasure to work with Dick on getting KUtrace to work on Android devices last year. It was a great experience to work with one of the greats in systems performance. He was a wealth of information regarding performance bottlenecks and optimizations. KUtrace is absolutely one of the most powerful tools I've used for deeply understanding performance bottlenecks (after isolating issues) such as poor scheduling behavior. I would highly recommend reading his book "Understanding Software Dynamics" [1] if you are interested in learning more about KUtrace or performance bottlenecks/optimizations in general. The book is quite dense and dives deep into the performance characteristics of many examples of the five fundamental resources (according to Dick): CPU, Memory, Disk/SSD, Network, and Software critical sections. [1]: https://www.oreilly.com/library/view/understanding-software-dynamics/9780137589692/ https://www.oreilly.com/library/view/understanding-software-...
- fschutze 2y agoInteresting! Does it also work on non-rooted Android devices?
- crissaegrim 2y agoNo it unfortunately does not. You require root to remount the read-only system partition, to insert the kernel modules, and to turn SELinux off. We used a "userdebug" build to get root, but I imagine most of this could also be done with a phone rooted through other means (I haven't tried it, however).
- anonymousDan 2y agoThanks, looks interesting. Does it cover measuring memory bandwidth consumption? This is something I feel there is a lack of good tooling for.
- crissaegrim 2y agoWhat are you precisely trying to measure? Theoretically if you know the performance counters you want to measure, you can replace the IPC counter in the kernel module. I believe Dick has a different version of the kernel module which measure LLC misses instead of CPU cycles. Does that answer your question?
- deleted 2y ago[deleted]
- mgaunard 2y agoWhy not just use the eBPF system?
- jeffbee 2y agoMost people would. The author makes a comparison in this interview: https://www.usenix.org/system/files/login/articles/login_fall20_04_sites.pdf https://www.usenix.org/system/files/login/articles/login_fal... Note also that this work emerged within Google a decade before eBPF was really useful.
- wyldfire 2y agoWasn't there similar useful stuff from systemtap before eBPF? That's been around for quite a while.
- jeffbee 2y agoTrue again, but SystemTap overhead is 20-100x higher.
- zokier 2y agoSystemtap, for whatever reason, remained largely RedHat solution. It didn't really have much traction outside that diaspora.
- tanelpoder 2y agoThe other alternative is LTTng, released in 2005 (requires loading a kernel module): https://lttng.org/ https://lttng.org/
- mgaunard 2y agoArgument is that KUTrace is faster, which honestly I'm not sure of.
- mydriasis 2y agoI thought this read "training facility", and I was excited to sign up!
- vhvj 2y ago[dead]
- dfgdgf 2y ago[flagged]