3 ms·
BPF is indeed a pretty interesting technology. As the knowledge about it becomes more widespread, I anticipate that we will unlock some new capabilities both in
by pwnna 5y ago
BPF is indeed a pretty interesting technology. As the knowledge about it becomes more widespread, I anticipate that we will unlock some new capabilities both in terms of tracing. Brendan Gregg's book (https://www.brendangregg.com/bpf-performance-tools-book.html https://www.brendangregg.com/bpf-performance-tools-book.html) serves as a good intro to this, although you probably only need to read a small chunk of it as a lot of it is reference-book-style material.
The author's mentioned that you can trace MySQL with USDT, which is a tracepoint inserted by the developer at select locations in the code. This kind of tracepoints form a "stable interface" for tracing/performance debugging, whereas uprobe, which hooks into select userspace functions, are unstable as the binary is recompiled. Unfortunately, the USDT tracepoints (via DTrace) have been removed in MySQL 8.0. This makes it significantly more difficult to trace MySQL, although it's not impossiblhttps://news.ycombinator.com/item?id=29772927e https://news.ycombinator.com/item?id=29772927e. I've done a proof of concept of tracing MySQL with uprobe instead of USDT in this repo[1], which can kind of give you the same results (and possibly more stuff, as I can more easily read arbitrary memory address due to how the old USDT tracepoints are structured). This is not stable tho, as any MySQL upgrade may introduce incompatibility with the trace script, as I read memory address based on offsets (whereas with USDT this can be kept pretty stable). My appeal to Oracle to re-add this functionality[2] has unfortunately been rejected, which I think is a mistake given the wide range of possibilities unlocked via BPF.
[1]: https://github.com/shuhaowu/mysqld-bpf https://github.com/shuhaowu/mysqld-bpf
[2]: https://bugs.mysql.com/bug.php?id=105741 https://bugs.mysql.com/bug.php?id=105741
Another thing that I've been recently thinking of is using BPF to validate programs written for real-time Linux (via PREEMPT_RT). To my understanding, one of the main thing to avoid is page faults [3]. With the proper BPF tracing scripts, I think we can validate that programs indeed avoids page faults in integration testing. I'm not sure if it is super useful yet, but as I'm trying to write a few RT programs, it's something that came to my mind.
[3]: https://lwn.net/Articles/837019/ https://lwn.net/Articles/837019/
In addition to tracing (so bpftrace-based/bcc-based tools), I've recently discovered that there there are:
1. ebpfsnitch (https://github.com/harporoeder/ebpfsnitch https://github.com/harporoeder/ebpfsnitch): which is an application-level firewall without kernel modules.
2. ebpf-traffic-monitor (https://source.android.com/devices/tech/datausage/ebpf-traffic-monitor https://source.android.com/devices/tech/datausage/ebpf-traff...): which appears to be using BPF to account for traffic for different apps on Android.
3. kubectl trace (https://github.com/iovisor/kubectl-trace https://github.com/iovisor/kubectl-trace): Run tracing on k8s.
There are apparently also use cases in the context of security, but I'm not familiar with it.
- Sirened 5y ago>With the proper BPF tracing scripts, I think we can validate that programs indeed avoids page faults Sorry, I'm a little confused why this would be necessary? Like, sure, it's a nice to have on a CI as a basic sanity check but if you just invoke mlockall you'll end up with everything wired down and you're good to go regardless?
- justicezyx 5y agomloclall () requires the application under check to call them such that the application process cannot modify memory pages. Whereas ebpf allows instrumentation free enforcement. Plus, app devs do not need to be aware of this fact. This facilitate separate of responsibility in code and organization.
- pwnna 5y agoYeah you're right on that one. I probably over-extended the use case here as I'm still just learning about this and thus tried to apply it to everywhere. I guess one thing you can maybe do is to use CI to validate that mlockall indeed has been called, haha. That said it's probably overkill as other tools can probably do this too. While on that thought experiment, perhaps in general if you can use BPF/USDT to help trace/debug RT programs? I'm thinking of being able to verify/visualize timing for better tracing? Or maybe there are already tools that existing that I don't really know how to use (like ftrace + trace compass, maybe)