4 ms·

Awesome post, as always. I know you've at one point taken a look at shark (https://github.com/sharklinux/shark https://github.com/sharklinux/shark), which seems

by aktau 11y ago

Awesome post, as always. I know you've at one point taken a look at shark (https://github.com/sharklinux/shark https://github.com/sharklinux/shark), which seems to be similar in a way except one makes probes (either perf_events or eBPF) in just Lua. Do you have any notes about how it compares vis-a-vis BCC? This seems to be an example eBPF back and front-end: https://github.com/sharklinux/shark/blob/master/samples/bpf/io_sys_write_hist.lua https://github.com/sharklinux/shark/blob/master/samples/bpf/... From the looks of it, the eBPF program is also specified in some C (dialect?). I wonder how it's similar/different to BCC. From the dependencies on shark, LLVM is also involved. EDIT: seems like they're encouraging people to download a precompiled llc-bpf for now: https://github.com/sharklinux/llc-bpf https://github.com/sharklinux/llc-bpf EDIT2: (sorry for all the edits). My last question would be: you say "What's new in Linux 4.3 is the ability to print strings from Extended Berkeley Packet Filters (eBPF) programs.", and then that you needed this for many tools. However, I can't see that in the biolatency source you pasted. It seems to me like it's aggregating a map in kernel-space, and printing it out from userspace once it's done. Just like was possible in kernel 4.1+. What am I missing?

drzaeus77 11y ago
Author of BCC here... I hadn't seen shark before, but the mindset does appear similar. A couple comparison points I noticed while briefly investigating shark: C code is passed to clang+llc as external calls, whereas in BCC clang+llvm are statically linked in. Both support native (lua/python) bindings to the eBPF maps. In shark, I don't see that it is easy to dereference kprobe'd function arguments, as it is in `bpf_trace_printk("1 W %s %d %d ?\\n", req->rq_disk->disk_name, ...)` of <bcc>/tools/biosnoop. This should also answer your last question, which was where is "%s" used. tools/opensnoop also uses string printks. Comparison points aside, I intentionally made sure that the clang legwork that is being done in BCC is wrapped with a C api, so any language bindings besides python should be trivial to implement. It would be ideal (in my mind) if shark could leverage libbcc and make both tools better in the process.
- fche 11y ago
  Can you outline how bcc translates that req->rq_disk->disk_name expression to bytecode? AIUI, there are no pointer-dereferencing bytecodes. Is it using the BPF_FUNC_probe_read?
  - drzaeus77 11y ago
    Exactly, it is using bpf_probe_read. Internally, BCC uses clang's Rewriter functionality to mangle valid C (but invalid BPF) into valid C with bpf helper functions. The req->rq_disk->disk_name expression would expand into: ({ typeof(char [32]) _val; memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (u64)({ typeof(struct gendisk *) _val; memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (u64)req + offsetof(struct request, rq_disk)); _val; }) + offsetof(struct gendisk, disk_name)); _val; })); If you are playing with the tools, the BPF() class takes an optional argument debug=, where bit 2 (0x4) will print the rewritten C output for your edification.
brendangregg 11y ago
Shark looks promising... I haven't tried it yet so I don't have much of an opinion. Contributions seem a bit sparse: it's not in heavy development. Maybe that will change over time.

by aktau 11y ago

drzaeus77 11y ago

Author of BCC here... I hadn't seen shark before, but the mindset does appear similar. A couple comparison points I noticed while briefly investigating shark: C code is passed to clang+llc as external calls, whereas in BCC clang+llvm are statically linked in. Both support native (lua/python) bindings to the eBPF maps. In shark, I don't see that it is easy to dereference kprobe'd function arguments, as it is in `bpf_trace_printk("1 W %s %d %d ?\\n", req->rq_disk->disk_name, ...)` of <bcc>/tools/biosnoop. This should also answer your last question, which was where is "%s" used. tools/opensnoop also uses string printks. Comparison points aside, I intentionally made sure that the clang legwork that is being done in BCC is wrapped with a C api, so any language bindings besides python should be trivial to implement. It would be ideal (in my mind) if shark could leverage libbcc and make both tools better in the process.

fche 11y ago
Can you outline how bcc translates that req->rq_disk->disk_name expression to bytecode? AIUI, there are no pointer-dereferencing bytecodes. Is it using the BPF_FUNC_probe_read?
- drzaeus77 11y ago
  Exactly, it is using bpf_probe_read. Internally, BCC uses clang's Rewriter functionality to mangle valid C (but invalid BPF) into valid C with bpf helper functions. The req->rq_disk->disk_name expression would expand into: ({ typeof(char [32]) _val; memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (u64)({ typeof(struct gendisk *) _val; memset(&_val, 0, sizeof(_val)); bpf_probe_read(&_val, sizeof(_val), (u64)req + offsetof(struct request, rq_disk)); _val; }) + offsetof(struct gendisk, disk_name)); _val; })); If you are playing with the tools, the BPF() class takes an optional argument debug=, where bit 2 (0x4) will print the rewritten C output for your edification.

brendangregg 11y ago

Shark looks promising... I haven't tried it yet so I don't have much of an opinion. Contributions seem a bit sparse: it's not in heavy development. Maybe that will change over time.