5 ms·

Learn EBPF Tracing: Tutorial and Examples

147 points by knoxa2511 8y ago

letstrynvm 8y ago
I use HTTPS Everywhere plugin for firefox... it's pretty surprising in 2019 how many network-related blogs and articles are on http links and the https equivalent is broken. You can use Let's Encrypt, it's free. It makes me not want to listen to what's supposed to be their wisdom on networking matters if they can't even get that right.
- hossbeast 8y ago
  You're going to miss out on something amazing, then.
- perfmode 8y ago
  Gregg isn’t a bad engineer. In fact, quite the opposite. Do yourself a favor and overlook the TLS on his read only blog.
  - letstrynvm 8y ago
    Maybe he should do himself a favor and fix his broken :443, or disable it.
- zamadatix 8y ago
  It's a performance blog not a networking blog. eBOF isn't limited to networking nor was this blog post.
  - letstrynvm 8y ago
    ... I said his https is broken. His https is broken. Does that really need downvoting into oblivion?
    - letstrynvm 8y ago
      His https is still broken. Maybe he'll fix it if someone points it out to him.
    - cyphar 8y ago
      That isn't all you said. The last line was unnecessary in making your point, and wasn't polite. That's why people are downvoting you.
- dogecoinbase 8y ago
  HTTPS (and DNSSEC!) are antithetical to the idea of proper network engineering. Hierarchical, centralized control systems. Let's Encrypt being free should only make you more suspicious about who's making money from whom. No thank you.
  - letstrynvm 8y ago
    Who is making money off let's encrypt? The service is provided for free.
    - dogecoinbase 8y ago
      Nothing is free.
m1el 8y ago
I want to try out eBPF tracing in Linux. I really do. Last three times I tried to install bcc/bpftrace on my Debian boxes, I failed to do so. The situation has not changed for over a year. I'm not the only one: https://github.com/iovisor/bcc/issues/678 https://github.com/iovisor/bcc/issues/678 https://github.com/iovisor/bcc/issues/1434 https://github.com/iovisor/bcc/issues/1434 https://github.com/iovisor/bcc/issues/1985 https://github.com/iovisor/bcc/issues/1985
- qdog 8y ago
  I have ran ebpf on ubuntu, so not sure what the difference is with Debian. I will note that clang 3.7 is fairly old at this point, I have started building a more recent llvm+clang (7.0.0 stable) myself instead of relying on the installed package, then building bcc against that. You might try that route, although llvm takes an eternity to build.
  - codetrotter 8y ago
    > You might try that route, although llvm takes an eternity to build. You don’t need to build it from source yourself. The LLVM people build and host packages for Debian and Ubuntu in package repos that anyone can install from. Instructions here: https://apt.llvm.org/ https://apt.llvm.org/
    - qdog 8y ago
      Ah, true. I should note I build a statically linked llvm so I can deploy my project on different versions. The pre-built llvm should work for most people.
lower 8y ago
From the bpftrace tutorial, I would have expected the one-liner bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%d %s\n", pid, str(args->filename)); }' to show me all open calls as they happen. I would have expected to see an open when I cat a file, for example. But trying the one-liner, I only see a few opens of files in /proc. Can anyone explain what's happening?
- danobi 8y ago
  It actually turns out most of the calls are `openat(2)`. I had the same question initially.
  - livueta 8y ago
    (don't have access to a linux box I can test this on at the moment, so sorry for the stupid question) In DTrace, you can specify a probe like syscall::*open*:entry / / { } to grab open(2), openat(2), etc. Does eBPF allow wildcards in probe specifications?
    - lower 8y ago
      You can do bpftrace -e 'tracepoint:syscalls:sys_enter_open* { printf("%d\n", pid); }' but then you can't access the arguments of the different probes uniformly, i.e. bpftrace -e 'tracepoint:syscalls:sys_enter_open* { printf("%d %s\n", pid, str(args->filename)); }' does not work. You can do it like this: bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%d %s\n", pid, str(args->filename)); } tracepoint:syscalls:sys_enter_openat { printf("%d %s\n", pid, str(args->filename)); }' This is a bit awkward, but it seems that this will be fixed: https://github.com/iovisor/bpftrace/issues/132 https://github.com/iovisor/bpftrace/issues/132
marius_k 8y ago
There was this talk[0] at 35c3 introducing eBPF tracing. I still have it on my towatch list as I could not attend it. [0] https://media.ccc.de/v/35c3-9532-kernel_tracing_with_ebpf https://media.ccc.de/v/35c3-9532-kernel_tracing_with_ebpf

147 points by knoxa2511 8y ago

I use HTTPS Everywhere plugin for firefox... it's pretty surprising in 2019 how many network-related blogs and articles are on http links and the https equivalent is broken. You can use Let's Encrypt, it's free. It makes me not want to listen to what's supposed to be their wisdom on networking matters if they can't even get that right.

hossbeast 8y ago
You're going to miss out on something amazing, then.
perfmode 8y ago
Gregg isn’t a bad engineer. In fact, quite the opposite. Do yourself a favor and overlook the TLS on his read only blog.
- letstrynvm 8y ago
  Maybe he should do himself a favor and fix his broken :443, or disable it.
zamadatix 8y ago
It's a performance blog not a networking blog. eBOF isn't limited to networking nor was this blog post.
- letstrynvm 8y ago
  ... I said his https is broken. His https is broken. Does that really need downvoting into oblivion?
  - letstrynvm 8y ago
    His https is still broken. Maybe he'll fix it if someone points it out to him.
  - cyphar 8y ago
    That isn't all you said. The last line was unnecessary in making your point, and wasn't polite. That's why people are downvoting you.
dogecoinbase 8y ago
HTTPS (and DNSSEC!) are antithetical to the idea of proper network engineering. Hierarchical, centralized control systems. Let's Encrypt being free should only make you more suspicious about who's making money from whom. No thank you.
- letstrynvm 8y ago
  Who is making money off let's encrypt? The service is provided for free.
  - dogecoinbase 8y ago
    Nothing is free.

m1el 8y ago

I want to try out eBPF tracing in Linux. I really do. Last three times I tried to install bcc/bpftrace on my Debian boxes, I failed to do so. The situation has not changed for over a year. I'm not the only one: https://github.com/iovisor/bcc/issues/678 https://github.com/iovisor/bcc/issues/678 https://github.com/iovisor/bcc/issues/1434 https://github.com/iovisor/bcc/issues/1434 https://github.com/iovisor/bcc/issues/1985 https://github.com/iovisor/bcc/issues/1985

qdog 8y ago
I have ran ebpf on ubuntu, so not sure what the difference is with Debian. I will note that clang 3.7 is fairly old at this point, I have started building a more recent llvm+clang (7.0.0 stable) myself instead of relying on the installed package, then building bcc against that. You might try that route, although llvm takes an eternity to build.
- codetrotter 8y ago
  > You might try that route, although llvm takes an eternity to build. You don’t need to build it from source yourself. The LLVM people build and host packages for Debian and Ubuntu in package repos that anyone can install from. Instructions here: https://apt.llvm.org/ https://apt.llvm.org/
  - qdog 8y ago
    Ah, true. I should note I build a statically linked llvm so I can deploy my project on different versions. The pre-built llvm should work for most people.

lower 8y ago

From the bpftrace tutorial, I would have expected the one-liner bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%d %s\n", pid, str(args->filename)); }' to show me all open calls as they happen. I would have expected to see an open when I cat a file, for example. But trying the one-liner, I only see a few opens of files in /proc. Can anyone explain what's happening?

danobi 8y ago
It actually turns out most of the calls are `openat(2)`. I had the same question initially.
- livueta 8y ago
  (don't have access to a linux box I can test this on at the moment, so sorry for the stupid question) In DTrace, you can specify a probe like syscall::*open*:entry / / { } to grab open(2), openat(2), etc. Does eBPF allow wildcards in probe specifications?
  - lower 8y ago
    You can do bpftrace -e 'tracepoint:syscalls:sys_enter_open* { printf("%d\n", pid); }' but then you can't access the arguments of the different probes uniformly, i.e. bpftrace -e 'tracepoint:syscalls:sys_enter_open* { printf("%d %s\n", pid, str(args->filename)); }' does not work. You can do it like this: bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%d %s\n", pid, str(args->filename)); } tracepoint:syscalls:sys_enter_openat { printf("%d %s\n", pid, str(args->filename)); }' This is a bit awkward, but it seems that this will be fixed: https://github.com/iovisor/bpftrace/issues/132 https://github.com/iovisor/bpftrace/issues/132

marius_k 8y ago

There was this talk[0] at 35c3 introducing eBPF tracing. I still have it on my towatch list as I could not attend it. [0] https://media.ccc.de/v/35c3-9532-kernel_tracing_with_ebpf https://media.ccc.de/v/35c3-9532-kernel_tracing_with_ebpf