5 ms·

Bpftop: Streamlining eBPF performance optimization

133 points by mochomocha 3y ago

jackhalford 3y ago
Looks great. I remember using bpftrace at work to debug a nasty performance issue, I went down the rabbit hole only to find a certain syscall was being called much too often. We managed to trace it back inside the sourcecode to a sleep(1 second), which was some sort of manual io scheduling commited by the CTO when the startup was early stage. Removing those few lines and installing kyber fixed the issues!
- gtirloni 3y ago
  Interesting, what was the storage type?
klysm 3y ago
I find this somewhat amusing given one of the primary use cases of eBPF is measuring performance
- dirtyhand 3y ago
  Who watches the Watchmen?
- ongy 3y ago
  There's plenty other applications. Network routing can be implemented in (e)bpf. It's even the original use case. But there's also the LSM based on ebpf, there's a user space scheduler (Google iirc), seccomp and some cgroup filters can be done in bpf... It's the Lua of the kernel at this point. Provides a lot of extension points.
  - tubs 3y ago
    sched-ext is meta, Google have something else but less open i believe. https://github.com/sched-ext/scx/blob/main/OVERVIEW.md https://github.com/sched-ext/scx/blob/main/OVERVIEW.md
    - kapilvt 3y ago
      re google's https://github.com/google/ghost-userspace https://github.com/google/ghost-userspace also one from canonical in rust on top of scx https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_rustland https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_r... https://arighi.blogspot.com/2024/02/writing-scheduler-for-linux-in-rust.html https://arighi.blogspot.com/2024/02/writing-scheduler-for-li...
maayank 3y ago
What use cases people use eBPF for these days?
- riv991 3y ago
  The eBPF website has a list of projects using it, that can give you a decent flavour of what people use it for. https://ebpf.io/applications/ https://ebpf.io/applications/
  - llotter 3y ago
    Stackstate, my current employer uses eBPF in addition to Open Telemetry for collecting observability data. https://www.stackstate.com/platform/features/ https://www.stackstate.com/platform/features/
- stefan_ 3y ago
  Tongue in cheek: lots of people have discovered they can replace Linux kernel modules with brittle eBPF code instead, which attaches itself to various parts of the kernel that are even less stable than the things modules have to deal with.
  - gtirloni 3y ago
    They are nice for quick experimentation, yes. But there are rock solid projects like Cilium using them. I think your point is that the barrier to abuse is lower?
- darkr 3y ago
  We had a recent use case to log outbound TCP connections _excluding_ internal and known addresses from our k8s infrastructure, with the log including the process name/pid, uid a bunch of other metadata. I wrote a tool that compiles to a small, statically linked binary (using CO-RE/libbpf), deployed to every node as a DaemonSet. It just works and uses minimal CPU and memory resources.
- yla92 3y ago
  Using eBPF based tools (like bcc) to debug the issues https://github.com/iovisor/bcc https://github.com/iovisor/bcc
bschuur 3y ago
Just tried this on the eBPF code I am working on. Works great! This one is going straight into the toolbox. Even though eBPF is super fast, I have found triggering complex probes many times will have performance implications, which can not easily be tracked down to the instrumenting application. This will help with that a lot.
- dirtyhand 3y ago
  I am happy to hear that you had a good first impression. At Netflix, we do some Linux scheduler instrumentation with eBPF and overhead matters. I was inspired to create the tool to enable the traditional performance work loop: get a baseline, tweak code, get another reading, rinse & repeat.
bewo001 3y ago
Nice! But I got it to freeze under higher load. Removing the load does not help.
- dirtyhand 3y ago
  bpftop author here. Would you mind creating an issue to track this? https://github.com/Netflix/bpftop/issues https://github.com/Netflix/bpftop/issues
  - bewo001 3y ago
    done (https://github.com/Netflix/bpftop/issues/17 https://github.com/Netflix/bpftop/issues/17). Seems to be some futex issue, the kind of bugs that tend to be hard to replicate.

133 points by mochomocha 3y ago

Looks great. I remember using bpftrace at work to debug a nasty performance issue, I went down the rabbit hole only to find a certain syscall was being called much too often. We managed to trace it back inside the sourcecode to a sleep(1 second), which was some sort of manual io scheduling commited by the CTO when the startup was early stage. Removing those few lines and installing kyber fixed the issues!

gtirloni 3y ago
Interesting, what was the storage type?

klysm 3y ago

I find this somewhat amusing given one of the primary use cases of eBPF is measuring performance

dirtyhand 3y ago
Who watches the Watchmen?
ongy 3y ago
There's plenty other applications. Network routing can be implemented in (e)bpf. It's even the original use case. But there's also the LSM based on ebpf, there's a user space scheduler (Google iirc), seccomp and some cgroup filters can be done in bpf... It's the Lua of the kernel at this point. Provides a lot of extension points.
- tubs 3y ago
  sched-ext is meta, Google have something else but less open i believe. https://github.com/sched-ext/scx/blob/main/OVERVIEW.md https://github.com/sched-ext/scx/blob/main/OVERVIEW.md
  - kapilvt 3y ago
    re google's https://github.com/google/ghost-userspace https://github.com/google/ghost-userspace also one from canonical in rust on top of scx https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_rustland https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_r... https://arighi.blogspot.com/2024/02/writing-scheduler-for-linux-in-rust.html https://arighi.blogspot.com/2024/02/writing-scheduler-for-li...

maayank 3y ago

What use cases people use eBPF for these days?

riv991 3y ago
The eBPF website has a list of projects using it, that can give you a decent flavour of what people use it for. https://ebpf.io/applications/ https://ebpf.io/applications/
- llotter 3y ago
  Stackstate, my current employer uses eBPF in addition to Open Telemetry for collecting observability data. https://www.stackstate.com/platform/features/ https://www.stackstate.com/platform/features/
stefan_ 3y ago
Tongue in cheek: lots of people have discovered they can replace Linux kernel modules with brittle eBPF code instead, which attaches itself to various parts of the kernel that are even less stable than the things modules have to deal with.
- gtirloni 3y ago
  They are nice for quick experimentation, yes. But there are rock solid projects like Cilium using them. I think your point is that the barrier to abuse is lower?
darkr 3y ago
We had a recent use case to log outbound TCP connections _excluding_ internal and known addresses from our k8s infrastructure, with the log including the process name/pid, uid a bunch of other metadata. I wrote a tool that compiles to a small, statically linked binary (using CO-RE/libbpf), deployed to every node as a DaemonSet. It just works and uses minimal CPU and memory resources.
yla92 3y ago
Using eBPF based tools (like bcc) to debug the issues https://github.com/iovisor/bcc https://github.com/iovisor/bcc

bschuur 3y ago

Just tried this on the eBPF code I am working on. Works great! This one is going straight into the toolbox. Even though eBPF is super fast, I have found triggering complex probes many times will have performance implications, which can not easily be tracked down to the instrumenting application. This will help with that a lot.

dirtyhand 3y ago
I am happy to hear that you had a good first impression. At Netflix, we do some Linux scheduler instrumentation with eBPF and overhead matters. I was inspired to create the tool to enable the traditional performance work loop: get a baseline, tweak code, get another reading, rinse & repeat.

bewo001 3y ago

Nice! But I got it to freeze under higher load. Removing the load does not help.

dirtyhand 3y ago
bpftop author here. Would you mind creating an issue to track this? https://github.com/Netflix/bpftop/issues https://github.com/Netflix/bpftop/issues
- bewo001 3y ago
  done (https://github.com/Netflix/bpftop/issues/17 https://github.com/Netflix/bpftop/issues/17). Seems to be some futex issue, the kind of bugs that tend to be hard to replicate.