6 ms·

I have a hard time understanding what you would use it for. I could understand a use-case, but I fail to understand why it would be that much useful. I have a

by jokoon 7y ago

I have a hard time understanding what you would use it for. I could understand a use-case, but I fail to understand why it would be that much useful. I have a sense it allows much better performance for horizontal scaling, but I'm not sure...

rusk 7y ago
Real-time, low latency, network-based applications. At the pace of network events, CPU is still very fast by perhaps at least order of magnitude. However, latency introduced by system calls is significant. This allows you to run certain classes of application in kernel space with these overheads largely mitigated. Principally it's monitoring and "observability" applications, but apparently it's much more flexible now than it has been historically ...
- z3t4 7y ago
  Wouldn't virtualization kill the perf benefit, or is this supposed to run "on the metal"?
  - rusk 7y ago
    I think that depends ... I'm not an expert on virtualisation but I've seen some cases where VMs get full bare-metal access to system functions, albeit supervised. I think this is the purpose of virtualisation extensions on modern architectures (e.g. VT-x) - this is how it's possible to have a 64-bit OS run virtualised on a 64-bit processor. EDIT - but even without this, the comparison with user-space holds.
  - gjulianm 7y ago
    I don't know about other devices, but network cards have pretty good support for virtualization. A lot of them have virtual devices that the card generates and are added to the system as if they were different PCI devices. Then, each VM gets one of those virtual devices and they access it directly as a regular PCI device, with no virtualization layer whatsoever.
  - nickysielicki 7y ago
    It’s a VM in the JIT sense, not a VM in the VMWare sense.
  - zymhan 7y ago
    Virtualized OSes can still access network hardware directly via SR-IOV
- ignoramous 7y ago
  > Principally it's monitoring and "observability" applications, but apparently it's much more flexible now than it has been historically ... Yep: Here's a bredangregg presentation on the topic: https://www.youtube-nocookie.com/embed/7pmXdG8-7WU https://www.youtube-nocookie.com/embed/7pmXdG8-7WU
  - justusthane 7y ago
    That's exactly what the linked article is.
- jancsika 7y ago
  To get the realtime benefits I'd have to run everything that requires realtime scheduling as an eBPF program, right? I mean, suppose I want to make an eBPF program to pipe audio input into my very fancy DSP algo. Awesome. But now I've limited the remainder of my signal chain to remain in eBPF land, haven't I? The moment I throw the signal to Supercollider, Pd, some Jack client, etc., I'm back to being a mere userland mortal and taking the performance hit that entails. (Unless I sample that data coming into userland at a lower rate, but that severely limits the use cases for such a design.) Judging by some of the screenshots I've seen Linux audio users seem to want to mix their samples through every single audio program they can get their hands on. So at least in this domain I have a hard time seeing how eBPF could practically help realtime scheduling. Edit: clarification
  - rusk 7y ago
    Hm, good point. But couldn't you say the same thing about CUDA? There is an overhead involved in shunting data to and from the cores ... but the nature of the work being done, and the massive parallelism, mean the overall end-to-end volume of throughput increases dramatically. I don't know enough about audio processing to answer your specific question ... but processing network traffic for instance, you have the opportunity to "do things" with traffic. For instance partially decode PDUs to see if they interest you, and then pass them on to user-land for more in-depth processing if so ... EDIT I think you're thinking in terms exclusively of `mapping` operations, but this kind of thing can be great for `reducing` or filtering.
    - PaulDavisThe1st 7y ago
      There's bandwidth and there's latency. CUDA has the bandwidth, but not the latency, for generalizable low latency audio processing.
      - jancsika 7y ago
        It could still be considered low latency by adding the constraint that all use cases are in the category of call-and-response musical forms. :)
- dbancajas 7y ago
  Could you allocate a memory dynamically inside your BPF program?
  - base0x10 7y ago
    no* BPF programs are written in a very small subset of C (or rust or C++), which does not include dynamic memory allocation or unbounded loops. * you can use maps which are persistent key-value stores provided by the eBPF runtime, and you can call out to approved C functions provided by the kernel, but those won't allocate memory for you.
- the8472 7y ago
  > However, latency introduced by system calls is significant. io_uring was introduced to attack this too and essentially is an async, queue-based, batched syscall interface eliminating a lot of that latency overhead. With polling mode you only have to do a syscall when both the application and the kernel are out of work.
- waste_monk 7y ago
  > Real-time, low latency, network-based applications. I thought these kinds of applications generally (try to) avoid the kernel entirely e.g. high frequency trading algorithms running on FPGAs. Maybe using it to get data on/off of the FPGA for charting / updates / etc.
arminiusreturns 7y ago
The main use case for me as a linux admin is two fold. One, to augment iptables/nftables for increased speed and observability gains in them. It's possible to do BPF only netfilter (some firewall/IDS tools are likely to use it heavily) but I think it works better just helping the other tools, and you can lookup some benchmarks that show it. Two, as a better tool for general observability and problem tracing. For example, I recently listened to a Usenix (LISA19) talk by Brendan Gregg (author of this blog) about linux systems perf at Netflix where he talks about how much strace can impact performance and he posits the future replacement for it will be 'perf trace' which uses ring buffer and BPF. [1] 1[] https://youtu.be/fhBHvsi0Ql0?t=1300 https://youtu.be/fhBHvsi0Ql0?t=1300
- zippie 7y ago
  Not sure if you were hinting at it already, just in case, you may be interested in the ongoing work with bpfilter [0] which uses ebpf underneath existing xfilter rule interfaces. [0] https://lwn.net/Articles/747551/ https://lwn.net/Articles/747551/
  - arminiusreturns 7y ago
    Yeah I should have been more specific, you are right.
gnufx 7y ago
Look at all the examples Gregg provides.
jakeogh 7y ago
One could replace JS with something that halts.
convolvatron 7y ago
imagine how this might be useful in a distributed context. rather than specify a limited set of api entry points (like segment gates) that you try to make safe....you can provide general security and resource guarantees and let the consumer do what they need. so that's cute, but imagine the round trip reduction (nfsv4 does this to a very limited extent) and increased generality. what if your btree traversal could be safely run on the storage server.
brendangregg 7y ago
Cilium's container networking and security product [0]; Facebook's Katran [1]; Netflix's flowsrus (not public yet, but see my tcplife tool in BCC[2]). That's just the beginning. It's not just for performance, it's also security, as while BPF programs run in kernel-mode they have a limited and secured API for interacting with the system (BPF helpers). Since extended BPF is in the Linux kernel (and will be in other kernels in the future), everyone is getting it, and we'll see more use cases over time. In some ways it's like the birth of JavaScript for the browser, and all the new applications it made possible. But it goes further than that: we could still analyze and debug JavaScript applications using traditional tools. But BPF programs are neither process-space or kernel routines, and are outside the view of everything. No visibility in ps(1), top(1), or lsmod(8). We're having to create new tools to even see what's running on the CPUs. Every performance monitoring product that shows a process table with CPU consumption will now need a BPF program table as well. [0] https://cilium.io/ https://cilium.io/ [1] https://github.com/facebookincubator/katran https://github.com/facebookincubator/katran [2] http://www.brendangregg.com/blog/2016-11-30/linux-bcc-tcplife.html http://www.brendangregg.com/blog/2016-11-30/linux-bcc-tcplif...

by jokoon 7y ago

rusk 7y ago

Real-time, low latency, network-based applications. At the pace of network events, CPU is still very fast by perhaps at least order of magnitude. However, latency introduced by system calls is significant. This allows you to run certain classes of application in kernel space with these overheads largely mitigated. Principally it's monitoring and "observability" applications, but apparently it's much more flexible now than it has been historically ...

z3t4 7y ago
Wouldn't virtualization kill the perf benefit, or is this supposed to run "on the metal"?
- rusk 7y ago
  I think that depends ... I'm not an expert on virtualisation but I've seen some cases where VMs get full bare-metal access to system functions, albeit supervised. I think this is the purpose of virtualisation extensions on modern architectures (e.g. VT-x) - this is how it's possible to have a 64-bit OS run virtualised on a 64-bit processor. EDIT - but even without this, the comparison with user-space holds.
- gjulianm 7y ago
  I don't know about other devices, but network cards have pretty good support for virtualization. A lot of them have virtual devices that the card generates and are added to the system as if they were different PCI devices. Then, each VM gets one of those virtual devices and they access it directly as a regular PCI device, with no virtualization layer whatsoever.
- nickysielicki 7y ago
  It’s a VM in the JIT sense, not a VM in the VMWare sense.
- zymhan 7y ago
  Virtualized OSes can still access network hardware directly via SR-IOV
ignoramous 7y ago
> Principally it's monitoring and "observability" applications, but apparently it's much more flexible now than it has been historically ... Yep: Here's a bredangregg presentation on the topic: https://www.youtube-nocookie.com/embed/7pmXdG8-7WU https://www.youtube-nocookie.com/embed/7pmXdG8-7WU
- justusthane 7y ago
  That's exactly what the linked article is.
jancsika 7y ago
To get the realtime benefits I'd have to run everything that requires realtime scheduling as an eBPF program, right? I mean, suppose I want to make an eBPF program to pipe audio input into my very fancy DSP algo. Awesome. But now I've limited the remainder of my signal chain to remain in eBPF land, haven't I? The moment I throw the signal to Supercollider, Pd, some Jack client, etc., I'm back to being a mere userland mortal and taking the performance hit that entails. (Unless I sample that data coming into userland at a lower rate, but that severely limits the use cases for such a design.) Judging by some of the screenshots I've seen Linux audio users seem to want to mix their samples through every single audio program they can get their hands on. So at least in this domain I have a hard time seeing how eBPF could practically help realtime scheduling. Edit: clarification
- rusk 7y ago
  Hm, good point. But couldn't you say the same thing about CUDA? There is an overhead involved in shunting data to and from the cores ... but the nature of the work being done, and the massive parallelism, mean the overall end-to-end volume of throughput increases dramatically. I don't know enough about audio processing to answer your specific question ... but processing network traffic for instance, you have the opportunity to "do things" with traffic. For instance partially decode PDUs to see if they interest you, and then pass them on to user-land for more in-depth processing if so ... EDIT I think you're thinking in terms exclusively of `mapping` operations, but this kind of thing can be great for `reducing` or filtering.
  - PaulDavisThe1st 7y ago
    There's bandwidth and there's latency. CUDA has the bandwidth, but not the latency, for generalizable low latency audio processing.
    - jancsika 7y ago
      It could still be considered low latency by adding the constraint that all use cases are in the category of call-and-response musical forms. :)
dbancajas 7y ago
Could you allocate a memory dynamically inside your BPF program?
- base0x10 7y ago
  no* BPF programs are written in a very small subset of C (or rust or C++), which does not include dynamic memory allocation or unbounded loops. * you can use maps which are persistent key-value stores provided by the eBPF runtime, and you can call out to approved C functions provided by the kernel, but those won't allocate memory for you.
the8472 7y ago
> However, latency introduced by system calls is significant. io_uring was introduced to attack this too and essentially is an async, queue-based, batched syscall interface eliminating a lot of that latency overhead. With polling mode you only have to do a syscall when both the application and the kernel are out of work.
waste_monk 7y ago
> Real-time, low latency, network-based applications. I thought these kinds of applications generally (try to) avoid the kernel entirely e.g. high frequency trading algorithms running on FPGAs. Maybe using it to get data on/off of the FPGA for charting / updates / etc.

arminiusreturns 7y ago

The main use case for me as a linux admin is two fold. One, to augment iptables/nftables for increased speed and observability gains in them. It's possible to do BPF only netfilter (some firewall/IDS tools are likely to use it heavily) but I think it works better just helping the other tools, and you can lookup some benchmarks that show it. Two, as a better tool for general observability and problem tracing. For example, I recently listened to a Usenix (LISA19) talk by Brendan Gregg (author of this blog) about linux systems perf at Netflix where he talks about how much strace can impact performance and he posits the future replacement for it will be 'perf trace' which uses ring buffer and BPF. [1] 1[] https://youtu.be/fhBHvsi0Ql0?t=1300 https://youtu.be/fhBHvsi0Ql0?t=1300

zippie 7y ago
Not sure if you were hinting at it already, just in case, you may be interested in the ongoing work with bpfilter [0] which uses ebpf underneath existing xfilter rule interfaces. [0] https://lwn.net/Articles/747551/ https://lwn.net/Articles/747551/
- arminiusreturns 7y ago
  Yeah I should have been more specific, you are right.

gnufx 7y ago

Look at all the examples Gregg provides.

jakeogh 7y ago

One could replace JS with something that halts.

convolvatron 7y ago

imagine how this might be useful in a distributed context. rather than specify a limited set of api entry points (like segment gates) that you try to make safe....you can provide general security and resource guarantees and let the consumer do what they need. so that's cute, but imagine the round trip reduction (nfsv4 does this to a very limited extent) and increased generality. what if your btree traversal could be safely run on the storage server.

brendangregg 7y ago

Cilium's container networking and security product [0]; Facebook's Katran [1]; Netflix's flowsrus (not public yet, but see my tcplife tool in BCC[2]). That's just the beginning. It's not just for performance, it's also security, as while BPF programs run in kernel-mode they have a limited and secured API for interacting with the system (BPF helpers). Since extended BPF is in the Linux kernel (and will be in other kernels in the future), everyone is getting it, and we'll see more use cases over time. In some ways it's like the birth of JavaScript for the browser, and all the new applications it made possible. But it goes further than that: we could still analyze and debug JavaScript applications using traditional tools. But BPF programs are neither process-space or kernel routines, and are outside the view of everything. No visibility in ps(1), top(1), or lsmod(8). We're having to create new tools to even see what's running on the CPUs. Every performance monitoring product that shows a process table with CPU consumption will now need a BPF program table as well. [0] https://cilium.io/ https://cilium.io/ [1] https://github.com/facebookincubator/katran https://github.com/facebookincubator/katran [2] http://www.brendangregg.com/blog/2016-11-30/linux-bcc-tcplife.html http://www.brendangregg.com/blog/2016-11-30/linux-bcc-tcplif...