5 ms·

Standardizing BPF

105 points by signa11 4y ago

deleted 4y ago
[deleted]
rektide 4y ago
> NVMe vendors are also looking into supporting offloading functionality to BPF for storage devices with a new framework called eXpress Resubmission Path (XRP), though this effort is currently stalled due to BPF not being standardized Neat potential application! Worth pointing out AMD is fulfilling the first most obvious listed goal in this article, offloading network processing, with Nanotube, a eBPF XDP -> fpga suite, based around LLVM & Vites-HLS. https://www.phoronix.com/news/AMD-Xilinx-Nanotube-Compiler https://www.phoronix.com/news/AMD-Xilinx-Nanotube-Compiler
- benlwalker 4y ago
  Associating the desire of NVMe vendors to allow users to ship down eBPF programs to run on the device and XRP is a major mistake in the article. XRP has nothing to do with what the NVMe vendors want to do, and XRP is a pure kernel solution that doesn't need any participation from NVMe vendors. I think it's unclear whether XRP even has real value - it certainly may, but I believe the benchmarking in the paper was deeply flawed[1]. 1. https://github.com/xrp-project/BPF-KV/issues/3 https://github.com/xrp-project/BPF-KV/issues/3
  - manifault 4y ago
    I'm not closely tied to what the NVMe vendors want, so you could be right, but I very much doubt you are given that Christoph didn't flag this when he reviewed the article. Edit: And to be clear, from my understanding of XRP, the device itself calls back into a BPF function in the NVMe driver. That requires some notion of standardization. It's not exactly offloading directly to the storage device, but the storage device still relies on some standardized behavior in the BPF program, such as divide by zero, what instructions are supported in the ISA, etc.
    - benlwalker 4y ago
      I am very closely tied to what the NVMe vendors want, having written the first internal draft of the proposal to the standards body (since that draft many smart people have taken the pen and done a lot of great work). XRP is unrelated to offloading eBPF to NVMe devices.
      - manifault 4y ago
        Sure, whatever. "Offloading" was perhaps a poor choice of wording, but it is related to the standardization efforts. The NVMe vendors don't want to be calling out to BPF programs in the driver if the runtime semantics are not standardized.
        benlwalker 4y ago
        XRP is a regular BPF hook in Linux and requires no additional standardization. The device never "calls out to BPF programs in the driver" - it generates a normal completion interrupt and Linux runs a BPF hook in the completion path. This is no different than other kernel BPF hooks elsewhere and doesn't provide any additional reason or need to standardize BPF. The article misstated that XRP was a framework used for offloading BPF programs to NVMe devices. That's not correct, and XRP is not one of the emerging use cases for BPF that is driving standardization.
        deleted 4y ago
        [deleted]
        deleted 4y ago
        [deleted]
        rektide 4y ago
        XRP initially looks to be focused on moving work upstream into the kernel not devices. But absolutely, once you start shipping ebpf into the kernel, people do quickly start asking, "how can we hardware accelerate that?". Having standards would be helpful.
    - 4y ago
bheadmaster 4y ago
> BPF programs are written in C, and compiled into BPF bytecode. Like other bytecode instruction sets, BPF programs are platform-independent and just-in-time (JIT) compiled. That sounds similar to what Inferno was doing. Though I'm unaware whether Inferno did it in kernel space, but PhantomOS definitely did - their argument was that, since you can't run native code directly, only through the bytecode, memory safety can be achieved by simply not giving a program any reference to a memory address it's not supposed to access. Apparently, this approach achieves memory protection without using an MMU. Would be cool to see if Linux eventually goes down that path.
- pjmlp 4y ago
  Inferno uses C (Plan 9 C dialect) for the kernel and Limbo for the complete userspace, with exception of some C libraries like Tk for the GUI. DisVM runs in userspace. https://github.com/inferno-os https://github.com/inferno-os
- amelius 4y ago
  Java did this in the 90s. Not sure why we need yet another bytecode format.
  - bheadmaster 4y ago
    Bytecode format itself is not interesting (PhantomOS runs Java bytecode, Inferno was a competitor to Java in the 90s), but the way it is executed. Java runs bytecode in a VM as a userspace process, eBPF runs it in kernel space. That in itself is interesting.
    - amelius 4y ago
      Does running in kernel space justify an entirely new bytecode?
      - bheadmaster 4y ago
        Justify? I don't know. Require? I also don't know, but intuition tells me that it does, at least in the case of Linux.
akhayam 4y ago
This has been a long time coming… eBPF is the de facto tracing, networking and security stack on basically every platform that mankind cares about right now. Great to see the standardization kicking off now. eBPF Foundation is the right body for this standard, as that’s where experts (from users and vendors) come together. I have had the privilege of working with the first engineers who designed and upstreamed eBPF. I have no doubt these geniuses can put a solid standard together.
Owen69 4y ago
[dead]
112233 4y ago
What is the licensing situation with eBPF now? Last time I looked (a long long time ago), BPF was ok, because of the "B", but eBPF was linux-only in such a way that loading eBPF program into kernel made your userspace program a derived work of linux kernel and forced GPL-2 on the userspace. Has this changed? Is there a good writeup about eBPF wrt licensing? edit: quoting current manpage on struct bpf_attr member "license": license is a license string, which must be GPL compatible to call helper functions marked gpl_only. (The licensing rules are the same as for kernel modules, so that also dual licenses, such as "Dual BSD/GPL", may be used.) So what is gpl_only now? Used to be the whole of eBPF.
- jmillikin 4y ago
  > eBPF was linux-only in such a way that loading eBPF program into > kernel made your userspace program a derived work of linux kernel > and forced GPL-2 on the userspace. Copyright doesn't work like that: 1. If some particular eBPF program is a derived work of the kernel, then whether it's loaded or not doesn't matter. Distributing that program (such as embedded into an ELF binary) would still require being accompanied by its source code (/ a written offer, etc). 2. If some eBPF program is not a derived work, then loading it into the kernel doesn't cause the userspace program to become a derived work. Programs distributed as interpreted bytecode are not automatically derived works of their interpreter. 3. Even if a program depends on GPL'd code in such a way as to be a derived work when compiled, its own license only needs to be compatible with the GPL, it doesn't need to be GPL'd itself. For example, it's legal to write and distribute a Linux kernel module licensed under various permissive licenses (MIT, 3-clause BSD, etc).
  - 112233 4y ago
    Thanks, but it did't help ... :) Suppose a person is writing software that works on linux as part of dayjob/hobby/large project/tutorial or something. And, wanting to use some eBPF produced by clang, that person comes to this "license" member in the manpage. What to do? Call lawyer? Call police? Call reddit? [edit, a ramble]: next logical step after putting license in API is putting Code of Conduct in api. Half expecting vegan-only syscalls any time soon.
rwmj 4y ago
Is there a full userspace version of Linux eBPF now? Last time I looked the userspace implementations were all very partial and in particular didn't include the verifier.
- manifault 4y ago
  Not sure if there's a full user space runtime for BPF progs, but there are lots of user space tools which are useful for working with BPF progs. For example, you can use veristat (https://github.com/libbpf/veristat https://github.com/libbpf/veristat) to easily see whether a BPF prog would be verified, and collect statistics, etc.
- rendaw 4y ago
  I don't want an independent userspace implementation, I want the verifier the kernel uses. I'm was looking into this and I'm not sure why it can't be pulled out. I guess depending on what you're plugging the eBPF into the environment (functions, etc) may be tightly coupled to the kernel, but surely the loop analysis and stuff could? It basically forces you to program on a live system. No CI, no static analysis, no editor help, etc.
- rektide 4y ago
  UBPF https://github.com/iovisor/ubpf/ https://github.com/iovisor/ubpf/ and rbpf https://github.com/qmonnet/rbpf https://github.com/qmonnet/rbpf are userspace bpf interpreters. Via https://github.com/zoidbergwill/awesome-ebpf#user-space-ebpf https://github.com/zoidbergwill/awesome-ebpf#user-space-ebpf
teleforce 4y ago
It's about time that BPF is being standardized. It increasingly being used not only in the conventional field of network and security accelerations, now it's also being used for other uses cases for example 5G Open RAN [1]. For more comprehensive review of latest eBPF based applications please check it here [2]. [1] Taking 5G RAN Analytics and Control to a New Level: https://www.microsoft.com/en-us/research/publication/taking-5g-ran-analytics-and-control-to-a-new-level/ https://www.microsoft.com/en-us/research/publication/taking-... [2] Extended Berkeley Packet Filter: An Application Perspective: https://ieeexplore.ieee.org/abstract/document/9968265 https://ieeexplore.ieee.org/abstract/document/9968265

105 points by signa11 4y ago

deleted 4y ago

[deleted]

> NVMe vendors are also looking into supporting offloading functionality to BPF for storage devices with a new framework called eXpress Resubmission Path (XRP), though this effort is currently stalled due to BPF not being standardized Neat potential application! Worth pointing out AMD is fulfilling the first most obvious listed goal in this article, offloading network processing, with Nanotube, a eBPF XDP -> fpga suite, based around LLVM & Vites-HLS. https://www.phoronix.com/news/AMD-Xilinx-Nanotube-Compiler https://www.phoronix.com/news/AMD-Xilinx-Nanotube-Compiler

benlwalker 4y ago
Associating the desire of NVMe vendors to allow users to ship down eBPF programs to run on the device and XRP is a major mistake in the article. XRP has nothing to do with what the NVMe vendors want to do, and XRP is a pure kernel solution that doesn't need any participation from NVMe vendors. I think it's unclear whether XRP even has real value - it certainly may, but I believe the benchmarking in the paper was deeply flawed[1]. 1. https://github.com/xrp-project/BPF-KV/issues/3 https://github.com/xrp-project/BPF-KV/issues/3
- manifault 4y ago
  I'm not closely tied to what the NVMe vendors want, so you could be right, but I very much doubt you are given that Christoph didn't flag this when he reviewed the article. Edit: And to be clear, from my understanding of XRP, the device itself calls back into a BPF function in the NVMe driver. That requires some notion of standardization. It's not exactly offloading directly to the storage device, but the storage device still relies on some standardized behavior in the BPF program, such as divide by zero, what instructions are supported in the ISA, etc.
  - benlwalker 4y ago
    I am very closely tied to what the NVMe vendors want, having written the first internal draft of the proposal to the standards body (since that draft many smart people have taken the pen and done a lot of great work). XRP is unrelated to offloading eBPF to NVMe devices.
    - manifault 4y ago
      Sure, whatever. "Offloading" was perhaps a poor choice of wording, but it is related to the standardization efforts. The NVMe vendors don't want to be calling out to BPF programs in the driver if the runtime semantics are not standardized.
      - benlwalker 4y ago
        XRP is a regular BPF hook in Linux and requires no additional standardization. The device never "calls out to BPF programs in the driver" - it generates a normal completion interrupt and Linux runs a BPF hook in the completion path. This is no different than other kernel BPF hooks elsewhere and doesn't provide any additional reason or need to standardize BPF. The article misstated that XRP was a framework used for offloading BPF programs to NVMe devices. That's not correct, and XRP is not one of the emerging use cases for BPF that is driving standardization.
        deleted 4y ago
        [deleted]
        deleted 4y ago
        [deleted]
      - rektide 4y ago
        XRP initially looks to be focused on moving work upstream into the kernel not devices. But absolutely, once you start shipping ebpf into the kernel, people do quickly start asking, "how can we hardware accelerate that?". Having standards would be helpful.
  - 4y ago

bheadmaster 4y ago

> BPF programs are written in C, and compiled into BPF bytecode. Like other bytecode instruction sets, BPF programs are platform-independent and just-in-time (JIT) compiled. That sounds similar to what Inferno was doing. Though I'm unaware whether Inferno did it in kernel space, but PhantomOS definitely did - their argument was that, since you can't run native code directly, only through the bytecode, memory safety can be achieved by simply not giving a program any reference to a memory address it's not supposed to access. Apparently, this approach achieves memory protection without using an MMU. Would be cool to see if Linux eventually goes down that path.

pjmlp 4y ago
Inferno uses C (Plan 9 C dialect) for the kernel and Limbo for the complete userspace, with exception of some C libraries like Tk for the GUI. DisVM runs in userspace. https://github.com/inferno-os https://github.com/inferno-os
amelius 4y ago
Java did this in the 90s. Not sure why we need yet another bytecode format.
- bheadmaster 4y ago
  Bytecode format itself is not interesting (PhantomOS runs Java bytecode, Inferno was a competitor to Java in the 90s), but the way it is executed. Java runs bytecode in a VM as a userspace process, eBPF runs it in kernel space. That in itself is interesting.
  - amelius 4y ago
    Does running in kernel space justify an entirely new bytecode?
    - bheadmaster 4y ago
      Justify? I don't know. Require? I also don't know, but intuition tells me that it does, at least in the case of Linux.

akhayam 4y ago

This has been a long time coming… eBPF is the de facto tracing, networking and security stack on basically every platform that mankind cares about right now. Great to see the standardization kicking off now. eBPF Foundation is the right body for this standard, as that’s where experts (from users and vendors) come together. I have had the privilege of working with the first engineers who designed and upstreamed eBPF. I have no doubt these geniuses can put a solid standard together.

Owen69 4y ago

[dead]

112233 4y ago

What is the licensing situation with eBPF now? Last time I looked (a long long time ago), BPF was ok, because of the "B", but eBPF was linux-only in such a way that loading eBPF program into kernel made your userspace program a derived work of linux kernel and forced GPL-2 on the userspace. Has this changed? Is there a good writeup about eBPF wrt licensing? edit: quoting current manpage on struct bpf_attr member "license": license is a license string, which must be GPL compatible to call helper functions marked gpl_only. (The licensing rules are the same as for kernel modules, so that also dual licenses, such as "Dual BSD/GPL", may be used.) So what is gpl_only now? Used to be the whole of eBPF.

jmillikin 4y ago
> eBPF was linux-only in such a way that loading eBPF program into > kernel made your userspace program a derived work of linux kernel > and forced GPL-2 on the userspace. Copyright doesn't work like that: 1. If some particular eBPF program is a derived work of the kernel, then whether it's loaded or not doesn't matter. Distributing that program (such as embedded into an ELF binary) would still require being accompanied by its source code (/ a written offer, etc). 2. If some eBPF program is not a derived work, then loading it into the kernel doesn't cause the userspace program to become a derived work. Programs distributed as interpreted bytecode are not automatically derived works of their interpreter. 3. Even if a program depends on GPL'd code in such a way as to be a derived work when compiled, its own license only needs to be compatible with the GPL, it doesn't need to be GPL'd itself. For example, it's legal to write and distribute a Linux kernel module licensed under various permissive licenses (MIT, 3-clause BSD, etc).
- 112233 4y ago
  Thanks, but it did't help ... :) Suppose a person is writing software that works on linux as part of dayjob/hobby/large project/tutorial or something. And, wanting to use some eBPF produced by clang, that person comes to this "license" member in the manpage. What to do? Call lawyer? Call police? Call reddit? [edit, a ramble]: next logical step after putting license in API is putting Code of Conduct in api. Half expecting vegan-only syscalls any time soon.

rwmj 4y ago

Is there a full userspace version of Linux eBPF now? Last time I looked the userspace implementations were all very partial and in particular didn't include the verifier.

manifault 4y ago
Not sure if there's a full user space runtime for BPF progs, but there are lots of user space tools which are useful for working with BPF progs. For example, you can use veristat (https://github.com/libbpf/veristat https://github.com/libbpf/veristat) to easily see whether a BPF prog would be verified, and collect statistics, etc.
rendaw 4y ago
I don't want an independent userspace implementation, I want the verifier the kernel uses. I'm was looking into this and I'm not sure why it can't be pulled out. I guess depending on what you're plugging the eBPF into the environment (functions, etc) may be tightly coupled to the kernel, but surely the loop analysis and stuff could? It basically forces you to program on a live system. No CI, no static analysis, no editor help, etc.
rektide 4y ago
UBPF https://github.com/iovisor/ubpf/ https://github.com/iovisor/ubpf/ and rbpf https://github.com/qmonnet/rbpf https://github.com/qmonnet/rbpf are userspace bpf interpreters. Via https://github.com/zoidbergwill/awesome-ebpf#user-space-ebpf https://github.com/zoidbergwill/awesome-ebpf#user-space-ebpf

teleforce 4y ago

It's about time that BPF is being standardized. It increasingly being used not only in the conventional field of network and security accelerations, now it's also being used for other uses cases for example 5G Open RAN [1]. For more comprehensive review of latest eBPF based applications please check it here [2]. [1] Taking 5G RAN Analytics and Control to a New Level: https://www.microsoft.com/en-us/research/publication/taking-5g-ran-analytics-and-control-to-a-new-level/ https://www.microsoft.com/en-us/research/publication/taking-... [2] Extended Berkeley Packet Filter: An Application Perspective: https://ieeexplore.ieee.org/abstract/document/9968265 https://ieeexplore.ieee.org/abstract/document/9968265