5 ms·

PythonBPF – Writing eBPF Programs in Pure Python

158 points by JNRowe 1y ago

atoav 1y ago
Looks cool, I like the use of decorators as a means to use essentially turn python into some sort of DSL. One nitpick: Please include a paragraph/section/infobox explaining what eBPF is and what problems should be solved using it. I am a huge fan of making our tech world more accessible and as such we should think to some degree about people who don't know every acronym.
- varunrmallya 1y ago
  To be honest, this was really a hackathon project. The code quality is very very bad right now. We will be continuing to work on this to make it much better and we'll be adding documentation as we go as well. Thanks for taking a look :)
njharman 1y ago
Putting tldr; at the bottom defeats purpose of tldr. Guessing this is BPF https://en.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter But, reader shouldn't have to guess. That is the link that should be in your Introduction. Just after tldr;
- indigo945 1y ago
  Not the original BPF, but its successor in the Linux kernel called eBPF [1]. eBPF's virtual machine has additional registers, and crucially, eBPF programs can make some syscalls, which BPF programs can't. [1]: https://lwn.net/Articles/740157/ https://lwn.net/Articles/740157/
alexgartrell 1y ago
I did something similar a long time ago https://github.com/facebookresearch/py2bpf https://github.com/facebookresearch/py2bpf It was definitely a toy, I transliterated from python bytecode (a stack based vm) into bpf. I also wrote the full code gen stack myself (bpf was simpler back then) But using llvm and not marrying things to cpython implementation makes this approach way better
- varunrmallya 1y ago
  Thank you! Ours is a toy for now as well, but I think the idea is pretty good, so we'll continue to work on it. (This was actually a hackathon project, so the code is pretty messy and not something I am proud of)
indigo945 1y ago
The "How it works under the hood" section raises more question than it answers. What is the difference between step 3 and step 4? As described, step 3 goes from LLVM IR to BPF (via llc), and step 4 - goes from LLVM IR to eBPF bytecode? That's nonsensical.
- varunrmallya 1y ago
  I'm the co-author.The code is in a very very bad state right now, but the architecture is pretty ok to explain. In step 3, we translate from the Python frontend to the LLVM IR. In step 4 we compile it down to an object file using the LLVM backend `llc`. This object file gets loaded into the kernel and it is what actually contains the eBPF bytecode.
  - indigo945 1y ago
    You may want to edit the blog post, then, because that's not what it says.
the_duke 1y ago
So this is a "inline" Python to eBPF transpiler/compiler. Which is cool! But the description could be a bit clearer.
farnulfo 1y ago
For java, Johannes Bechberger has made a lot of articles about writing eBPF in java : https://mostlynerdless.de/blog/2023/12/31/hello-ebpf-developing-ebpf-apps-in-java-1/ https://mostlynerdless.de/blog/2023/12/31/hello-ebpf-develop... https://mostlynerdless.de/blog/category/computer-science/ebpf/ https://mostlynerdless.de/blog/category/computer-science/ebp...
- xbar 1y ago
  I missed these last year. Finding them now is truly very useful in my work.
pimterry 1y ago
Does anybody know if something similar exists for Node.js? I'd love to be able to integrate BPF into some of my Node projects with the same kind of approach.
- lloydatkinson 1y ago
  Please no...
  - deleted 1y ago
    [deleted]
- ranger_danger 1y ago
  Dear god... why
drivenextfunc 1y ago
Writing C for eBPF is cumbersome and you'd like to avoid it. Okay, that's reasonable. But I don't think it would be a good idea to write a compiler that emits eBPF binary from (a tiny subset of) Python. Why not just write code in pseudo-Python (or whatever language you're comfortable with) and have it translated by an LLM, and paste it in the source code? That would be much better because there would be fewer layers and a significant reduction in runtime cost.
- tecleandor 1y ago
  I don't understand... So, instead of having a defined and documented subset of Python that compiles to eBPF in a deterministic way... use an undefined pseudo language and let the LLM have fun with it without understanding if the result C is correct? What would be the advantage?
  - drivenextfunc 1y ago
    The behavior of CPython and a few other implementations of Python (such as PyPy) is well documented and well understood. The semantics of the tiny subset of Python that this Python-to-eBPF compiler understands is not. For example, inferring from the fact that it statically compiles Python-ish AST to LLVM IR, you can have a rough idea that dynamic elements of Python semantics are unlikely to be compiled, but you cannot know exactly which elements without carefully reading the documentation or source code of the compiler. You can guess globals() or locals() won't work, maybe .__dict__ won't as well, but how about type() or isinstance()? You don't know without digging into the documentation (which may be lacking), because the subset of Python this compiler understands is rather arbitrary. And also, having an LLM translate Python-ish pseudo code into C does not imply that you cannot examine it before putting it into a program. You can manually review it and make modifications as you want. It just reduces time spent compared with writing C code by hand.
    - tecleandor 1y ago
      But then we have to write the pseudocode anyway (that cannot be corrected by my IDE, so I don't know if I have pseudomistakes [sorry for the pun]), the LLM 'transpile' (that's not understood at all), and you have to review the C code anyway, so you have to know eBPF code really well. Would that represent a time advantage?
grantseltzer 1y ago
bcc hasn't been relevant for years.
- _bobm 1y ago
  I have been a bit out of the loop. what is relevant these days for writing ebpf code? what about ebpf code in python?
  - nickysielicki 1y ago
    bpftrace is nicer to work with and can replace bcc in most cases for debugging.
  - grantseltzer 1y ago
    Writing it in C, compiling with clang, and loading with either C(libbpf), Go (cilium/ebpf), or Rust (Aya). You can also write bpf in rust with Aya but i'm not sure how feature complete it is. For very simple use cases you can just bpftrace.
bieganski 1y ago
that's really cool. to gain traction i would start with reimplementing all the tools from https://github.com/iovisor/bcc/tree/master/libbpf-tools https://github.com/iovisor/bcc/tree/master/libbpf-tools in PythonBPF.
- varunrmallya 1y ago
  This is actually what we plan to do too!
robertlagrant 1y ago
Step 1: import numpy
setheron 1y ago
I had similar idea ! Loved seeing it here. We thought about doing it with ChocoPy to make the types more consistent.

158 points by JNRowe 1y ago

Looks cool, I like the use of decorators as a means to use essentially turn python into some sort of DSL. One nitpick: Please include a paragraph/section/infobox explaining what eBPF is and what problems should be solved using it. I am a huge fan of making our tech world more accessible and as such we should think to some degree about people who don't know every acronym.

varunrmallya 1y ago
To be honest, this was really a hackathon project. The code quality is very very bad right now. We will be continuing to work on this to make it much better and we'll be adding documentation as we go as well. Thanks for taking a look :)

njharman 1y ago

Putting tldr; at the bottom defeats purpose of tldr. Guessing this is BPF https://en.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter But, reader shouldn't have to guess. That is the link that should be in your Introduction. Just after tldr;

indigo945 1y ago
Not the original BPF, but its successor in the Linux kernel called eBPF [1]. eBPF's virtual machine has additional registers, and crucially, eBPF programs can make some syscalls, which BPF programs can't. [1]: https://lwn.net/Articles/740157/ https://lwn.net/Articles/740157/

alexgartrell 1y ago

I did something similar a long time ago https://github.com/facebookresearch/py2bpf https://github.com/facebookresearch/py2bpf It was definitely a toy, I transliterated from python bytecode (a stack based vm) into bpf. I also wrote the full code gen stack myself (bpf was simpler back then) But using llvm and not marrying things to cpython implementation makes this approach way better

varunrmallya 1y ago
Thank you! Ours is a toy for now as well, but I think the idea is pretty good, so we'll continue to work on it. (This was actually a hackathon project, so the code is pretty messy and not something I am proud of)

indigo945 1y ago

The "How it works under the hood" section raises more question than it answers. What is the difference between step 3 and step 4? As described, step 3 goes from LLVM IR to BPF (via llc), and step 4 - goes from LLVM IR to eBPF bytecode? That's nonsensical.

varunrmallya 1y ago
I'm the co-author.The code is in a very very bad state right now, but the architecture is pretty ok to explain. In step 3, we translate from the Python frontend to the LLVM IR. In step 4 we compile it down to an object file using the LLVM backend `llc`. This object file gets loaded into the kernel and it is what actually contains the eBPF bytecode.
- indigo945 1y ago
  You may want to edit the blog post, then, because that's not what it says.

the_duke 1y ago

So this is a "inline" Python to eBPF transpiler/compiler. Which is cool! But the description could be a bit clearer.

farnulfo 1y ago

For java, Johannes Bechberger has made a lot of articles about writing eBPF in java : https://mostlynerdless.de/blog/2023/12/31/hello-ebpf-developing-ebpf-apps-in-java-1/ https://mostlynerdless.de/blog/2023/12/31/hello-ebpf-develop... https://mostlynerdless.de/blog/category/computer-science/ebpf/ https://mostlynerdless.de/blog/category/computer-science/ebp...

xbar 1y ago
I missed these last year. Finding them now is truly very useful in my work.

pimterry 1y ago

Does anybody know if something similar exists for Node.js? I'd love to be able to integrate BPF into some of my Node projects with the same kind of approach.

lloydatkinson 1y ago
Please no...
- deleted 1y ago
  [deleted]
ranger_danger 1y ago
Dear god... why

drivenextfunc 1y ago

Writing C for eBPF is cumbersome and you'd like to avoid it. Okay, that's reasonable. But I don't think it would be a good idea to write a compiler that emits eBPF binary from (a tiny subset of) Python. Why not just write code in pseudo-Python (or whatever language you're comfortable with) and have it translated by an LLM, and paste it in the source code? That would be much better because there would be fewer layers and a significant reduction in runtime cost.

tecleandor 1y ago
I don't understand... So, instead of having a defined and documented subset of Python that compiles to eBPF in a deterministic way... use an undefined pseudo language and let the LLM have fun with it without understanding if the result C is correct? What would be the advantage?
- drivenextfunc 1y ago
  The behavior of CPython and a few other implementations of Python (such as PyPy) is well documented and well understood. The semantics of the tiny subset of Python that this Python-to-eBPF compiler understands is not. For example, inferring from the fact that it statically compiles Python-ish AST to LLVM IR, you can have a rough idea that dynamic elements of Python semantics are unlikely to be compiled, but you cannot know exactly which elements without carefully reading the documentation or source code of the compiler. You can guess globals() or locals() won't work, maybe .__dict__ won't as well, but how about type() or isinstance()? You don't know without digging into the documentation (which may be lacking), because the subset of Python this compiler understands is rather arbitrary. And also, having an LLM translate Python-ish pseudo code into C does not imply that you cannot examine it before putting it into a program. You can manually review it and make modifications as you want. It just reduces time spent compared with writing C code by hand.
  - tecleandor 1y ago
    But then we have to write the pseudocode anyway (that cannot be corrected by my IDE, so I don't know if I have pseudomistakes [sorry for the pun]), the LLM 'transpile' (that's not understood at all), and you have to review the C code anyway, so you have to know eBPF code really well. Would that represent a time advantage?

grantseltzer 1y ago

bcc hasn't been relevant for years.

_bobm 1y ago
I have been a bit out of the loop. what is relevant these days for writing ebpf code? what about ebpf code in python?
- nickysielicki 1y ago
  bpftrace is nicer to work with and can replace bcc in most cases for debugging.
- grantseltzer 1y ago
  Writing it in C, compiling with clang, and loading with either C(libbpf), Go (cilium/ebpf), or Rust (Aya). You can also write bpf in rust with Aya but i'm not sure how feature complete it is. For very simple use cases you can just bpftrace.

bieganski 1y ago

that's really cool. to gain traction i would start with reimplementing all the tools from https://github.com/iovisor/bcc/tree/master/libbpf-tools https://github.com/iovisor/bcc/tree/master/libbpf-tools in PythonBPF.

varunrmallya 1y ago
This is actually what we plan to do too!

robertlagrant 1y ago

Step 1: import numpy

setheron 1y ago

I had similar idea ! Loved seeing it here. We thought about doing it with ChocoPy to make the types more consistent.