7 ms·

Notes on BPF and eBPF

216 points by mlerner 5y ago

netsec_burn 5y ago
Many of the links under "things you can attach eBPF programs to" are broken, unfortunately.
daenz 5y ago
>eBPF programs can’t access arbitrary kernel memory. Instead the kernel provides functions to get at some restricted subset of things. I must finally becoming a security pessimist when I read those sentences and the first thing I think is: these statements will not age well.
- stefan_ 5y ago
  I think they already don't. eBPF has become a favorite jumping point for exploit chains that can very quickly escalate some read/write gadget into full execution.
  - kingforaday 5y ago
    I would love to read an in depth breakdown of an exploit using eBPF as part of its execution chain. Do you happen to have an example/link?
    - stefan_ 5y ago
      This was the one I was thinking of: https://googleprojectzero.blogspot.com/2020/ https://googleprojectzero.blogspot.com/2020/ Scroll down to "The ultimate ROP"
- throway453sde 5y ago
  Are you saying this is factually wrong? I assume eBPF is designed to only access certain points explicitly coded in the kernel but the exploits can access any part of kernel. Is this what you are hinting at? Could you please clarify the statement.
  - tptacek 5y ago
    There are at least two issues here. First: eBPF code is JIT'd in the kernel; at runtime, it is simply native code running at CPL0 alongside the rest of the kernel. Running eBPF code working with pointers is... working with raw pointers. There's no interpretation layer to bounds check or otherwise provide safety. But eBPF code is meant to be safe: you can get a handle to some kernel structure that's passed to you from trusted code, but you can't bounce from it to a random offset in kernel memory. The way eBPF does this is by verifying the CFG of your eBPF program before it's translated to amd64 or arm. eBPF programs are generally just C programs (the simplest and best way to write an eBPF program is just to write a C program and compile it with the right LLVM flags), and verifying C programs is a hard problem; eBPF gets around this by only accepting a subset of all possible programs (those where memory accesses are simple enough to prove safe, that don't jump anywhere outside of known narrow range of program text, and that don't have unbounded loops). The tricky thing here is that the eBPF verifier is pretty complicated and lives only in the kernel. People have found bugs in it. If you find a good verifier bug, you can launder an untrusted pointer into your eBPF program (in the end, these bugs end up looking sort of like the browser Javascript RCEs that finagle a bad pointer out of some part of the browser API). The biggest mitigating factor for these bugs is that Linux systems generally don't expose eBPF to any user other than root, so the upside to these kinds of bugs is limited (it gives you root->kernel, which is not nothing, but not the top of most people's priority list). The other big issue is that eBPF is a huge source of in-kernel flexibility about runnable code. Modern exploit mitigations are in large part about making sure that instructions running at CPL0 are all known, so that if you manage to corrupt allocator metadata or write an arbitrary 8 byte value at an arbitrary 8 byte offset you can't easily turn that into remote code execution. But, of course, eBPF is an in-kernel JIT; it's there to run essentially random code inside the kernel. eBPF code is normally constrained, but if you have a kernel memory corruption bug, you can aim it at the eBPF subsystem and violate the kernel's assumptions.
- Flocular 5y ago
  The BPF capability should really only be given to root. I don't think it really gives any new attack surface. All I could see is it giving black-hats an easier interface to "kernel-level-fuckery".
  - _3u10 5y ago
    It’s not an easier interface. It’s much easier to write kernel modules than to mess around with eBPF.
    - stormbrew 5y ago
      Easier to get started maybe, but there's something to be said for the 'ease' of having to worry a lot less about crashing the kernel you're working on.
      - xyzzy_plugh 5y ago
        I've crashed multiple kernels with eBPF programs. I don't buy this argument at all.
        justicezyx 5y ago
        Could you share the ebpf program and the external dependencies that can reproduce the crash? It'll be valuable to learn this, so that we might be able to proactively address them. Ebpf is the core of our product.
        deleted 5y ago
        [deleted]
        roblabla 5y ago
        Those would be considered kernel bugs though, and should be getting fixed when reported. While a crash in a kernel module is just a bug in the kernel module itself
        stormbrew 5y ago
- fragmede 5y ago
  That's definitely a security professional's read :) "Isn't supposed to be able to" is a lot longer and distracting vs the oversimplification-for-sake-of-understanding of "can't". As far as it being proven wrong though - that's already happened, eg CVE-2021-29154 https://blog.kernelcare.com/vulnerability/specially-crafted-bpf-code-can-allow-local-privilege-escalation-cve-2021-29154 https://blog.kernelcare.com/vulnerability/specially-crafted-...
  - daenz 5y ago
    That's fair. I understand that in the ideal world, it would be "can't." I guess my concern is that the wording kind of hand waves away any potential security issues, when people interested in this tech should absolutely be made aware of them.
    - quickthrower2 5y ago
      “is not authorized to” ?
- csdvrx 5y ago
  Yes, "can't" should be replace by "shouldn't". If there's a physical possibility, it's just a matter of time before someone finds a way, as was proved by the CPU cache bugs leaking information.
  - emmelaich 5y ago
    "shouldn't" is ambiguous as it might suggest the responsibility is with the the user of eBPF rather than internal to it. Perhaps "is designed not to" but that's a mouthful. I think we should accept "can't" and yet know the limitations of certainty.
- IshKebab 5y ago
  Yes, especially with Spectre. I looked into using eBPF with seccomp (which currently only supports BPF) and the consensus seemed to be that they're not going to add any more eBPF to the kernel and have basically given up on making it secure. I wouldn't be surprised if it becomes entirely root-only by default soon, if it isn't already.
  - staticassertion 5y ago
    For others who are curious, https://lwn.net/Articles/857228/ https://lwn.net/Articles/857228/
pwnna 5y ago
BPF is indeed a pretty interesting technology. As the knowledge about it becomes more widespread, I anticipate that we will unlock some new capabilities both in terms of tracing. Brendan Gregg's book (https://www.brendangregg.com/bpf-performance-tools-book.html https://www.brendangregg.com/bpf-performance-tools-book.html) serves as a good intro to this, although you probably only need to read a small chunk of it as a lot of it is reference-book-style material. The author's mentioned that you can trace MySQL with USDT, which is a tracepoint inserted by the developer at select locations in the code. This kind of tracepoints form a "stable interface" for tracing/performance debugging, whereas uprobe, which hooks into select userspace functions, are unstable as the binary is recompiled. Unfortunately, the USDT tracepoints (via DTrace) have been removed in MySQL 8.0. This makes it significantly more difficult to trace MySQL, although it's not impossiblhttps://news.ycombinator.com/item?id=29772927e https://news.ycombinator.com/item?id=29772927e. I've done a proof of concept of tracing MySQL with uprobe instead of USDT in this repo[1], which can kind of give you the same results (and possibly more stuff, as I can more easily read arbitrary memory address due to how the old USDT tracepoints are structured). This is not stable tho, as any MySQL upgrade may introduce incompatibility with the trace script, as I read memory address based on offsets (whereas with USDT this can be kept pretty stable). My appeal to Oracle to re-add this functionality[2] has unfortunately been rejected, which I think is a mistake given the wide range of possibilities unlocked via BPF. [1]: https://github.com/shuhaowu/mysqld-bpf https://github.com/shuhaowu/mysqld-bpf [2]: https://bugs.mysql.com/bug.php?id=105741 https://bugs.mysql.com/bug.php?id=105741 Another thing that I've been recently thinking of is using BPF to validate programs written for real-time Linux (via PREEMPT_RT). To my understanding, one of the main thing to avoid is page faults [3]. With the proper BPF tracing scripts, I think we can validate that programs indeed avoids page faults in integration testing. I'm not sure if it is super useful yet, but as I'm trying to write a few RT programs, it's something that came to my mind. [3]: https://lwn.net/Articles/837019/ https://lwn.net/Articles/837019/ In addition to tracing (so bpftrace-based/bcc-based tools), I've recently discovered that there there are: 1. ebpfsnitch (https://github.com/harporoeder/ebpfsnitch https://github.com/harporoeder/ebpfsnitch): which is an application-level firewall without kernel modules. 2. ebpf-traffic-monitor (https://source.android.com/devices/tech/datausage/ebpf-traffic-monitor https://source.android.com/devices/tech/datausage/ebpf-traff...): which appears to be using BPF to account for traffic for different apps on Android. 3. kubectl trace (https://github.com/iovisor/kubectl-trace https://github.com/iovisor/kubectl-trace): Run tracing on k8s. There are apparently also use cases in the context of security, but I'm not familiar with it.
kylequest 5y ago
Lots of good eBPF info from eBPF Summit: https://ebpf.io/summit-2021/ https://ebpf.io/summit-2021/ and https://ebpf.io/summit-2020/ https://ebpf.io/summit-2020/ Also videos from eBPF Day KubeCon 2021: https://www.youtube.com/playlist?list=PLj6h78yzYM2Pm5nF_GmNQHMyt9CUZr2uQ https://www.youtube.com/playlist?list=PLj6h78yzYM2Pm5nF_GmNQ...
rammy1234 5y ago
Is PDF link broken in the blog ?
- gnabgib 5y ago
  It's available at: https://files.speakerdeck.com/presentations/130bc7df16db4556a55105af45cdf3ba/PWL-Jun28-MTL.pdf https://files.speakerdeck.com/presentations/130bc7df16db4556... (or click the download button on the slides page)
crtxcr 5y ago
>things you can attach eBPF programs to >... >seccomp / landlock security things Landlock does not use *BPF. Seccomp can only use BPF at this point, not eBPF (though there has been some work on it).
halpert 5y ago
One cool project that uses eBPF is Cilium. It allows restricting network traffic to / from containers in Kubernetes. Many of the problems it solves, in my opinion, are better solved via user-space solutions, e.g. service-to-service traffic is better controlled via signing / encryption, but overall Cilium is a pretty cool piece of technology.
- staticassertion 5y ago
  It seems nice to have both layers. mTLS is great, but you're still exposing your TLS stack to the attacker. Dropping the packet altogether seems nicer.
  - halpert 5y ago
    In a perfect world, yes. What I’ve found in practice is that network policies add a mysterious failure point that makes debugging traffic issues hard, especially when providing a service platform to teams that don’t understand the inner workings. TLS failures tend to be easier to grok for most service devs.
Warlockcraft 5y ago
I tried clicking on the link with these words: "The BSD Packet Filter: A New Architecture for User-level Packet Capture". The link appears to be an unsecure website that my internet browser preevented me from going on.

216 points by mlerner 5y ago

Many of the links under "things you can attach eBPF programs to" are broken, unfortunately.

>eBPF programs can’t access arbitrary kernel memory. Instead the kernel provides functions to get at some restricted subset of things. I must finally becoming a security pessimist when I read those sentences and the first thing I think is: these statements will not age well.

stefan_ 5y ago
I think they already don't. eBPF has become a favorite jumping point for exploit chains that can very quickly escalate some read/write gadget into full execution.
- kingforaday 5y ago
  I would love to read an in depth breakdown of an exploit using eBPF as part of its execution chain. Do you happen to have an example/link?
  - stefan_ 5y ago
    This was the one I was thinking of: https://googleprojectzero.blogspot.com/2020/ https://googleprojectzero.blogspot.com/2020/ Scroll down to "The ultimate ROP"
throway453sde 5y ago
Are you saying this is factually wrong? I assume eBPF is designed to only access certain points explicitly coded in the kernel but the exploits can access any part of kernel. Is this what you are hinting at? Could you please clarify the statement.
- tptacek 5y ago
  There are at least two issues here. First: eBPF code is JIT'd in the kernel; at runtime, it is simply native code running at CPL0 alongside the rest of the kernel. Running eBPF code working with pointers is... working with raw pointers. There's no interpretation layer to bounds check or otherwise provide safety. But eBPF code is meant to be safe: you can get a handle to some kernel structure that's passed to you from trusted code, but you can't bounce from it to a random offset in kernel memory. The way eBPF does this is by verifying the CFG of your eBPF program before it's translated to amd64 or arm. eBPF programs are generally just C programs (the simplest and best way to write an eBPF program is just to write a C program and compile it with the right LLVM flags), and verifying C programs is a hard problem; eBPF gets around this by only accepting a subset of all possible programs (those where memory accesses are simple enough to prove safe, that don't jump anywhere outside of known narrow range of program text, and that don't have unbounded loops). The tricky thing here is that the eBPF verifier is pretty complicated and lives only in the kernel. People have found bugs in it. If you find a good verifier bug, you can launder an untrusted pointer into your eBPF program (in the end, these bugs end up looking sort of like the browser Javascript RCEs that finagle a bad pointer out of some part of the browser API). The biggest mitigating factor for these bugs is that Linux systems generally don't expose eBPF to any user other than root, so the upside to these kinds of bugs is limited (it gives you root->kernel, which is not nothing, but not the top of most people's priority list). The other big issue is that eBPF is a huge source of in-kernel flexibility about runnable code. Modern exploit mitigations are in large part about making sure that instructions running at CPL0 are all known, so that if you manage to corrupt allocator metadata or write an arbitrary 8 byte value at an arbitrary 8 byte offset you can't easily turn that into remote code execution. But, of course, eBPF is an in-kernel JIT; it's there to run essentially random code inside the kernel. eBPF code is normally constrained, but if you have a kernel memory corruption bug, you can aim it at the eBPF subsystem and violate the kernel's assumptions.
Flocular 5y ago
The BPF capability should really only be given to root. I don't think it really gives any new attack surface. All I could see is it giving black-hats an easier interface to "kernel-level-fuckery".
- _3u10 5y ago
  It’s not an easier interface. It’s much easier to write kernel modules than to mess around with eBPF.
  - stormbrew 5y ago
    Easier to get started maybe, but there's something to be said for the 'ease' of having to worry a lot less about crashing the kernel you're working on.
    - xyzzy_plugh 5y ago
      I've crashed multiple kernels with eBPF programs. I don't buy this argument at all.
      - justicezyx 5y ago
        Could you share the ebpf program and the external dependencies that can reproduce the crash? It'll be valuable to learn this, so that we might be able to proactively address them. Ebpf is the core of our product.
      - deleted 5y ago
        [deleted]
      - roblabla 5y ago
        Those would be considered kernel bugs though, and should be getting fixed when reported. While a crash in a kernel module is just a bug in the kernel module itself
      - stormbrew 5y ago
fragmede 5y ago
That's definitely a security professional's read :) "Isn't supposed to be able to" is a lot longer and distracting vs the oversimplification-for-sake-of-understanding of "can't". As far as it being proven wrong though - that's already happened, eg CVE-2021-29154 https://blog.kernelcare.com/vulnerability/specially-crafted-bpf-code-can-allow-local-privilege-escalation-cve-2021-29154 https://blog.kernelcare.com/vulnerability/specially-crafted-...
- daenz 5y ago
  That's fair. I understand that in the ideal world, it would be "can't." I guess my concern is that the wording kind of hand waves away any potential security issues, when people interested in this tech should absolutely be made aware of them.
  - quickthrower2 5y ago
    “is not authorized to” ?
csdvrx 5y ago
Yes, "can't" should be replace by "shouldn't". If there's a physical possibility, it's just a matter of time before someone finds a way, as was proved by the CPU cache bugs leaking information.
- emmelaich 5y ago
  "shouldn't" is ambiguous as it might suggest the responsibility is with the the user of eBPF rather than internal to it. Perhaps "is designed not to" but that's a mouthful. I think we should accept "can't" and yet know the limitations of certainty.
IshKebab 5y ago
Yes, especially with Spectre. I looked into using eBPF with seccomp (which currently only supports BPF) and the consensus seemed to be that they're not going to add any more eBPF to the kernel and have basically given up on making it secure. I wouldn't be surprised if it becomes entirely root-only by default soon, if it isn't already.
- staticassertion 5y ago
  For others who are curious, https://lwn.net/Articles/857228/ https://lwn.net/Articles/857228/

pwnna 5y ago

BPF is indeed a pretty interesting technology. As the knowledge about it becomes more widespread, I anticipate that we will unlock some new capabilities both in terms of tracing. Brendan Gregg's book (https://www.brendangregg.com/bpf-performance-tools-book.html https://www.brendangregg.com/bpf-performance-tools-book.html) serves as a good intro to this, although you probably only need to read a small chunk of it as a lot of it is reference-book-style material. The author's mentioned that you can trace MySQL with USDT, which is a tracepoint inserted by the developer at select locations in the code. This kind of tracepoints form a "stable interface" for tracing/performance debugging, whereas uprobe, which hooks into select userspace functions, are unstable as the binary is recompiled. Unfortunately, the USDT tracepoints (via DTrace) have been removed in MySQL 8.0. This makes it significantly more difficult to trace MySQL, although it's not impossiblhttps://news.ycombinator.com/item?id=29772927e https://news.ycombinator.com/item?id=29772927e. I've done a proof of concept of tracing MySQL with uprobe instead of USDT in this repo[1], which can kind of give you the same results (and possibly more stuff, as I can more easily read arbitrary memory address due to how the old USDT tracepoints are structured). This is not stable tho, as any MySQL upgrade may introduce incompatibility with the trace script, as I read memory address based on offsets (whereas with USDT this can be kept pretty stable). My appeal to Oracle to re-add this functionality[2] has unfortunately been rejected, which I think is a mistake given the wide range of possibilities unlocked via BPF. [1]: https://github.com/shuhaowu/mysqld-bpf https://github.com/shuhaowu/mysqld-bpf [2]: https://bugs.mysql.com/bug.php?id=105741 https://bugs.mysql.com/bug.php?id=105741 Another thing that I've been recently thinking of is using BPF to validate programs written for real-time Linux (via PREEMPT_RT). To my understanding, one of the main thing to avoid is page faults [3]. With the proper BPF tracing scripts, I think we can validate that programs indeed avoids page faults in integration testing. I'm not sure if it is super useful yet, but as I'm trying to write a few RT programs, it's something that came to my mind. [3]: https://lwn.net/Articles/837019/ https://lwn.net/Articles/837019/ In addition to tracing (so bpftrace-based/bcc-based tools), I've recently discovered that there there are: 1. ebpfsnitch (https://github.com/harporoeder/ebpfsnitch https://github.com/harporoeder/ebpfsnitch): which is an application-level firewall without kernel modules. 2. ebpf-traffic-monitor (https://source.android.com/devices/tech/datausage/ebpf-traffic-monitor https://source.android.com/devices/tech/datausage/ebpf-traff...): which appears to be using BPF to account for traffic for different apps on Android. 3. kubectl trace (https://github.com/iovisor/kubectl-trace https://github.com/iovisor/kubectl-trace): Run tracing on k8s. There are apparently also use cases in the context of security, but I'm not familiar with it.

kylequest 5y ago

Lots of good eBPF info from eBPF Summit: https://ebpf.io/summit-2021/ https://ebpf.io/summit-2021/ and https://ebpf.io/summit-2020/ https://ebpf.io/summit-2020/ Also videos from eBPF Day KubeCon 2021: https://www.youtube.com/playlist?list=PLj6h78yzYM2Pm5nF_GmNQHMyt9CUZr2uQ https://www.youtube.com/playlist?list=PLj6h78yzYM2Pm5nF_GmNQ...

rammy1234 5y ago

Is PDF link broken in the blog ?

gnabgib 5y ago
It's available at: https://files.speakerdeck.com/presentations/130bc7df16db4556a55105af45cdf3ba/PWL-Jun28-MTL.pdf https://files.speakerdeck.com/presentations/130bc7df16db4556... (or click the download button on the slides page)

crtxcr 5y ago

>things you can attach eBPF programs to >... >seccomp / landlock security things Landlock does not use *BPF. Seccomp can only use BPF at this point, not eBPF (though there has been some work on it).

halpert 5y ago

One cool project that uses eBPF is Cilium. It allows restricting network traffic to / from containers in Kubernetes. Many of the problems it solves, in my opinion, are better solved via user-space solutions, e.g. service-to-service traffic is better controlled via signing / encryption, but overall Cilium is a pretty cool piece of technology.

staticassertion 5y ago
It seems nice to have both layers. mTLS is great, but you're still exposing your TLS stack to the attacker. Dropping the packet altogether seems nicer.
- halpert 5y ago
  In a perfect world, yes. What I’ve found in practice is that network policies add a mysterious failure point that makes debugging traffic issues hard, especially when providing a service platform to teams that don’t understand the inner workings. TLS failures tend to be easier to grok for most service devs.

Warlockcraft 5y ago

I tried clicking on the link with these words: "The BSD Packet Filter: A New Architecture for User-level Packet Capture". The link appears to be an unsecure website that my internet browser preevented me from going on.