5 ms·

Kernel Pwning with eBPF: A Love Story

136 points by arkadiyt 5y ago

ericbarrett 5y ago
This is about CVE-2021-3490: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3490 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3490
- tptacek 5y ago
  It's a truly excellent bug, a gnarly logic problem (it superficially looks like a 32/64 problem, but it's really just a logic bug that follows those same contours) that kind of betrays how brittle the eBPF verifier is. I look at this bug and what it shows about the assumptions baked into eBPF verification and conclude that it is not safe for unprivileged programs to use. What's nice is, this bug says nothing about the verifier's ability to protect kernel stability from stupid, rather than malicious, code. Which, for 99% of eBPF applications, is all you care about! It's still a great way to let systems programmers extend the kernel.
  - staticassertion 5y ago
    > I look at this bug and what it shows about the assumptions baked into eBPF verification and conclude that it is not safe for unprivileged programs to use. This is actually something we took away from this (I work at Grapl) as well. We consider the Linux kernel to be unsuitable for isolation in some of our services, and it at least requires hardening (such as limiting unprivileged ebpf) in others.
infogulch 5y ago
How does eBPF compare to WebAsm? I'm a bit more familiar with the latter (thanks to some great articles) and I know that they use the same general approach of safety via linear-time verification and translation to machine code+buffer overflow checks, but I'm curious about the differences.
- tptacek 5y ago
  BPF has a much more limited execution model (in general, eBPF programs aren't even allowed to loop, though the verifier may allow some loops that it can prove are bounded), but freer access to the underlying machine (by design, your programs are usually handling live kernel memory directly).
  - monocasa 5y ago
    In more general terms, the eBPF VM goes out of it's way to not be Turing complete in order to solve it's version of the halting problem. Instruction stream graphs must be able to be translated to a acyclic graph by the verifier.
    - infogulch 5y ago
      Interesting, that makes me wonder if a similar restricted "acyclic control flow" mode could be built into WA; I bet that would be pretty useful.
      - tptacek 5y ago
        It is extremely limiting and makes programming very difficult.
        infogulch 5y ago
        Yes, clearly. But it could be used in places where you want performance guarantees. Like, a client sends a custom sort function to a server to build responses, wide-open WA would not be appropriate for this but maybe a guaranteed acyclic control flow could work. Etc.
        monocasa 5y ago
        Sort is interesting because it's difficult to make a sort algorithm without unbounded loops. Although I guess a comparison function is probably easier, and leave the sort itself in the trusted computing base.
pjmlp 5y ago
> An LPE exploit writeup by Qualys published in July 2021 uses a memory corruption vulnerability to overwrite an eBPF program buffer, which contains the eBPF bytecode for a loaded program. Instead of fooling the verifier, the program is valid and passes verification. The buffer is overwritten after verification but before JIT compilation to achieve OOB read/write capabilities Lovely.

136 points by arkadiyt 5y ago

This is about CVE-2021-3490: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3490 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3490

tptacek 5y ago
It's a truly excellent bug, a gnarly logic problem (it superficially looks like a 32/64 problem, but it's really just a logic bug that follows those same contours) that kind of betrays how brittle the eBPF verifier is. I look at this bug and what it shows about the assumptions baked into eBPF verification and conclude that it is not safe for unprivileged programs to use. What's nice is, this bug says nothing about the verifier's ability to protect kernel stability from stupid, rather than malicious, code. Which, for 99% of eBPF applications, is all you care about! It's still a great way to let systems programmers extend the kernel.
- staticassertion 5y ago
  > I look at this bug and what it shows about the assumptions baked into eBPF verification and conclude that it is not safe for unprivileged programs to use. This is actually something we took away from this (I work at Grapl) as well. We consider the Linux kernel to be unsuitable for isolation in some of our services, and it at least requires hardening (such as limiting unprivileged ebpf) in others.

infogulch 5y ago

How does eBPF compare to WebAsm? I'm a bit more familiar with the latter (thanks to some great articles) and I know that they use the same general approach of safety via linear-time verification and translation to machine code+buffer overflow checks, but I'm curious about the differences.

tptacek 5y ago
BPF has a much more limited execution model (in general, eBPF programs aren't even allowed to loop, though the verifier may allow some loops that it can prove are bounded), but freer access to the underlying machine (by design, your programs are usually handling live kernel memory directly).
- monocasa 5y ago
  In more general terms, the eBPF VM goes out of it's way to not be Turing complete in order to solve it's version of the halting problem. Instruction stream graphs must be able to be translated to a acyclic graph by the verifier.
  - infogulch 5y ago
    Interesting, that makes me wonder if a similar restricted "acyclic control flow" mode could be built into WA; I bet that would be pretty useful.
    - tptacek 5y ago
      It is extremely limiting and makes programming very difficult.
      - infogulch 5y ago
        Yes, clearly. But it could be used in places where you want performance guarantees. Like, a client sends a custom sort function to a server to build responses, wide-open WA would not be appropriate for this but maybe a guaranteed acyclic control flow could work. Etc.
        monocasa 5y ago
        Sort is interesting because it's difficult to make a sort algorithm without unbounded loops. Although I guess a comparison function is probably easier, and leave the sort itself in the trusted computing base.

pjmlp 5y ago

> An LPE exploit writeup by Qualys published in July 2021 uses a memory corruption vulnerability to overwrite an eBPF program buffer, which contains the eBPF bytecode for a loaded program. Instead of fooling the verifier, the program is valid and passes verification. The buffer is overwritten after verification but before JIT compilation to achieve OOB read/write capabilities Lovely.