4 ms·

Compiling dynamically generated C programs on demand to provide packet filtering? It's clever but not in a good way. This is a really dangerous approach to solv

Compiling dynamically generated C programs on demand to provide packet filtering? It's clever but not in a good way. This is a really dangerous approach to solving this problem.

alexgartrell 7y ago
First, there's no need to do this "on demand." You can ship a .o around and load it with bpftool. My team does a version of this at Facebook (using libbpf directly) and it works really well. The big use case for on-demand programming is in tracing where you want to capture structs, but BTF and Compile-once, run-everywhere will mitigate this need [0] Second, I don't actually agree. JIT compilation is a generally accepted approach -- who cares what the IR is? And the bpf runtime is ULTRA constrained, so you can't slip a `system("rm -rf /")` in there (by a long shot). [0] https://www.kernel.org/doc/html/latest/bpf/btf.html https://www.kernel.org/doc/html/latest/bpf/btf.html
- jdc 7y ago
  I'm thoroughly skeptical that general acceptance means that JIT compilation doesn't increase the attack surface of the stack.
  - Dylan16807 7y ago
    You'd need an interpreter at least, to avoid some kind of hideously complex system. So it's a question of interpreter vs. JIT, and a JIT makes it more feasible to use an ultra-simple language with aggressive verification, without losing too much speed. Every feature has an attack surface, but you have to compare against the alternative, not the lack of feature. (This assumes that you can't force it to use a kernel interpreter despite the JIT existing. Otherwise perhaps that is the part that should be disabled.)
- ignoramous 7y ago
  Tangential: Does your team publish papers or write blog posts on the kind of engineering or research it does? I'd like to read it, or at least add it to my queue. Thanks.
  - jauer 7y ago
    There was a recent talk on tupperware that gets into some of what Alex's team does: https://engineering.fb.com/data-center-engineering/tupperware/ https://engineering.fb.com/data-center-engineering/tupperwar...
    - ignoramous 7y ago
      That was interesting. Thanks.
edoo 7y ago
How is it any different than letting a script run as root? This seems very similar to varnish cache which is amazing.
rustaceanunited 7y ago
Almost 2020 and we still have to deal with that legacy language.
justicezyx 7y ago
What are the dangers? AFAIK, BPF has in-kernel verifier on the safety of the BPF program, which is conservative and would reject safe program, let alone the real dangerous ones.
- dfay 7y ago
  The talk hasn't been posted to youtube yet, but slides alone show some of the risks of eBPF. The speaker's previous talk at 35C3 went into how the verifier is arbitrary and not really designed to prevent attackers anyway. https://media.defcon.org/DEF%20CON%2027/DEF%20CON%2027%20presentations/DEFCON-27-Jeff-Dileo-Evil-eBPF-In-Depth.pdf https://media.defcon.org/DEF%20CON%2027/DEF%20CON%2027%20pre...
- geofft 7y ago
  The verifier has had bugs in the past. Here's six of them from 2017: https://www.openwall.com/lists/oss-security/2017/12/23/2 https://www.openwall.com/lists/oss-security/2017/12/23/2 one of which was discovered to have an incomplete fix in 2018: https://crbug.com/project-zero/1686 https://crbug.com/project-zero/1686 https://medium.com/@tyanir/understanding-bpf-check-alu-op-vulnerability-cve-2017-16995-3eb3017f598a https://medium.com/@tyanir/understanding-bpf-check-alu-op-vu... has a longer description of one of them - namely, you can convince the verifier that a certain part of the program is dead code and therefore doesn't need to be verified when it does, and thereby get arbitrary unverified eBPF into the kernel.
Klasiaster 7y ago
Hi, I chose this way because it's a proof-of-concept. Normally this filtering expression would be stored in a BPF map but I didn't invest more time and wanted to keep it simple as an example for writing BPF programs for systemd services.
azifali 7y ago
There's a limited instruction set, no loops, built in verifier exists so that no anomalous code can be executed on the in kernel virtual machine. More so, this feature is in 4.X kernel and I think there's been no exploit discovered in the wild.
- viraptor 7y ago
  > More so, this feature is in 4.X kernel and I think there's been no exploit discovered in the wild. There won't be an exploit for bpf. It's kind of a different layer and it's own system. "Exploit in bpf" is about the same level as "exploit in C". There's just no such general thing.

Compiling dynamically generated C programs on demand to provide packet filtering? It's clever but not in a good way. This is a really dangerous approach to solving this problem.

alexgartrell 7y ago

First, there's no need to do this "on demand." You can ship a .o around and load it with bpftool. My team does a version of this at Facebook (using libbpf directly) and it works really well. The big use case for on-demand programming is in tracing where you want to capture structs, but BTF and Compile-once, run-everywhere will mitigate this need [0] Second, I don't actually agree. JIT compilation is a generally accepted approach -- who cares what the IR is? And the bpf runtime is ULTRA constrained, so you can't slip a `system("rm -rf /")` in there (by a long shot). [0] https://www.kernel.org/doc/html/latest/bpf/btf.html https://www.kernel.org/doc/html/latest/bpf/btf.html

jdc 7y ago
I'm thoroughly skeptical that general acceptance means that JIT compilation doesn't increase the attack surface of the stack.
- Dylan16807 7y ago
  You'd need an interpreter at least, to avoid some kind of hideously complex system. So it's a question of interpreter vs. JIT, and a JIT makes it more feasible to use an ultra-simple language with aggressive verification, without losing too much speed. Every feature has an attack surface, but you have to compare against the alternative, not the lack of feature. (This assumes that you can't force it to use a kernel interpreter despite the JIT existing. Otherwise perhaps that is the part that should be disabled.)
ignoramous 7y ago
Tangential: Does your team publish papers or write blog posts on the kind of engineering or research it does? I'd like to read it, or at least add it to my queue. Thanks.
- jauer 7y ago
  There was a recent talk on tupperware that gets into some of what Alex's team does: https://engineering.fb.com/data-center-engineering/tupperware/ https://engineering.fb.com/data-center-engineering/tupperwar...
  - ignoramous 7y ago
    That was interesting. Thanks.

edoo 7y ago

How is it any different than letting a script run as root? This seems very similar to varnish cache which is amazing.

rustaceanunited 7y ago

Almost 2020 and we still have to deal with that legacy language.

justicezyx 7y ago

What are the dangers? AFAIK, BPF has in-kernel verifier on the safety of the BPF program, which is conservative and would reject safe program, let alone the real dangerous ones.

dfay 7y ago
The talk hasn't been posted to youtube yet, but slides alone show some of the risks of eBPF. The speaker's previous talk at 35C3 went into how the verifier is arbitrary and not really designed to prevent attackers anyway. https://media.defcon.org/DEF%20CON%2027/DEF%20CON%2027%20presentations/DEFCON-27-Jeff-Dileo-Evil-eBPF-In-Depth.pdf https://media.defcon.org/DEF%20CON%2027/DEF%20CON%2027%20pre...
geofft 7y ago
The verifier has had bugs in the past. Here's six of them from 2017: https://www.openwall.com/lists/oss-security/2017/12/23/2 https://www.openwall.com/lists/oss-security/2017/12/23/2 one of which was discovered to have an incomplete fix in 2018: https://crbug.com/project-zero/1686 https://crbug.com/project-zero/1686 https://medium.com/@tyanir/understanding-bpf-check-alu-op-vulnerability-cve-2017-16995-3eb3017f598a https://medium.com/@tyanir/understanding-bpf-check-alu-op-vu... has a longer description of one of them - namely, you can convince the verifier that a certain part of the program is dead code and therefore doesn't need to be verified when it does, and thereby get arbitrary unverified eBPF into the kernel.

Klasiaster 7y ago

Hi, I chose this way because it's a proof-of-concept. Normally this filtering expression would be stored in a BPF map but I didn't invest more time and wanted to keep it simple as an example for writing BPF programs for systemd services.

azifali 7y ago

There's a limited instruction set, no loops, built in verifier exists so that no anomalous code can be executed on the in kernel virtual machine. More so, this feature is in 4.X kernel and I think there's been no exploit discovered in the wild.

viraptor 7y ago
> More so, this feature is in 4.X kernel and I think there's been no exploit discovered in the wild. There won't be an exploit for bpf. It's kind of a different layer and it's own system. "Exploit in bpf" is about the same level as "exploit in C". There's just no such general thing.