10 ms·

BPF port-based firewall for systemd services

103 points by Klasiaster 7y ago

skywhopper 7y ago
Compiling dynamically generated C programs on demand to provide packet filtering? It's clever but not in a good way. This is a really dangerous approach to solving this problem.
- alexgartrell 7y ago
  First, there's no need to do this "on demand." You can ship a .o around and load it with bpftool. My team does a version of this at Facebook (using libbpf directly) and it works really well. The big use case for on-demand programming is in tracing where you want to capture structs, but BTF and Compile-once, run-everywhere will mitigate this need [0] Second, I don't actually agree. JIT compilation is a generally accepted approach -- who cares what the IR is? And the bpf runtime is ULTRA constrained, so you can't slip a `system("rm -rf /")` in there (by a long shot). [0] https://www.kernel.org/doc/html/latest/bpf/btf.html https://www.kernel.org/doc/html/latest/bpf/btf.html
  - jdc 7y ago
    I'm thoroughly skeptical that general acceptance means that JIT compilation doesn't increase the attack surface of the stack.
    - Dylan16807 7y ago
      You'd need an interpreter at least, to avoid some kind of hideously complex system. So it's a question of interpreter vs. JIT, and a JIT makes it more feasible to use an ultra-simple language with aggressive verification, without losing too much speed. Every feature has an attack surface, but you have to compare against the alternative, not the lack of feature. (This assumes that you can't force it to use a kernel interpreter despite the JIT existing. Otherwise perhaps that is the part that should be disabled.)
  - ignoramous 7y ago
    Tangential: Does your team publish papers or write blog posts on the kind of engineering or research it does? I'd like to read it, or at least add it to my queue. Thanks.
    - jauer 7y ago
      There was a recent talk on tupperware that gets into some of what Alex's team does: https://engineering.fb.com/data-center-engineering/tupperware/ https://engineering.fb.com/data-center-engineering/tupperwar...
      - ignoramous 7y ago
        That was interesting. Thanks.
- edoo 7y ago
  How is it any different than letting a script run as root? This seems very similar to varnish cache which is amazing.
- rustaceanunited 7y ago
  Almost 2020 and we still have to deal with that legacy language.
- justicezyx 7y ago
  What are the dangers? AFAIK, BPF has in-kernel verifier on the safety of the BPF program, which is conservative and would reject safe program, let alone the real dangerous ones.
  - dfay 7y ago
    The talk hasn't been posted to youtube yet, but slides alone show some of the risks of eBPF. The speaker's previous talk at 35C3 went into how the verifier is arbitrary and not really designed to prevent attackers anyway. https://media.defcon.org/DEF%20CON%2027/DEF%20CON%2027%20presentations/DEFCON-27-Jeff-Dileo-Evil-eBPF-In-Depth.pdf https://media.defcon.org/DEF%20CON%2027/DEF%20CON%2027%20pre...
  - geofft 7y ago
    The verifier has had bugs in the past. Here's six of them from 2017: https://www.openwall.com/lists/oss-security/2017/12/23/2 https://www.openwall.com/lists/oss-security/2017/12/23/2 one of which was discovered to have an incomplete fix in 2018: https://crbug.com/project-zero/1686 https://crbug.com/project-zero/1686 https://medium.com/@tyanir/understanding-bpf-check-alu-op-vulnerability-cve-2017-16995-3eb3017f598a https://medium.com/@tyanir/understanding-bpf-check-alu-op-vu... has a longer description of one of them - namely, you can convince the verifier that a certain part of the program is dead code and therefore doesn't need to be verified when it does, and thereby get arbitrary unverified eBPF into the kernel.
- Klasiaster 7y ago
  Hi, I chose this way because it's a proof-of-concept. Normally this filtering expression would be stored in a BPF map but I didn't invest more time and wanted to keep it simple as an example for writing BPF programs for systemd services.
- azifali 7y ago
  There's a limited instruction set, no loops, built in verifier exists so that no anomalous code can be executed on the in kernel virtual machine. More so, this feature is in 4.X kernel and I think there's been no exploit discovered in the wild.
  - viraptor 7y ago
    > More so, this feature is in 4.X kernel and I think there's been no exploit discovered in the wild. There won't be an exploit for bpf. It's kind of a different layer and it's own system. "Exploit in bpf" is about the same level as "exploit in C". There's just no such general thing.
arianvanp 7y ago
This approach is very similar to what https://github.com/cilium/cilium https://github.com/cilium/cilium is doing for containers right? I wonder if it would be easy to reuse the battle-tested bpf programs that cilium provides and load them into systemd units. There is more crazy shit that we can do. Like set up entire service meshes with load balancers for your systemd units. Very neat.
- eloycoto 7y ago
  yes, it uses the same technology. Snabb I think that also has XDP/BPF support and you can do similar things using Lua
nailer 7y ago
If you're wondering what BPF is: https://linux-audit.com/bpfilter-next-generation-linux-firewall/ https://linux-audit.com/bpfilter-next-generation-linux-firew... and https://en.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter
- cgh 7y ago
  Many people are surprised to learn that Linux (and I guess some of the BSDs?) have a virtual machine that runs in the kernel and executes its own specific bytecode. LLVM has a BPF backend, as mentioned in this article.
  - bifrost 7y ago
    Pretty sure BPF originated in BSD land :) Prior to BPF there was "Enet Packet Filter", then Ultrix Packet Filter then something under SunOS before it became BPF. BPF was created in 1990 (at Berkeley) which was widely BSD oriented.
  - lelf 7y ago
    > and I guess some of the BSDs? BPF, BSD — notice the first letter is the same ;-) Also, many systems (some BSDs, SmartOS, macOS, Windows10, …) have an in-kernel VM for running dtrace bytecode.
ldng 7y ago
Maybe someone can enlighten me because I'm failing to see the relevance of SystemD here. So the idea is, instead of having a central firewall managing all the host rules, each service define it's own firewall policy ? How do I override a policy ? I maybe missing something but somehow I'm not sure it's the right place to do this. I'll end up joining the camp of SystemD does too much and breaks a lot of POSIX semantics making Linux systems hard to debug. Lately it's been getting more and more in my way. Things that I have problems with lately, DNS, cgroup and namespace. Every time I've lost a considerable amount of time because of poorly documented and mostly unexpected SystemD behavior. Color me annoyed. Edit: Hum, well, wasn't supposed to but it end up into a rant
- dharmab 7y ago
  This isn't a systemd feature. It's a Linux kernel feature (BPF) which is simply being called from a systemd exec line. This could also be implemented with sysvinit.
  - Klasiaster 7y ago
    The systemd feature is that in v243 you can now specify a BPF program for all sockets in the cgroup via `IPEgressFilterPath=/sys/fs/bpf/yourfilter` (same for `IPIngressFilterPath`).
  - marios 7y ago
    This is the same kind of reply I got when systemd implemented IP accouting in unit files. Lennart said systemd gets a lot of flak, but in the end it's only implementing kernel features. I proceeded to ask him: why is it necessary to have this ? Do you feel obligated to make every single feature available ? Parent comment raises a very valid question. How do you manage the firewall policy with this system ? Netfilter configuration is already a mess on Linux (iptables ? nftables ? iptables-persistent package ? netfilter-persistent package ? some custom shell script that calls individual iptables rules ? rules dynamically inserted by scripts ?) Each of these tools/methods has its flaws, but it becomes completely unmanageable if two are used at the same time. When this won't work as expected, how should a sysadmin handle the situation ? Where do you even start debugging ? Is he/she expected to inspect every single unit file in search of the one that is amiss? Can one get a list of all currently loaded rules ? (preferably with counters about matched packets and ideally with the possibility to log a packet matched by a rule) systemd making this 'easy' to use may be a bad idea. In my eyes, it's just giving users more rope to hang themselves with. The unit file format is often touted as an asset, because it's much simpler than the shell-goo you would find on most distributions (Debian and derivatives for example provided a skeleton file for people to write their own init.d services. Just the boilerplate was almost 100 lines of sh. Contrast with OpenBSD where most scripts to configure service startup are only a couple of lines). Having a key=value format was touted as a plug as it made things easier. Turns out that's not exactly true because some settings will have the expected effect only if you add enable a different setting at the same time. In my mind, this translates as an if/else which makes me think systemd.unit(5) format is not INI style configuration, but a small programming language masquerading as a configuration. Anyhow, this turned out longer than I expected. The fact that it would be possible to do with sysvinit does not mean it would be a good idea to do this sysvinit.
ausjke 7y ago
what's the user case for this and how does this complements iptables? why do I need this? BPF is very interesting, I remember one thing is that it's of very small size and has no loops, but I don't understand its use case for firewall yet.
- jimktrains2 7y ago
  My understanding is that it allows you to write a program to decide what to do with a packet and allow that program to run in kernel space for efficiency. The limitations of bpf are to protect the kernel from the non-kernel code running in its memory space and ring 0.
- Klasiaster 7y ago
  There is already support in systemd for IP address filtering, so that a service can or cannot communicate with certain IPs. Internally this uses BPF as well. The blogpost was about extending this to port filtering through a custom BPF program.
- arianvanp 7y ago
  This is a good talk about the how's and why's of BPF as a firewall: https://youtu.be/_Iq1xxNZOAo https://youtu.be/_Iq1xxNZOAo
justicezyx 7y ago
Isn't the BPF program has to be attached with root privilege? If so, the idea that to have per service is not enforceable anyway, right? As potential my filter can affect any other process running on the host.
cat199 7y ago
sooo.. tcpwrappers?

103 points by Klasiaster 7y ago

Compiling dynamically generated C programs on demand to provide packet filtering? It's clever but not in a good way. This is a really dangerous approach to solving this problem.

alexgartrell 7y ago
First, there's no need to do this "on demand." You can ship a .o around and load it with bpftool. My team does a version of this at Facebook (using libbpf directly) and it works really well. The big use case for on-demand programming is in tracing where you want to capture structs, but BTF and Compile-once, run-everywhere will mitigate this need [0] Second, I don't actually agree. JIT compilation is a generally accepted approach -- who cares what the IR is? And the bpf runtime is ULTRA constrained, so you can't slip a `system("rm -rf /")` in there (by a long shot). [0] https://www.kernel.org/doc/html/latest/bpf/btf.html https://www.kernel.org/doc/html/latest/bpf/btf.html
- jdc 7y ago
  I'm thoroughly skeptical that general acceptance means that JIT compilation doesn't increase the attack surface of the stack.
  - Dylan16807 7y ago
    You'd need an interpreter at least, to avoid some kind of hideously complex system. So it's a question of interpreter vs. JIT, and a JIT makes it more feasible to use an ultra-simple language with aggressive verification, without losing too much speed. Every feature has an attack surface, but you have to compare against the alternative, not the lack of feature. (This assumes that you can't force it to use a kernel interpreter despite the JIT existing. Otherwise perhaps that is the part that should be disabled.)
- ignoramous 7y ago
  Tangential: Does your team publish papers or write blog posts on the kind of engineering or research it does? I'd like to read it, or at least add it to my queue. Thanks.
  - jauer 7y ago
    There was a recent talk on tupperware that gets into some of what Alex's team does: https://engineering.fb.com/data-center-engineering/tupperware/ https://engineering.fb.com/data-center-engineering/tupperwar...
    - ignoramous 7y ago
      That was interesting. Thanks.
edoo 7y ago
How is it any different than letting a script run as root? This seems very similar to varnish cache which is amazing.
rustaceanunited 7y ago
Almost 2020 and we still have to deal with that legacy language.
justicezyx 7y ago
What are the dangers? AFAIK, BPF has in-kernel verifier on the safety of the BPF program, which is conservative and would reject safe program, let alone the real dangerous ones.
- dfay 7y ago
  The talk hasn't been posted to youtube yet, but slides alone show some of the risks of eBPF. The speaker's previous talk at 35C3 went into how the verifier is arbitrary and not really designed to prevent attackers anyway. https://media.defcon.org/DEF%20CON%2027/DEF%20CON%2027%20presentations/DEFCON-27-Jeff-Dileo-Evil-eBPF-In-Depth.pdf https://media.defcon.org/DEF%20CON%2027/DEF%20CON%2027%20pre...
- geofft 7y ago
  The verifier has had bugs in the past. Here's six of them from 2017: https://www.openwall.com/lists/oss-security/2017/12/23/2 https://www.openwall.com/lists/oss-security/2017/12/23/2 one of which was discovered to have an incomplete fix in 2018: https://crbug.com/project-zero/1686 https://crbug.com/project-zero/1686 https://medium.com/@tyanir/understanding-bpf-check-alu-op-vulnerability-cve-2017-16995-3eb3017f598a https://medium.com/@tyanir/understanding-bpf-check-alu-op-vu... has a longer description of one of them - namely, you can convince the verifier that a certain part of the program is dead code and therefore doesn't need to be verified when it does, and thereby get arbitrary unverified eBPF into the kernel.
Klasiaster 7y ago
Hi, I chose this way because it's a proof-of-concept. Normally this filtering expression would be stored in a BPF map but I didn't invest more time and wanted to keep it simple as an example for writing BPF programs for systemd services.
azifali 7y ago
There's a limited instruction set, no loops, built in verifier exists so that no anomalous code can be executed on the in kernel virtual machine. More so, this feature is in 4.X kernel and I think there's been no exploit discovered in the wild.
- viraptor 7y ago
  > More so, this feature is in 4.X kernel and I think there's been no exploit discovered in the wild. There won't be an exploit for bpf. It's kind of a different layer and it's own system. "Exploit in bpf" is about the same level as "exploit in C". There's just no such general thing.

arianvanp 7y ago

This approach is very similar to what https://github.com/cilium/cilium https://github.com/cilium/cilium is doing for containers right? I wonder if it would be easy to reuse the battle-tested bpf programs that cilium provides and load them into systemd units. There is more crazy shit that we can do. Like set up entire service meshes with load balancers for your systemd units. Very neat.

eloycoto 7y ago
yes, it uses the same technology. Snabb I think that also has XDP/BPF support and you can do similar things using Lua

nailer 7y ago

If you're wondering what BPF is: https://linux-audit.com/bpfilter-next-generation-linux-firewall/ https://linux-audit.com/bpfilter-next-generation-linux-firew... and https://en.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter

cgh 7y ago
Many people are surprised to learn that Linux (and I guess some of the BSDs?) have a virtual machine that runs in the kernel and executes its own specific bytecode. LLVM has a BPF backend, as mentioned in this article.
- bifrost 7y ago
  Pretty sure BPF originated in BSD land :) Prior to BPF there was "Enet Packet Filter", then Ultrix Packet Filter then something under SunOS before it became BPF. BPF was created in 1990 (at Berkeley) which was widely BSD oriented.
- lelf 7y ago
  > and I guess some of the BSDs? BPF, BSD — notice the first letter is the same ;-) Also, many systems (some BSDs, SmartOS, macOS, Windows10, …) have an in-kernel VM for running dtrace bytecode.

ldng 7y ago

Maybe someone can enlighten me because I'm failing to see the relevance of SystemD here. So the idea is, instead of having a central firewall managing all the host rules, each service define it's own firewall policy ? How do I override a policy ? I maybe missing something but somehow I'm not sure it's the right place to do this. I'll end up joining the camp of SystemD does too much and breaks a lot of POSIX semantics making Linux systems hard to debug. Lately it's been getting more and more in my way. Things that I have problems with lately, DNS, cgroup and namespace. Every time I've lost a considerable amount of time because of poorly documented and mostly unexpected SystemD behavior. Color me annoyed. Edit: Hum, well, wasn't supposed to but it end up into a rant

dharmab 7y ago
This isn't a systemd feature. It's a Linux kernel feature (BPF) which is simply being called from a systemd exec line. This could also be implemented with sysvinit.
- Klasiaster 7y ago
  The systemd feature is that in v243 you can now specify a BPF program for all sockets in the cgroup via `IPEgressFilterPath=/sys/fs/bpf/yourfilter` (same for `IPIngressFilterPath`).
- marios 7y ago
  This is the same kind of reply I got when systemd implemented IP accouting in unit files. Lennart said systemd gets a lot of flak, but in the end it's only implementing kernel features. I proceeded to ask him: why is it necessary to have this ? Do you feel obligated to make every single feature available ? Parent comment raises a very valid question. How do you manage the firewall policy with this system ? Netfilter configuration is already a mess on Linux (iptables ? nftables ? iptables-persistent package ? netfilter-persistent package ? some custom shell script that calls individual iptables rules ? rules dynamically inserted by scripts ?) Each of these tools/methods has its flaws, but it becomes completely unmanageable if two are used at the same time. When this won't work as expected, how should a sysadmin handle the situation ? Where do you even start debugging ? Is he/she expected to inspect every single unit file in search of the one that is amiss? Can one get a list of all currently loaded rules ? (preferably with counters about matched packets and ideally with the possibility to log a packet matched by a rule) systemd making this 'easy' to use may be a bad idea. In my eyes, it's just giving users more rope to hang themselves with. The unit file format is often touted as an asset, because it's much simpler than the shell-goo you would find on most distributions (Debian and derivatives for example provided a skeleton file for people to write their own init.d services. Just the boilerplate was almost 100 lines of sh. Contrast with OpenBSD where most scripts to configure service startup are only a couple of lines). Having a key=value format was touted as a plug as it made things easier. Turns out that's not exactly true because some settings will have the expected effect only if you add enable a different setting at the same time. In my mind, this translates as an if/else which makes me think systemd.unit(5) format is not INI style configuration, but a small programming language masquerading as a configuration. Anyhow, this turned out longer than I expected. The fact that it would be possible to do with sysvinit does not mean it would be a good idea to do this sysvinit.

ausjke 7y ago

what's the user case for this and how does this complements iptables? why do I need this? BPF is very interesting, I remember one thing is that it's of very small size and has no loops, but I don't understand its use case for firewall yet.

jimktrains2 7y ago
My understanding is that it allows you to write a program to decide what to do with a packet and allow that program to run in kernel space for efficiency. The limitations of bpf are to protect the kernel from the non-kernel code running in its memory space and ring 0.
Klasiaster 7y ago
There is already support in systemd for IP address filtering, so that a service can or cannot communicate with certain IPs. Internally this uses BPF as well. The blogpost was about extending this to port filtering through a custom BPF program.
arianvanp 7y ago
This is a good talk about the how's and why's of BPF as a firewall: https://youtu.be/_Iq1xxNZOAo https://youtu.be/_Iq1xxNZOAo

justicezyx 7y ago

Isn't the BPF program has to be attached with root privilege? If so, the idea that to have per service is not enforceable anyway, right? As potential my filter can affect any other process running on the host.

cat199 7y ago

sooo.. tcpwrappers?