8 ms·

Systemd service sandboxing and security hardening (2020)

293 points by capableweb 5y ago

max002 5y ago
Great article :) thank you!
- deleted 5y ago
  [deleted]
kenniskrag 5y ago
if you want to test these settings I can recommend `sudo systemd-run -p "DynamicUser=yes" -p "ProtectSystem=yes" -p "ProtectHome=yes" --shell` but be in a readable directory like /tmp or you receive an error.
- 5e92cb50239222b 5y ago
  This is a very handy command in day-to-day work, actually. For example, I use to limit the total amount of memory available to an application, including page cache: $ systemd-run --user --scope --property=MemoryHigh=1G qbittorrent It works just as you'd expect — if qbittorrent's working set goes above 1024 MiB, it pushes the least recently used page out of the page cache. Doesn't really have any effects on upload or download speeds, while helping to keep more useful data in memory. Many isolation flags are not available in `systemd-run --user`, though, so if you'd like to have some protection you either have to combine `sudo systemd-run` with `su -c`, or wrap the command in firejail. https://github.com/netblue30/firejail/ https://github.com/netblue30/firejail/
  - wmanley 5y ago
    I have a bash alias for `make` and `ninja` to do something similar. Just having all the spawned processes in a cgroup helps with system interactivity while building. This works because the kernel will then schedule the whole build as a single unit against the other work on the system, rather than scheduling each process that the build spawns against every other process that I'm running.
  - t0astbread 5y ago
    Interesting, a few months ago I tried using systemd-run to implement unprivileged memory limits for a process and I'm pretty sure it didn't work with the user manager. Is this a recent addition? (I'm not sure what version of systemd I had at the time.)
- pram 5y ago
  Ooh, is this a good way to sandbox execs like ImageMagick or stuff like that?
  - 5e92cb50239222b 5y ago
    Use firejail, it's a "one click" solution with prepackaged profiles. https://github.com/netblue30/firejail/ https://github.com/netblue30/firejail/ It uses the same kernel knobs as systemd does, but is more user-friendly and has more features. I use it for every application that handles data received from other machines: books, images, documents, whatever.
    - YorickPeterse 5y ago
      You can also use Bubblewrap, but getting it up and running requires a lot more fiddling around. For example, this is what I use to isolate Zoom from the rest of my system: https://gitlab.com/yorickpeterse/dotfiles/-/blob/0a0492c78b6fb7629fd9b482353e36798c7bf31a/dotfiles/bin/zoom-jail https://gitlab.com/yorickpeterse/dotfiles/-/blob/0a0492c78b6... In my case I'm using Bubblewrap because Firejail was only used for Zoom, and this felt a bit of a waste considering Bubblewrap was already installed.
the8472 5y ago
Alas, no whitelisting option. A service should start in an empty filesystem root without network access - and if we had something as convenient as pledge() also without any allowed syscalls - and then you could only add what is needed. firejail does this a bit better but it also started out with a blacklist approach and it's more geared towards desktop application use, not system services.
- Icathian 5y ago
  One of my favorite podcasts, Risky Business[0] regularly plugs Airlock[1]. They seem like they might be the one out front, at least as a paid service. [0] https://risky.biz/netcasts/risky-business/ https://risky.biz/netcasts/risky-business/ [1] https://www.airlockdigital.com/ https://www.airlockdigital.com/
- 5e92cb50239222b 5y ago
  What's the problem with firejail? Start with an empty profile, blacklist everything, and whitelist only the stuff you need. It works just fine for server applications, and unlike systemd isolation flags you can setup a proper separate firewall with the `netfilter` option.
  - the8472 5y ago
    > blacklist everything, That isn't a whitelist approach.
    - Someone 5y ago
      > blacklist everything, and whitelist only the stuff you need That _is_ a whitelist approach.
      - the8472 5y ago
        Not really. For example if you have a construct like "blacklist *" and that wildcard is evaluated at construction time on some overlay filesystem then additional entries may sneak in later from the lower over the overlay because the wildcard expansion doesn't get updated. On the other hand if you start with a blank slate filesystem root and only bind exactly the whitelisted paths then there is nothing to leak through. There are other ways in which blacklist-all can fail to be equivalent to whitelisting.
5e92cb50239222b 5y ago
This is a pretty lax policy IMHO, you can go much farther. These days I usually start with this, it's much more strict: https://news.ycombinator.com/item?id=29976096 https://news.ycombinator.com/item?id=29976096 Or simply follow whatever `systemd-analyze security` recommends, just make sure you run it on a system with recent systemd.
- westurner 5y ago
  Which distro has the best out-of-the-box output for:? systemd-analyze security Is there a tool like `audit2allow` for systemd units? selinux/python/audit2allow/audit2allow: https://github.com/SELinuxProject/selinux/blob/master/python/audit2allow/audit2allow https://github.com/SELinuxProject/selinux/blob/master/python... https://stopdisablingselinux.com/ https://stopdisablingselinux.com/
  - 5e92cb50239222b 5y ago
    > Which distro has the best out-of-the-box output I haven't seen any difference between distributions with the same systemd version. Anything with a recent one should do fine. More recent than RHEL8, mind you (which is on systemd 239): for example, a syscall allow/deny analysis is buggy there and asks you to enable some protections, and then disable them. The same unit is analyzed correctly on my desktop with v250 (I use the popular rolling release distribution). I haven't seen anything like audit2allow. It's probably not especially necessary because of the difference in philosophies: SELinux is deny by default, while in systemd you're playing whack-a-mole anyway, and are expected to add directives one by one until the application stops working. Unit logs usually make it obvious if something was denied.
    - Arnavion 5y ago
      The usual way I've seen (and do myself) is to just let the process be killed and have its coredump taken, then `coredumpctl gdb $process_name -A '-ex "print $rax" -ex "quit"'` to get the syscall number, then check `systemd-analyze syscall-filter` for whether I want to allow just that one syscall or the whole group it's in.
HowardStark 5y ago
Is there any advice for working with older systemd versions? Right off the bat, systemd 237 is out because there is no security feature for that version of systemd-analyze.
- 5e92cb50239222b 5y ago
  Use the same config you'd use for the latest systemd version. It will ignore flags it doesn't know (and warn you in unit logs).
bloopernova 5y ago
Not meant to be a snarky comment, but a serious question: how does this differ from SELinux?
- staticassertion 5y ago
  It's vastly simpler, for one thing. SELinux is basically a weird DSL/ programming language for describing system interactions whereas systemd is providing a very basic interface for common restrictions. I would pretty much never ask a human being to write SELinux policies unless that was explicitly part of their job whereas I can pretty much point any developer to what systemd is providing and they'll be able to work with it.
- tyingq 5y ago
  It seems to be using mostly the linux capabilities: https://man7.org/linux/man-pages/man7/capabilities.7.html https://man7.org/linux/man-pages/man7/capabilities.7.html So the overlap choice seems to be more around SELinux versus Capabilities. Where SELinux is more fine-grained and tunable, but more complicated also.
  - aseipp 5y ago
    It's not just Linux capabilities; on their own Linux capabilities actually suck majorly and are very limited (AKA "crapabilities"). But systemd also makes extensive usage of cgroups and namespacing facilities to back it up e.g. preventing runaway memory/CPU quotas and stopping applications from accessing paths they shouldn't, restricting network access, stuff like that. Some of this overlaps with SELinux (e.g. restricting file access) but the mechanism is fairly different. The overlap/comparison between capabilities, systemds features, and selinux features isn't really well defined in any meaningful way IMO. It's really like 5 different features being used in various ways.
    - PeterWhittaker 5y ago
      I’m curious what you mean by SELinux features not being well-defined? While poorly documented, they are extraordinarily precisely defined, allowing fine-grained control of pretty much everything, all enforced by the kernel with no workarounds, at least in enforcing mode.
loudtieblahblah 5y ago
Does your box still touch local dns before connecting to VPN? No? Then anything with systemd and security can stuff it.
- getcrunk 5y ago
  What?
ape4 5y ago
Apache needs to start as `root` but then drops to an non-privileged user. systemd's `User=<user>` can't really express that. Perhaps an option that says a unit needs to be root until the first fork when it has to be a specified user. `ForkUser=apache`
- candiddevmike 5y ago
  It only needs to root to bind to privileged ports I believe. You should be able to use a non-root user and give it CAP_NET_BIND_SERVICE: [Service] AmbientCapabilities=CAP_NET_BIND_SERVICE
  - ape4 5y ago
    Cool! But then I suppose the forked processes could then bind to a low numbered port - something they can't do now. So Apache would have to make sure to revoke that capability when forking.
    - 5e92cb50239222b 5y ago
      You could combine it with something like this SocketBindDeny=any SocketBindAllow=tcp:80 SocketBindAllow=tcp:443 These ports should be denied by the kernel because they're already taken by httpd, and all other will be denied by bpf filters installed by systemd. It feels like plugging holes in a dam, but that's what you do with popular operating systems.
- 5e92cb50239222b 5y ago
  I don't know about httpd specifically, but many applications want root only to be able to bind to a privileged port (like :80). This can be circumvented in one of a few ways: 1. add this to .service AmbientCapabilities=CAP_NET_BIND_SERVICE 2. or listen on :8080 and use NAT: iptables -t nat -I OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080 3. or make the port unprivileged sysctl -w net.ipv4.ip_unprivileged_port_start=80 It may work for httpd too, I haven't tested it.
  - 5y ago
a-dub 5y ago
can you limit outbound network access to specified masks/ports/devices on a per-service level?
deleted 5y ago
[deleted]
egberts1 5y ago
Whose gonna write THE holy-grail of analyzer of many executables to determine what Linux capabilities, cgroups, and syscalls are just being referenced? Caveat: it has to dig into ALL the linked libraries as well.

293 points by capableweb 5y ago

max002 5y ago

Great article :) thank you!

deleted 5y ago
[deleted]

kenniskrag 5y ago

if you want to test these settings I can recommend `sudo systemd-run -p "DynamicUser=yes" -p "ProtectSystem=yes" -p "ProtectHome=yes" --shell` but be in a readable directory like /tmp or you receive an error.

5e92cb50239222b 5y ago
This is a very handy command in day-to-day work, actually. For example, I use to limit the total amount of memory available to an application, including page cache: $ systemd-run --user --scope --property=MemoryHigh=1G qbittorrent It works just as you'd expect — if qbittorrent's working set goes above 1024 MiB, it pushes the least recently used page out of the page cache. Doesn't really have any effects on upload or download speeds, while helping to keep more useful data in memory. Many isolation flags are not available in `systemd-run --user`, though, so if you'd like to have some protection you either have to combine `sudo systemd-run` with `su -c`, or wrap the command in firejail. https://github.com/netblue30/firejail/ https://github.com/netblue30/firejail/
- wmanley 5y ago
  I have a bash alias for `make` and `ninja` to do something similar. Just having all the spawned processes in a cgroup helps with system interactivity while building. This works because the kernel will then schedule the whole build as a single unit against the other work on the system, rather than scheduling each process that the build spawns against every other process that I'm running.
- t0astbread 5y ago
  Interesting, a few months ago I tried using systemd-run to implement unprivileged memory limits for a process and I'm pretty sure it didn't work with the user manager. Is this a recent addition? (I'm not sure what version of systemd I had at the time.)
pram 5y ago
Ooh, is this a good way to sandbox execs like ImageMagick or stuff like that?
- 5e92cb50239222b 5y ago
  Use firejail, it's a "one click" solution with prepackaged profiles. https://github.com/netblue30/firejail/ https://github.com/netblue30/firejail/ It uses the same kernel knobs as systemd does, but is more user-friendly and has more features. I use it for every application that handles data received from other machines: books, images, documents, whatever.
  - YorickPeterse 5y ago
    You can also use Bubblewrap, but getting it up and running requires a lot more fiddling around. For example, this is what I use to isolate Zoom from the rest of my system: https://gitlab.com/yorickpeterse/dotfiles/-/blob/0a0492c78b6fb7629fd9b482353e36798c7bf31a/dotfiles/bin/zoom-jail https://gitlab.com/yorickpeterse/dotfiles/-/blob/0a0492c78b6... In my case I'm using Bubblewrap because Firejail was only used for Zoom, and this felt a bit of a waste considering Bubblewrap was already installed.

the8472 5y ago

Alas, no whitelisting option. A service should start in an empty filesystem root without network access - and if we had something as convenient as pledge() also without any allowed syscalls - and then you could only add what is needed. firejail does this a bit better but it also started out with a blacklist approach and it's more geared towards desktop application use, not system services.

Icathian 5y ago
One of my favorite podcasts, Risky Business[0] regularly plugs Airlock[1]. They seem like they might be the one out front, at least as a paid service. [0] https://risky.biz/netcasts/risky-business/ https://risky.biz/netcasts/risky-business/ [1] https://www.airlockdigital.com/ https://www.airlockdigital.com/
5e92cb50239222b 5y ago
What's the problem with firejail? Start with an empty profile, blacklist everything, and whitelist only the stuff you need. It works just fine for server applications, and unlike systemd isolation flags you can setup a proper separate firewall with the `netfilter` option.
- the8472 5y ago
  > blacklist everything, That isn't a whitelist approach.
  - Someone 5y ago
    > blacklist everything, and whitelist only the stuff you need That _is_ a whitelist approach.
    - the8472 5y ago
      Not really. For example if you have a construct like "blacklist *" and that wildcard is evaluated at construction time on some overlay filesystem then additional entries may sneak in later from the lower over the overlay because the wildcard expansion doesn't get updated. On the other hand if you start with a blank slate filesystem root and only bind exactly the whitelisted paths then there is nothing to leak through. There are other ways in which blacklist-all can fail to be equivalent to whitelisting.

5e92cb50239222b 5y ago

This is a pretty lax policy IMHO, you can go much farther. These days I usually start with this, it's much more strict: https://news.ycombinator.com/item?id=29976096 https://news.ycombinator.com/item?id=29976096 Or simply follow whatever `systemd-analyze security` recommends, just make sure you run it on a system with recent systemd.

westurner 5y ago
Which distro has the best out-of-the-box output for:? systemd-analyze security Is there a tool like `audit2allow` for systemd units? selinux/python/audit2allow/audit2allow: https://github.com/SELinuxProject/selinux/blob/master/python/audit2allow/audit2allow https://github.com/SELinuxProject/selinux/blob/master/python... https://stopdisablingselinux.com/ https://stopdisablingselinux.com/
- 5e92cb50239222b 5y ago
  > Which distro has the best out-of-the-box output I haven't seen any difference between distributions with the same systemd version. Anything with a recent one should do fine. More recent than RHEL8, mind you (which is on systemd 239): for example, a syscall allow/deny analysis is buggy there and asks you to enable some protections, and then disable them. The same unit is analyzed correctly on my desktop with v250 (I use the popular rolling release distribution). I haven't seen anything like audit2allow. It's probably not especially necessary because of the difference in philosophies: SELinux is deny by default, while in systemd you're playing whack-a-mole anyway, and are expected to add directives one by one until the application stops working. Unit logs usually make it obvious if something was denied.
  - Arnavion 5y ago
    The usual way I've seen (and do myself) is to just let the process be killed and have its coredump taken, then `coredumpctl gdb $process_name -A '-ex "print $rax" -ex "quit"'` to get the syscall number, then check `systemd-analyze syscall-filter` for whether I want to allow just that one syscall or the whole group it's in.

HowardStark 5y ago

Is there any advice for working with older systemd versions? Right off the bat, systemd 237 is out because there is no security feature for that version of systemd-analyze.

5e92cb50239222b 5y ago
Use the same config you'd use for the latest systemd version. It will ignore flags it doesn't know (and warn you in unit logs).

bloopernova 5y ago

Not meant to be a snarky comment, but a serious question: how does this differ from SELinux?

staticassertion 5y ago
It's vastly simpler, for one thing. SELinux is basically a weird DSL/ programming language for describing system interactions whereas systemd is providing a very basic interface for common restrictions. I would pretty much never ask a human being to write SELinux policies unless that was explicitly part of their job whereas I can pretty much point any developer to what systemd is providing and they'll be able to work with it.
tyingq 5y ago
It seems to be using mostly the linux capabilities: https://man7.org/linux/man-pages/man7/capabilities.7.html https://man7.org/linux/man-pages/man7/capabilities.7.html So the overlap choice seems to be more around SELinux versus Capabilities. Where SELinux is more fine-grained and tunable, but more complicated also.
- aseipp 5y ago
  It's not just Linux capabilities; on their own Linux capabilities actually suck majorly and are very limited (AKA "crapabilities"). But systemd also makes extensive usage of cgroups and namespacing facilities to back it up e.g. preventing runaway memory/CPU quotas and stopping applications from accessing paths they shouldn't, restricting network access, stuff like that. Some of this overlaps with SELinux (e.g. restricting file access) but the mechanism is fairly different. The overlap/comparison between capabilities, systemds features, and selinux features isn't really well defined in any meaningful way IMO. It's really like 5 different features being used in various ways.
  - PeterWhittaker 5y ago
    I’m curious what you mean by SELinux features not being well-defined? While poorly documented, they are extraordinarily precisely defined, allowing fine-grained control of pretty much everything, all enforced by the kernel with no workarounds, at least in enforcing mode.

loudtieblahblah 5y ago

Does your box still touch local dns before connecting to VPN? No? Then anything with systemd and security can stuff it.

getcrunk 5y ago
What?

ape4 5y ago

Apache needs to start as `root` but then drops to an non-privileged user. systemd's `User=<user>` can't really express that. Perhaps an option that says a unit needs to be root until the first fork when it has to be a specified user. `ForkUser=apache`

candiddevmike 5y ago
It only needs to root to bind to privileged ports I believe. You should be able to use a non-root user and give it CAP_NET_BIND_SERVICE: [Service] AmbientCapabilities=CAP_NET_BIND_SERVICE
- ape4 5y ago
  Cool! But then I suppose the forked processes could then bind to a low numbered port - something they can't do now. So Apache would have to make sure to revoke that capability when forking.
  - 5e92cb50239222b 5y ago
    You could combine it with something like this SocketBindDeny=any SocketBindAllow=tcp:80 SocketBindAllow=tcp:443 These ports should be denied by the kernel because they're already taken by httpd, and all other will be denied by bpf filters installed by systemd. It feels like plugging holes in a dam, but that's what you do with popular operating systems.
5e92cb50239222b 5y ago
I don't know about httpd specifically, but many applications want root only to be able to bind to a privileged port (like :80). This can be circumvented in one of a few ways: 1. add this to .service AmbientCapabilities=CAP_NET_BIND_SERVICE 2. or listen on :8080 and use NAT: iptables -t nat -I OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080 3. or make the port unprivileged sysctl -w net.ipv4.ip_unprivileged_port_start=80 It may work for httpd too, I haven't tested it.
- 5y ago

a-dub 5y ago

can you limit outbound network access to specified masks/ports/devices on a per-service level?

deleted 5y ago

[deleted]

egberts1 5y ago

Whose gonna write THE holy-grail of analyzer of many executables to determine what Linux capabilities, cgroups, and syscalls are just being referenced? Caveat: it has to dig into ALL the linked libraries as well.