6 ms·

How We Built SELinux Support for Kubernetes

70 points by twakefield 6y ago

rwmj 6y ago
I've written a bunch of SELinux policy, also without being especially expert. The tooling is definitely getting better, but still there are many annoying things from the obscure file layout, the giant all-in-one everything-is-a-global policy[1], to the small annoyances like m4 in the policy files. Of course you need to buy into the whole premise of SELinux - that in addition to simple Unix permissions you can use mandatory access control to fine-grain control everything that a process can do and get access to. Obviously this is a giant boil-the-ocean project that has consumed many good people for decades. For SVirt (secure virtualization) this was pretty successful, allowing many qemu instances to run on a single server, and confining them with SELinux so each can only access its own files even when they are all running as the same Unix user. The linked article is a good intro. [1] But they're fixing this, yay! I can't find a link now but there's a project to decentralise SELinux policy. Perhaps it's still an internal-only Red Hat project.
- als0 6y ago
  > [1] But they're fixing this, yay! I can't find a link now but there's a project to decentralise SELinux policy. Perhaps it's still an internal-only Red Hat project. +1 this is sorely needed and I wasn't sure if the design would ever evolve to include this
- yrro 6y ago
  Worth mentioning that SVirt is also used for running containers on RHEL/Fedora/CentOS etc.
  - totony 6y ago
    I don't think that's the case, svirt is specific to libvirt + selinux-virt, whereas containers use selinux-container which are integrated into docker & al. Same principle tho
    - yrro 6y ago
      Originally all the domains had 'svirt' in the name: https://www.projectatomic.io/blog/2016/03/dwalsh_selinux_containers/ https://www.projectatomic.io/blog/2016/03/dwalsh_selinux_con... These days containers have their own domains: https://www.mankier.com/8/container_selinux https://www.mankier.com/8/container_selinux As you say, the principle is the same. [edit] container_t is (or was?) an alias for svirt_lxc_net_t: https://danwalsh.livejournal.com/79191.html https://danwalsh.livejournal.com/79191.html
kapilvt 6y ago
Tools write selinux policies, humans write apparmor policies .
- VWWHFSfQ 6y ago
  apparmor is dead at this point right. selinux is the standard
  - nisa 6y ago
    could you explain why you think so? We are are Debian/Ubuntu shop and it looks like it's all AppArmor.
    - totony 6y ago
      AppArmor is used by debian/ubuntu, while selinux is used by Redhat and Android. Don't think any of them is more dead than the other. The refpol used by redhat isn't very restrictive/complete and no one except for big corps are using it. Apparmor is barely looked at too in my experience.
    - TimWolla 6y ago
      In fact AppArmor was only "recently" enabled by default in Debian starting with Debian Buster: https://www.debian.org/releases/stable/amd64/release-notes/ch-whats-new.en.html#apparmor https://www.debian.org/releases/stable/amd64/release-notes/c...
    - kapilvt 6y ago
      apparmor is also the default linux security module for suse
- LinuxBender 6y ago
  Both tools can be managed by both tools and humans. SELinux is for Redhat distros and operates on inodes. AppArmor is for the Ubuntu distros and operates on files/dirs. Each have pros and cons, too long to cover here at HN.
marbu 6y ago
While the article shows how they build SELinux policy from scrath, they also mention udica tool[1] which generates SELinux profile based on container inspection: # podman inspect my_container_id | udica my_container I wonder how much would the result be different in their case or how much time would they save by having the first version of the policy being generated this way instead of writing it from scratch. [1] https://github.com/containers/udica https://github.com/containers/udica
lima 6y ago
What's the point of SELinux support in a k8s environment, where services are already sandboxed by the container runtime? It's unsafe to run untrusted workloads in the host kernel using only namespaces - SELinux does not change this, no matter how much Red Hat wants it to be true. The kernel has a massive attack surface. For untrusted workloads, you need virtualization like gVisor or Kata/Firecracker which isolates workloads from the host kernel. SELinux can mitigate some logic bugs in the container runtime, yes, but it doesn't make your containers secure. Also, the title of this submission on HN is inaccurate - Gravitational did not build SELinux support for k8s. Red Hat did, a long time ago. They added support for SELinux to their own product, Gravity.
- blitmap 6y ago
  My naive understanding is that containers do not provide complete security. Also, I could see giving a user shell access within a container and still restricting what they do with SELinux.
- VWWHFSfQ 6y ago
  Defense in depth. Running untrusted code in a kubernetes cluster is risky even with the sandbox. SElinux adds another layer of protection
  - Spivak 6y ago
    I would go further and say that SELinux (and AppArmor) aren't defense in depth but actually a necessary part of the tooling to implement sandboxing of containers.
- cpuguy83 6y ago
  SELinux would have prevented a number of container breakout issues. That said, as someone one time worked on an SELinux issue in Docker and then became the defacto "expert"... it's a terrible mess, often coming with incompatible policy changes tied to kernel changes and policies which are stacked on top of each other like a house of cards that are difficult to tease apart for other distros.
- 6y ago
tedk-42 6y ago
SELinux was a good way of securing linux machines with many services sharing things like file systems and binaries. k8s solves this in a different way. You can write applications that are completely stateless when spun up on a k8s node and can be limited to a read only file system. Your container can be a scratch container with no other executables binaries. I don't know what their product is, but i seriously doubt mixing these two concepts is the right thing to do.
- rwmj 6y ago
  I think you need to look at how it really works under the hood if you believe that. RHEL runs containers with a few layers around them, one of which is SELinux (example of an exploit being blocked: https://www.redhat.com/en/blog/latest-container-exploit-runc-can-be-blocked-selinux https://www.redhat.com/en/blog/latest-container-exploit-runc...)
- kevin_nisbet 6y ago
  I'm not sure it's fair to say the advantages of selinux are solely around sharing services and filesystems on a host. I believe selinux also allows a more expressive policy on how a running piece of software can interact with the kernel than say seccomp can, which is probably what you're container runtime is using to limit kernel access. And unless you're configuring seccomp, you're probably picking up some default policy designed to work on as many systems as possible, which sort of violated least privilege. I'm also not sure I would consider this a totally solved problem within kubernetes. In supporting gravity, our kubernetes distribution, we used to hard-disable all privileged containers. This become a common point of friction, since many pieces of common kubernetes software seemed to just use privileged containers instead of properly listing and setting needed capabilities. Although this is a separate issue from selinux, I'm just pointing out that many kubernetes clusters may not be as isolated between running processes as expected. And to answer what the product is, gravity is a kubernetes distribution, that in this use case allows for packaging a kubernetes based application together with the runtime, for a unified installer for the application + runtime. Think something like packaging up a SAAS application, and being able to install a copy within a banks network who doesn't want their systems talking to the internet while abstracting away the kubernetes portions. The problem we were running into, is that end user would keep trying to install the application onto selinux enabled hosts, which would break the install or runtime. And whether you believe in selinux or not, it causes significant friction to try and tell your customers we got this, just turn off that security feature in production.
ape4 6y ago
NSA made SELinux. My theory is that they made it so hard to use so nobody would use it /s
- LinuxBender 6y ago
  Redhat helped the NSA make SELinux substantially easier for more people to use. Disclaimer: It's still not super easy to use. The implementation at the NSA did not have the concept of "unconfined" anything and they use MLS policies. At least, that was the intent. This may have changed with time and perhaps by now they are using Redhat's version of SELinux.
heywire 6y ago
An off-topic but somewhat related question. What is a good way to whitelist which applications can access the network on my desktop linux system? (Currently running Arch, but I distro-hop quite often). Years ago when I had a MacBook, I used a program called Little Snitch. I see that there is a similar application for Linux called OpenSnitch, but I’m hoping there is a better way to utilize the built in features of Linux rather than introducing a 3rd party kernel module. Are SELinux / AppArmor the answer? Cgroups? Something else?
- totony 6y ago
  Selinux can add labels to network packets which can be matched with nftables & al. Its a bit complicated tho. Since you need to add and create labels. You can also match by owner in iptables. So you can have a network restricted user and sudo to it.

70 points by twakefield 6y ago

rwmj 6y ago

I've written a bunch of SELinux policy, also without being especially expert. The tooling is definitely getting better, but still there are many annoying things from the obscure file layout, the giant all-in-one everything-is-a-global policy[1], to the small annoyances like m4 in the policy files. Of course you need to buy into the whole premise of SELinux - that in addition to simple Unix permissions you can use mandatory access control to fine-grain control everything that a process can do and get access to. Obviously this is a giant boil-the-ocean project that has consumed many good people for decades. For SVirt (secure virtualization) this was pretty successful, allowing many qemu instances to run on a single server, and confining them with SELinux so each can only access its own files even when they are all running as the same Unix user. The linked article is a good intro. [1] But they're fixing this, yay! I can't find a link now but there's a project to decentralise SELinux policy. Perhaps it's still an internal-only Red Hat project.

als0 6y ago
> [1] But they're fixing this, yay! I can't find a link now but there's a project to decentralise SELinux policy. Perhaps it's still an internal-only Red Hat project. +1 this is sorely needed and I wasn't sure if the design would ever evolve to include this
yrro 6y ago
Worth mentioning that SVirt is also used for running containers on RHEL/Fedora/CentOS etc.
- totony 6y ago
  I don't think that's the case, svirt is specific to libvirt + selinux-virt, whereas containers use selinux-container which are integrated into docker & al. Same principle tho
  - yrro 6y ago
    Originally all the domains had 'svirt' in the name: https://www.projectatomic.io/blog/2016/03/dwalsh_selinux_containers/ https://www.projectatomic.io/blog/2016/03/dwalsh_selinux_con... These days containers have their own domains: https://www.mankier.com/8/container_selinux https://www.mankier.com/8/container_selinux As you say, the principle is the same. [edit] container_t is (or was?) an alias for svirt_lxc_net_t: https://danwalsh.livejournal.com/79191.html https://danwalsh.livejournal.com/79191.html

kapilvt 6y ago

Tools write selinux policies, humans write apparmor policies .

VWWHFSfQ 6y ago
apparmor is dead at this point right. selinux is the standard
- nisa 6y ago
  could you explain why you think so? We are are Debian/Ubuntu shop and it looks like it's all AppArmor.
  - totony 6y ago
    AppArmor is used by debian/ubuntu, while selinux is used by Redhat and Android. Don't think any of them is more dead than the other. The refpol used by redhat isn't very restrictive/complete and no one except for big corps are using it. Apparmor is barely looked at too in my experience.
  - TimWolla 6y ago
    In fact AppArmor was only "recently" enabled by default in Debian starting with Debian Buster: https://www.debian.org/releases/stable/amd64/release-notes/ch-whats-new.en.html#apparmor https://www.debian.org/releases/stable/amd64/release-notes/c...
  - kapilvt 6y ago
    apparmor is also the default linux security module for suse
LinuxBender 6y ago
Both tools can be managed by both tools and humans. SELinux is for Redhat distros and operates on inodes. AppArmor is for the Ubuntu distros and operates on files/dirs. Each have pros and cons, too long to cover here at HN.

marbu 6y ago

While the article shows how they build SELinux policy from scrath, they also mention udica tool[1] which generates SELinux profile based on container inspection: # podman inspect my_container_id | udica my_container I wonder how much would the result be different in their case or how much time would they save by having the first version of the policy being generated this way instead of writing it from scratch. [1] https://github.com/containers/udica https://github.com/containers/udica

lima 6y ago

What's the point of SELinux support in a k8s environment, where services are already sandboxed by the container runtime? It's unsafe to run untrusted workloads in the host kernel using only namespaces - SELinux does not change this, no matter how much Red Hat wants it to be true. The kernel has a massive attack surface. For untrusted workloads, you need virtualization like gVisor or Kata/Firecracker which isolates workloads from the host kernel. SELinux can mitigate some logic bugs in the container runtime, yes, but it doesn't make your containers secure. Also, the title of this submission on HN is inaccurate - Gravitational did not build SELinux support for k8s. Red Hat did, a long time ago. They added support for SELinux to their own product, Gravity.

blitmap 6y ago
My naive understanding is that containers do not provide complete security. Also, I could see giving a user shell access within a container and still restricting what they do with SELinux.
VWWHFSfQ 6y ago
Defense in depth. Running untrusted code in a kubernetes cluster is risky even with the sandbox. SElinux adds another layer of protection
- Spivak 6y ago
  I would go further and say that SELinux (and AppArmor) aren't defense in depth but actually a necessary part of the tooling to implement sandboxing of containers.
cpuguy83 6y ago
SELinux would have prevented a number of container breakout issues. That said, as someone one time worked on an SELinux issue in Docker and then became the defacto "expert"... it's a terrible mess, often coming with incompatible policy changes tied to kernel changes and policies which are stacked on top of each other like a house of cards that are difficult to tease apart for other distros.
6y ago

tedk-42 6y ago

SELinux was a good way of securing linux machines with many services sharing things like file systems and binaries. k8s solves this in a different way. You can write applications that are completely stateless when spun up on a k8s node and can be limited to a read only file system. Your container can be a scratch container with no other executables binaries. I don't know what their product is, but i seriously doubt mixing these two concepts is the right thing to do.

rwmj 6y ago
I think you need to look at how it really works under the hood if you believe that. RHEL runs containers with a few layers around them, one of which is SELinux (example of an exploit being blocked: https://www.redhat.com/en/blog/latest-container-exploit-runc-can-be-blocked-selinux https://www.redhat.com/en/blog/latest-container-exploit-runc...)
kevin_nisbet 6y ago
I'm not sure it's fair to say the advantages of selinux are solely around sharing services and filesystems on a host. I believe selinux also allows a more expressive policy on how a running piece of software can interact with the kernel than say seccomp can, which is probably what you're container runtime is using to limit kernel access. And unless you're configuring seccomp, you're probably picking up some default policy designed to work on as many systems as possible, which sort of violated least privilege. I'm also not sure I would consider this a totally solved problem within kubernetes. In supporting gravity, our kubernetes distribution, we used to hard-disable all privileged containers. This become a common point of friction, since many pieces of common kubernetes software seemed to just use privileged containers instead of properly listing and setting needed capabilities. Although this is a separate issue from selinux, I'm just pointing out that many kubernetes clusters may not be as isolated between running processes as expected. And to answer what the product is, gravity is a kubernetes distribution, that in this use case allows for packaging a kubernetes based application together with the runtime, for a unified installer for the application + runtime. Think something like packaging up a SAAS application, and being able to install a copy within a banks network who doesn't want their systems talking to the internet while abstracting away the kubernetes portions. The problem we were running into, is that end user would keep trying to install the application onto selinux enabled hosts, which would break the install or runtime. And whether you believe in selinux or not, it causes significant friction to try and tell your customers we got this, just turn off that security feature in production.

ape4 6y ago

NSA made SELinux. My theory is that they made it so hard to use so nobody would use it /s

LinuxBender 6y ago
Redhat helped the NSA make SELinux substantially easier for more people to use. Disclaimer: It's still not super easy to use. The implementation at the NSA did not have the concept of "unconfined" anything and they use MLS policies. At least, that was the intent. This may have changed with time and perhaps by now they are using Redhat's version of SELinux.

heywire 6y ago

An off-topic but somewhat related question. What is a good way to whitelist which applications can access the network on my desktop linux system? (Currently running Arch, but I distro-hop quite often). Years ago when I had a MacBook, I used a program called Little Snitch. I see that there is a similar application for Linux called OpenSnitch, but I’m hoping there is a better way to utilize the built in features of Linux rather than introducing a 3rd party kernel module. Are SELinux / AppArmor the answer? Cgroups? Something else?

totony 6y ago
Selinux can add labels to network packets which can be matched with nftables & al. Its a bit complicated tho. Since you need to add and create labels. You can also match by owner in iptables. So you can have a network restricted user and sudo to it.