4 ms·

SELinux was a good way of securing linux machines with many services sharing things like file systems and binaries. k8s solves this in a different way. You can

SELinux was a good way of securing linux machines with many services sharing things like file systems and binaries. k8s solves this in a different way. You can write applications that are completely stateless when spun up on a k8s node and can be limited to a read only file system. Your container can be a scratch container with no other executables binaries. I don't know what their product is, but i seriously doubt mixing these two concepts is the right thing to do.

rwmj 6y ago
I think you need to look at how it really works under the hood if you believe that. RHEL runs containers with a few layers around them, one of which is SELinux (example of an exploit being blocked: https://www.redhat.com/en/blog/latest-container-exploit-runc-can-be-blocked-selinux https://www.redhat.com/en/blog/latest-container-exploit-runc...)
kevin_nisbet 6y ago
I'm not sure it's fair to say the advantages of selinux are solely around sharing services and filesystems on a host. I believe selinux also allows a more expressive policy on how a running piece of software can interact with the kernel than say seccomp can, which is probably what you're container runtime is using to limit kernel access. And unless you're configuring seccomp, you're probably picking up some default policy designed to work on as many systems as possible, which sort of violated least privilege. I'm also not sure I would consider this a totally solved problem within kubernetes. In supporting gravity, our kubernetes distribution, we used to hard-disable all privileged containers. This become a common point of friction, since many pieces of common kubernetes software seemed to just use privileged containers instead of properly listing and setting needed capabilities. Although this is a separate issue from selinux, I'm just pointing out that many kubernetes clusters may not be as isolated between running processes as expected. And to answer what the product is, gravity is a kubernetes distribution, that in this use case allows for packaging a kubernetes based application together with the runtime, for a unified installer for the application + runtime. Think something like packaging up a SAAS application, and being able to install a copy within a banks network who doesn't want their systems talking to the internet while abstracting away the kubernetes portions. The problem we were running into, is that end user would keep trying to install the application onto selinux enabled hosts, which would break the install or runtime. And whether you believe in selinux or not, it causes significant friction to try and tell your customers we got this, just turn off that security feature in production.
- pm90 6y ago
  Given the additional context “we need out k8s deployment to work for customers who use selinux on their nodes”, the motivation to do this makes more sense. The majority of k8s users likely don’t share their clusters with untrusted/low trust users so it seems unlikely to help them much. Not to dunk on the work you’ve done: it’s still awesome. Just want to point out the disconnect from users wondering how this would help them.
  - kevin_nisbet 6y ago
    We do try and point this out in the article itself, but apparently failed to convey the total context by trying not to spend a lot of time explaining gravity, which for the particular article shouldn't be that important. > The problem we had was that Gravity was never designed to run on SELinux-enabled hosts and our customers kept randomly breaking their installations when trying to run Gravity with SELinux enabled.
  - jandrewrogers 6y ago
    I think this misrepresents the use case. Most customers that require k8s on SELinux don't share their clusters with untrusted users. It provides much stronger isolation between containers in the case that one is compromised, reducing the scope of damage when a system is successfully attacked. In other words, they don't assume that a trusted container/user stays "trusted" operationally.
jonwalch 6y ago
Their product is (or at least was, when I used it back in late 2018) bootstrapping a K8S cluster on bare metal with "one command". Obviously this is an extremely aspirational claim.

rwmj 6y ago

I think you need to look at how it really works under the hood if you believe that. RHEL runs containers with a few layers around them, one of which is SELinux (example of an exploit being blocked: https://www.redhat.com/en/blog/latest-container-exploit-runc-can-be-blocked-selinux https://www.redhat.com/en/blog/latest-container-exploit-runc...)

kevin_nisbet 6y ago

I'm not sure it's fair to say the advantages of selinux are solely around sharing services and filesystems on a host. I believe selinux also allows a more expressive policy on how a running piece of software can interact with the kernel than say seccomp can, which is probably what you're container runtime is using to limit kernel access. And unless you're configuring seccomp, you're probably picking up some default policy designed to work on as many systems as possible, which sort of violated least privilege. I'm also not sure I would consider this a totally solved problem within kubernetes. In supporting gravity, our kubernetes distribution, we used to hard-disable all privileged containers. This become a common point of friction, since many pieces of common kubernetes software seemed to just use privileged containers instead of properly listing and setting needed capabilities. Although this is a separate issue from selinux, I'm just pointing out that many kubernetes clusters may not be as isolated between running processes as expected. And to answer what the product is, gravity is a kubernetes distribution, that in this use case allows for packaging a kubernetes based application together with the runtime, for a unified installer for the application + runtime. Think something like packaging up a SAAS application, and being able to install a copy within a banks network who doesn't want their systems talking to the internet while abstracting away the kubernetes portions. The problem we were running into, is that end user would keep trying to install the application onto selinux enabled hosts, which would break the install or runtime. And whether you believe in selinux or not, it causes significant friction to try and tell your customers we got this, just turn off that security feature in production.

pm90 6y ago
Given the additional context “we need out k8s deployment to work for customers who use selinux on their nodes”, the motivation to do this makes more sense. The majority of k8s users likely don’t share their clusters with untrusted/low trust users so it seems unlikely to help them much. Not to dunk on the work you’ve done: it’s still awesome. Just want to point out the disconnect from users wondering how this would help them.
- kevin_nisbet 6y ago
  We do try and point this out in the article itself, but apparently failed to convey the total context by trying not to spend a lot of time explaining gravity, which for the particular article shouldn't be that important. > The problem we had was that Gravity was never designed to run on SELinux-enabled hosts and our customers kept randomly breaking their installations when trying to run Gravity with SELinux enabled.
- jandrewrogers 6y ago
  I think this misrepresents the use case. Most customers that require k8s on SELinux don't share their clusters with untrusted users. It provides much stronger isolation between containers in the case that one is compromised, reducing the scope of damage when a system is successfully attacked. In other words, they don't assume that a trusted container/user stays "trusted" operationally.

jonwalch 6y ago

Their product is (or at least was, when I used it back in late 2018) bootstrapping a K8S cluster on bare metal with "one command". Obviously this is an extremely aspirational claim.