15 ms·

Docker 0-Day Stopped Cold by SELinux

275 points by jwildeboer 10y ago

sofaofthedamned 10y ago
Does AppArmor also block this I wonder?
- placchetti 10y ago
  Depends on how well it's configured. My guess is that an out-of-the-box profile probably wouldn't.
- w8rbt 10y ago
  I think it could. Here's a write-up showing how AppArmor can protect Docker containers and the underlying host... quote from the article, "So without even patching the container we have prevented rouge pid from spawning using a correct security profile with AppArmor." https://scottydoesntknow.io/container-secure-right/
- wlamartin 10y ago
  AFAIK as I know, the answer is yes. In fact if you look at the Docker apparmor documentation you can see an example where ptrace was blocked https://docs.docker.com/engine/security/apparmor/ https://docs.docker.com/engine/security/apparmor/
  - wlamartin 10y ago
    I just want to update this to clarify that it blocks ptrace, but this is only part of the issue and you shouldn't rely on AppArmor to mitigate this CVE entirely.
- ajross 10y ago
  AppArmor and SELinux share the same set of LSM hooks underneath, so there's comparatively little difference in capability. The big fight is over the philosophy of how they are configured (very broadly: AppArmor tends to tolerate ad hoc configuration by doing what you want instead of what you say, and SELinux like to break everything it doesn't understand perfectly). I really wish SELinux would evolve some better configuration tools/culture, but after a decade and a half I despair of this ever happening. Every single Fedora release I leave it enabled on my personal machine thinking "THIS will be the moment where I really puzzle it out". Then I disable it a week later when I realize how much annoying work configuring rules for my rando backup and device management scripting will be, and when I see that it still lacks rules for a bunch of in-distro tools I need to use.
  - brians 10y ago
    Last I looked, Apparmor still applied rules based on file names, not on types assigned to the underlying objects. That's a big limit on writing correct policies: every hardlink is like a firewall-spanning device.
    - viraptor 10y ago
      I see it mentioned so often, but in practice... is that really an issue for you? You need to have root privs with unrestricted access (or at least hardlink creation in the directory explicitly allowed by apparmor) for that. That means the attack would have to look something like: 1. gain access to the system (unrelated exploit) 2. elevate to root with profile which can create hardlinks at all 3. have access to a directory unrestricted in the profile 4. have local, unrestricted application which can be exploited by hardlink manipulation This is theoretically possible on some systems. But it's a massive effort and fairly easily mitigated by having a profile for all running services which disallows hardlinks in the first place. As far as risk of service exploitation goes, this should be a fairly minimal one. (and requires targeted approach)
  - peterwwillis 10y ago
    AppArmor is a DAC+MAC and SELinux is an RBAC+MAC. AppArmor is not nearly as powerful. Grsec is much more effective than AppArmor at preventing a wide range of attacks, but SELinux is more or less king.
- jacques_chester 10y ago
  My understanding is: yes, it is blocked by AppArmor. Based on seeing email chatter about this report in the context of Cloud Foundry[0]. Under the hood CF uses runC, partly to allow AppArmor to be applied. Disclosure: I work for Pivotal, the majority contributor of engineering to Cloud Foundry. [0] https://www.cloudfoundry.org/cve-2016-9962/ https://www.cloudfoundry.org/cve-2016-9962/
  - justincormack 10y ago
    I would recommend not giving that advice, and recommend that all your customers upgrade. There are a number of ways to exploit this, and as stated elsewhere on this thread Red Hat's advice that SeLinux entirely mitigates this is not correct, it is highly likely that it can be exploited on Cloud Foundry too. I would never advise people to not upgrade software when there is a CVE because someone says there is a mitigation, it is very high risk. Disclosure: I worked on testing the fixes for this CVE.
    - jacques_chester 10y ago
      If you'd like to email me, I can connect you to our security triage team to argue the case. If we're wrong, we'll change it. Edit: looks like we already did.
    - wlamartin 10y ago
      Thanks for your comment. We (Garden - Cloud Foundry container project) released with the runc patch yesterday (https://github.com/cloudfoundry/garden-runc-release/releases/tag/v1.1.1 https://github.com/cloudfoundry/garden-runc-release/releases...). It was our understanding from the original report that the vulnerability was mitigated by AppArmor disabling ptrace, by no user process running as pid 1 inside the container, and because in CF buildpack apps, user processes run as unprivileged users. This is the stance communicated in the CVE report. However, with some further consideration and updated information yesterday, we decided it would be prudent to patch and release immediately to be on the safe side. This was communicated to the Cloud Foundry Security team.
CrLf 10y ago
SELinux used to be one of those things you'd disable immediately upon installing a new RHEL/CentOS box for all the troubles it would cause. But default policies have evolved a lot, making this the wrong thing to do, for a few years now. But people still do it out of habit. If you ignore SELinux, it won't cause issues besides the ocasional need to run "restorecon" (which one gets into the habit of doing whenever an "access denied" error happens when permissions seem otherwise correct). But one problem still remains. SELinux is (very) complex and people (myself included) have a very hard time groking its base concepts. This limits adoption greatly, and I'm still to find a decent document that starts from the simple stuff and lets one build a mental concept of how it works before jumping into the more complicated (real-world) use cases.
- w8rbt 10y ago
  That's the issue with LSMs in general. They all work, but historically have been difficult to configure and impossible to maintain. That's changing. There used to be only one (SELinux), however, there's competition now from other LSMs. Smack, AppArmor, Tomoyo, etc. In part, that's why SELinux is improving. I've tried them all and settled on Tomoyo. The documentation is outstanding and it is (to me at least) the easiest LSM to reason about and configure. http://tomoyo.osdn.jp/ http://tomoyo.osdn.jp/
  - jwildeboer 10y ago
    How up2date is tomoyo, though? The Changelog and updates on the front page seem to indicate that not much happened after 2015.
    - w8rbt 10y ago
      Last git commits in the Linux kernel were in 2016: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=40d273782ff16fe1a7445cc05c66a447dfea3433 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.... https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7560ef39dc0bfebba8f43766d8bb4c1ec5eb66fc https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.... https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/log/?id=refs%2Ftags%2Fv4.10-rc3&qt=author&q=tetsuo+handa https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....
deleted 10y ago
[deleted]
giis 10y ago
SELinux is one of our main protection against user abuse. Our project (webminal.org) provides free terminal access anyone. Thus we need protection. Behind the scenes we rely on things like SELinux/Quota/Pam.d/limits.conf/rootkits etc But SELinux has a learning curve than others, but its worth a ton.
jlhawn 10y ago
Since when is a security issue which is known to and patched by the vendor a "0-day"? Have there been any reported exploitations of this vulnerability in the wild before it was publicly disclosed with a patch made immediately available?
- justincormack 10y ago
  No, there were not, and it was disclosed to related vendors some weeks earlier. Some people seem to thing "0 day" is a l33t term for "vulnerability" it seems.
  - jeremyjh 10y ago
    I think all they are suggesting is that running SELinux can protect you from vulnerabilities that no one knows about yet. Obviously they can't give an example of an vulnerability that is still a 0 day.
- notriddle 10y ago
  I think they mean "SELinux was stopping this vulnerability from being exploited before it was even discovered."
hobarrera 10y ago
I'm not sure if the situation has changed, but I recall installing Fedora (and another distro I cannot recall) years ago, and SELinux would kill sshd when a connection was received, on a clean, out-of-the-box installation, making the host inaccessible. These sort of super-critical bug make software go immediately into my blacklist, and it's very hard to come back from that - it basically meant that it had to be disabled immediately, because it's defaults were completely broken.
- chucky_z 10y ago
  I installed Fedora Server 25 a week or so ago on a small server, sshd was open (with an extremely strong random password) and fully functional at install time. SELinux has caused no issues, I actually chose to install docker in the installer at install-time and it came pre-configured to play nice with SELinux. Firewalld on the other hand, I'm still figuring out (firewall-cmd is useful, but trying to translate iptables rules -> firewalld is proving harder than I expected)
  - ams6110 10y ago
    One of the reasons I like OpenBSD. Linux has a habit of taking well established things and replacing them with incompatible things (firewalld, systemd for example). I have enough to do, I don't need to throw away years of acquired experience every few releases. It's one thing if the replacements are clearly better, but for me they just seem to be new, different ways to do the same things.
    - Gracana 10y ago
      I'd argue that (at least in recent years) OpenBSD does a lot of replacing as well, but I happen to like their direction, where the replacements are simpler rather than more complex.
    - djsumdog 10y ago
      firewalld is just a daemon frontend to iptables. The underlying firewall hasn't changed since Linux moved from ipchains to iptables.
bigmac 10y ago
This post is incorrect. SELinux does not fully mitigate this issue. We recommend users update to 1.12.6. I expect Red Hat to issue a retraction shortly. We notified them last night that this post was incorrect. Source: Security at Docker.
- saycheese 10y ago
  >> "Source: Security at Docker" In case it is not obvious, the comment above is by Nathan McCauley, who is the Director of Security for Docker. Source: https://news.ycombinator.com/user?id=bigmac https://news.ycombinator.com/user?id=bigmac
  - philtar 10y ago
    I don't care if Jesus, the director of security in heaven said that. I'm going to take a look at both arguments and decide for myself. No need to name drop.
    - ben0x539 10y ago
      The name (or title) drop might effect appropriate urgency, seems legit. Edit: Don't downvote people trying to help me improve my english. :(
      - MrF3ynmann 10y ago
        affect
        dzamo_norton 10y ago
        effect
        isbadawi 10y ago
        Effect is correct here.
        chowells 10y ago
throw2016 10y ago
Selinux is badly designed and it's not suprisingly people don't use it. Experts are supposed to simplify complexity. You can't design convulated and complex applications that are user hostile and break things without warning and then accuse people of laziness. Few who deploy apps take security casually and selinux is not the only way to gain security. In a typical scenario when deploying software things can already get hairy and with selinux in the way you could end up going down multiple rabbit holes and squander hours only to discover selinux is somehow in the way disabling some functionality but not logging clearly what exactly it is disabling with proper messages. That's why most advice online is to disable it. Given its connected to the NSA and Redhat tried its best to get it into the kernel at one time is all the more reason for anyone concerned about NSA to avoid it. Security experts like the author of Grsec also doesn't think too highly of it.
- ungamed 10y ago
  Spender/GRSec mainly doesn't like it because he has a competing RBAC which people pay for. It might be better, it might not.. but I don't have a GRSEC license.
astrostl 10y ago
Productivity is also stopped cold by SELinux :P
opk 10y ago
Just a pity that SELinux is widely disabled because it ends up being a pain.

275 points by jwildeboer 10y ago

sofaofthedamned 10y ago

Does AppArmor also block this I wonder?

placchetti 10y ago
Depends on how well it's configured. My guess is that an out-of-the-box profile probably wouldn't.
w8rbt 10y ago
I think it could. Here's a write-up showing how AppArmor can protect Docker containers and the underlying host... quote from the article, "So without even patching the container we have prevented rouge pid from spawning using a correct security profile with AppArmor." https://scottydoesntknow.io/container-secure-right/
wlamartin 10y ago
AFAIK as I know, the answer is yes. In fact if you look at the Docker apparmor documentation you can see an example where ptrace was blocked https://docs.docker.com/engine/security/apparmor/ https://docs.docker.com/engine/security/apparmor/
- wlamartin 10y ago
  I just want to update this to clarify that it blocks ptrace, but this is only part of the issue and you shouldn't rely on AppArmor to mitigate this CVE entirely.
ajross 10y ago
AppArmor and SELinux share the same set of LSM hooks underneath, so there's comparatively little difference in capability. The big fight is over the philosophy of how they are configured (very broadly: AppArmor tends to tolerate ad hoc configuration by doing what you want instead of what you say, and SELinux like to break everything it doesn't understand perfectly). I really wish SELinux would evolve some better configuration tools/culture, but after a decade and a half I despair of this ever happening. Every single Fedora release I leave it enabled on my personal machine thinking "THIS will be the moment where I really puzzle it out". Then I disable it a week later when I realize how much annoying work configuring rules for my rando backup and device management scripting will be, and when I see that it still lacks rules for a bunch of in-distro tools I need to use.
- brians 10y ago
  Last I looked, Apparmor still applied rules based on file names, not on types assigned to the underlying objects. That's a big limit on writing correct policies: every hardlink is like a firewall-spanning device.
  - viraptor 10y ago
    I see it mentioned so often, but in practice... is that really an issue for you? You need to have root privs with unrestricted access (or at least hardlink creation in the directory explicitly allowed by apparmor) for that. That means the attack would have to look something like: 1. gain access to the system (unrelated exploit) 2. elevate to root with profile which can create hardlinks at all 3. have access to a directory unrestricted in the profile 4. have local, unrestricted application which can be exploited by hardlink manipulation This is theoretically possible on some systems. But it's a massive effort and fairly easily mitigated by having a profile for all running services which disallows hardlinks in the first place. As far as risk of service exploitation goes, this should be a fairly minimal one. (and requires targeted approach)
- peterwwillis 10y ago
  AppArmor is a DAC+MAC and SELinux is an RBAC+MAC. AppArmor is not nearly as powerful. Grsec is much more effective than AppArmor at preventing a wide range of attacks, but SELinux is more or less king.
jacques_chester 10y ago
My understanding is: yes, it is blocked by AppArmor. Based on seeing email chatter about this report in the context of Cloud Foundry[0]. Under the hood CF uses runC, partly to allow AppArmor to be applied. Disclosure: I work for Pivotal, the majority contributor of engineering to Cloud Foundry. [0] https://www.cloudfoundry.org/cve-2016-9962/ https://www.cloudfoundry.org/cve-2016-9962/
- justincormack 10y ago
  I would recommend not giving that advice, and recommend that all your customers upgrade. There are a number of ways to exploit this, and as stated elsewhere on this thread Red Hat's advice that SeLinux entirely mitigates this is not correct, it is highly likely that it can be exploited on Cloud Foundry too. I would never advise people to not upgrade software when there is a CVE because someone says there is a mitigation, it is very high risk. Disclosure: I worked on testing the fixes for this CVE.
  - jacques_chester 10y ago
    If you'd like to email me, I can connect you to our security triage team to argue the case. If we're wrong, we'll change it. Edit: looks like we already did.
  - wlamartin 10y ago
    Thanks for your comment. We (Garden - Cloud Foundry container project) released with the runc patch yesterday (https://github.com/cloudfoundry/garden-runc-release/releases/tag/v1.1.1 https://github.com/cloudfoundry/garden-runc-release/releases...). It was our understanding from the original report that the vulnerability was mitigated by AppArmor disabling ptrace, by no user process running as pid 1 inside the container, and because in CF buildpack apps, user processes run as unprivileged users. This is the stance communicated in the CVE report. However, with some further consideration and updated information yesterday, we decided it would be prudent to patch and release immediately to be on the safe side. This was communicated to the Cloud Foundry Security team.

CrLf 10y ago

SELinux used to be one of those things you'd disable immediately upon installing a new RHEL/CentOS box for all the troubles it would cause. But default policies have evolved a lot, making this the wrong thing to do, for a few years now. But people still do it out of habit. If you ignore SELinux, it won't cause issues besides the ocasional need to run "restorecon" (which one gets into the habit of doing whenever an "access denied" error happens when permissions seem otherwise correct). But one problem still remains. SELinux is (very) complex and people (myself included) have a very hard time groking its base concepts. This limits adoption greatly, and I'm still to find a decent document that starts from the simple stuff and lets one build a mental concept of how it works before jumping into the more complicated (real-world) use cases.

w8rbt 10y ago
That's the issue with LSMs in general. They all work, but historically have been difficult to configure and impossible to maintain. That's changing. There used to be only one (SELinux), however, there's competition now from other LSMs. Smack, AppArmor, Tomoyo, etc. In part, that's why SELinux is improving. I've tried them all and settled on Tomoyo. The documentation is outstanding and it is (to me at least) the easiest LSM to reason about and configure. http://tomoyo.osdn.jp/ http://tomoyo.osdn.jp/
- jwildeboer 10y ago
  How up2date is tomoyo, though? The Changelog and updates on the front page seem to indicate that not much happened after 2015.
  - w8rbt 10y ago
    Last git commits in the Linux kernel were in 2016: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=40d273782ff16fe1a7445cc05c66a447dfea3433 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.... https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7560ef39dc0bfebba8f43766d8bb4c1ec5eb66fc https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.... https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/log/?id=refs%2Ftags%2Fv4.10-rc3&qt=author&q=tetsuo+handa https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

deleted 10y ago

[deleted]

giis 10y ago

SELinux is one of our main protection against user abuse. Our project (webminal.org) provides free terminal access anyone. Thus we need protection. Behind the scenes we rely on things like SELinux/Quota/Pam.d/limits.conf/rootkits etc But SELinux has a learning curve than others, but its worth a ton.

jlhawn 10y ago

Since when is a security issue which is known to and patched by the vendor a "0-day"? Have there been any reported exploitations of this vulnerability in the wild before it was publicly disclosed with a patch made immediately available?

justincormack 10y ago
No, there were not, and it was disclosed to related vendors some weeks earlier. Some people seem to thing "0 day" is a l33t term for "vulnerability" it seems.
- jeremyjh 10y ago
  I think all they are suggesting is that running SELinux can protect you from vulnerabilities that no one knows about yet. Obviously they can't give an example of an vulnerability that is still a 0 day.
notriddle 10y ago
I think they mean "SELinux was stopping this vulnerability from being exploited before it was even discovered."

hobarrera 10y ago

I'm not sure if the situation has changed, but I recall installing Fedora (and another distro I cannot recall) years ago, and SELinux would kill sshd when a connection was received, on a clean, out-of-the-box installation, making the host inaccessible. These sort of super-critical bug make software go immediately into my blacklist, and it's very hard to come back from that - it basically meant that it had to be disabled immediately, because it's defaults were completely broken.

chucky_z 10y ago
I installed Fedora Server 25 a week or so ago on a small server, sshd was open (with an extremely strong random password) and fully functional at install time. SELinux has caused no issues, I actually chose to install docker in the installer at install-time and it came pre-configured to play nice with SELinux. Firewalld on the other hand, I'm still figuring out (firewall-cmd is useful, but trying to translate iptables rules -> firewalld is proving harder than I expected)
- ams6110 10y ago
  One of the reasons I like OpenBSD. Linux has a habit of taking well established things and replacing them with incompatible things (firewalld, systemd for example). I have enough to do, I don't need to throw away years of acquired experience every few releases. It's one thing if the replacements are clearly better, but for me they just seem to be new, different ways to do the same things.
  - Gracana 10y ago
    I'd argue that (at least in recent years) OpenBSD does a lot of replacing as well, but I happen to like their direction, where the replacements are simpler rather than more complex.
  - djsumdog 10y ago
    firewalld is just a daemon frontend to iptables. The underlying firewall hasn't changed since Linux moved from ipchains to iptables.

bigmac 10y ago

This post is incorrect. SELinux does not fully mitigate this issue. We recommend users update to 1.12.6. I expect Red Hat to issue a retraction shortly. We notified them last night that this post was incorrect. Source: Security at Docker.

saycheese 10y ago
>> "Source: Security at Docker" In case it is not obvious, the comment above is by Nathan McCauley, who is the Director of Security for Docker. Source: https://news.ycombinator.com/user?id=bigmac https://news.ycombinator.com/user?id=bigmac
- philtar 10y ago
  I don't care if Jesus, the director of security in heaven said that. I'm going to take a look at both arguments and decide for myself. No need to name drop.
  - ben0x539 10y ago
    The name (or title) drop might effect appropriate urgency, seems legit. Edit: Don't downvote people trying to help me improve my english. :(
    - MrF3ynmann 10y ago
      affect
      - dzamo_norton 10y ago
        effect
      - isbadawi 10y ago
        Effect is correct here.
      - chowells 10y ago

throw2016 10y ago

Selinux is badly designed and it's not suprisingly people don't use it. Experts are supposed to simplify complexity. You can't design convulated and complex applications that are user hostile and break things without warning and then accuse people of laziness. Few who deploy apps take security casually and selinux is not the only way to gain security. In a typical scenario when deploying software things can already get hairy and with selinux in the way you could end up going down multiple rabbit holes and squander hours only to discover selinux is somehow in the way disabling some functionality but not logging clearly what exactly it is disabling with proper messages. That's why most advice online is to disable it. Given its connected to the NSA and Redhat tried its best to get it into the kernel at one time is all the more reason for anyone concerned about NSA to avoid it. Security experts like the author of Grsec also doesn't think too highly of it.

ungamed 10y ago
Spender/GRSec mainly doesn't like it because he has a competing RBAC which people pay for. It might be better, it might not.. but I don't have a GRSEC license.

astrostl 10y ago

Productivity is also stopped cold by SELinux :P

opk 10y ago

Just a pity that SELinux is widely disabled because it ends up being a pain.