8 ms·

Introducing rkt’s ability to detect privilege escalation attacks on containers

111 points by manacit 10y ago

buckhx 10y ago
Who is using rkt in production? I'm working on a system from scratch and planning on using k8s and just assumed docker would be the container engine of choice, but the more I hear/ read about rkt the more it seems like the better choice.
- Perceptes 10y ago
  We will be as soon as it has the same level of support in Kubernetes.
  - rwvhp 10y ago
    Should be around 1.6 or 1.7 from a talk given in December by one of the core rkt team. Logging changes and being able to 'exec' into a container seem to be the main bugbears at the moment (may be other things but this was my take-away from the talk). I looked into this on the 1.4 kubernetes release but ended up with file descriptors not being freed up leading to a very unstable kubernetes cluster nodes. We are really enjoying using Ceph under rkt-fly[1] upon CoreOS though. [1] https://coreos.com/rkt/docs/latest/running-fly-stage1.html https://coreos.com/rkt/docs/latest/running-fly-stage1.html
- rwvhp 10y ago
  BlaBlaCar have been using rkt since some really early version. Simon Lallemand gave a talk in Berlin at the CoreOS meetup about it. I don't recall it being recorded, though there is a blog entry covering some of the details[1]. Interesting how they built a lot of tooling for the container building and management before other ways appeared, and the approach they took to it. [1] http://blablatech.com/blog/why-and-how-blablacar-went-full-containers http://blablatech.com/blog/why-and-how-blablacar-went-full-c...
  - robszumski 10y ago
    Video of a talk from BlaBlaCar in Paris on rkt: https://www.youtube.com/watch?v=dW7U2PZ16ek https://www.youtube.com/watch?v=dW7U2PZ16ek And a blog post on their site: http://blablatech.com/blog/why-and-how-blablacar-went-full-containers http://blablatech.com/blog/why-and-how-blablacar-went-full-c...
- tyingq 10y ago
  This list of known issues with rkt on k8s might be helpful: http://kubernetes.io/docs/getting-started-guides/rkt/notes/ http://kubernetes.io/docs/getting-started-guides/rkt/notes/
lima 10y ago
Glad to see that they acknowledge GRSecurity.
- technofiend 10y ago
  GRSecurity has some interesting ideas and if systemd is any indication some Linux vendors will happily work with iconoclasts. However it seems like much Theo De Raadt can rub people the wrong way resulting in OpenBSD, GR Security's alleged refusal to submit patches in a manner Linus will accept has resulted in their rejection from the kernel mainline. Considering for some applications Linux needs all the hardening it can get, this is truly unfortunate.
  - geofft 10y ago
    There's good news here: the Kernel Self Protection Project is a team of Linux devs who have been successfully upstreaming good, compatible ideas from grsecurity. https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Pr... https://lwn.net/Articles/698827/ https://lwn.net/Articles/698827/ (note the admission in the comments by all sides that KSPP is essentially about upstreaming all the upstreamable ideas in grsecurity, to bypass the sort of yelling that you see in those comments) The trouble is that the incompatible ideas run contrary to Linux's philosophy of compatibility / "don't break userspace" being the highest virtue, but that at least isn't about personalities, just differing design goals.
  - SEJeff 10y ago
    You've got it a bit wrong. Brad Spengler (aka spender) and much of the grsecurity team's goal is making Linux more secure by working on grsecurity. They're not interested in totally rewriting large portions of Linux technical debt to make Linux more secure, but upstream demands they do it. Is spender often a bit of a prick, yes, but is he virtually 100% right on security things, also yes. It wasn't really a rejection from kernel mainline so much as they don't have any interest in submitting it. You can't have rejected what you never submitted in the first place. Fast forward to now and you have kernel heavyweights like Kees Cook (former Canonical kernel/security lead now chromeos security badass extraordinaire) working on bringing these features into mainline via the aforementioned KSP projects. In the end, Linux is getting more secure, and it is largely in part to the hard work of the entire GRSec team, even if they're a bit abrasive to work with.
geertj 10y ago
Sounds very familiar to VMware's VMsafe [1], which apparently they EOL'd [2]. [1] http://www.gabesvirtualworld.com/vmsafe-what-is-it-exactly/ http://www.gabesvirtualworld.com/vmsafe-what-is-it-exactly/ [2] https://kb.vmware.com/kb/2058911 https://kb.vmware.com/kb/2058911
geofft 10y ago
How close are we to a world without SUID binaries? It seems like you could just replace sudo with RPC to some privileged daemon (which could be as simple as the existing sshd and "ssh root@localhost", but another daemon would be fine), and most of the other use cases (mount, pt-chown, dbus-daemon-launch-helper, etc.) have been slowly disappearing. That would avoid the weird special case for SUID binaries, which looks like the sort of thing that someone is going to discover an attack on in a year or two. (And it's a special case in lots of systems, not just this one.)
- SEJeff 10y ago
  You mean like Polkit (the software formerly known as Policy Kit): https://en.wikipedia.org/wiki/Polkit https://en.wikipedia.org/wiki/Polkit
  - geofft 10y ago
    pkexec is still SUID, though. If it asked polkitd to actually execute the process, then it would work. (Something like systemd-run, which backed by polkit, is closer, except it's not set up for interactive commands. You need to proxy I/O as well as the terminal state and terminal-generated signals - and SSH already does all of that.) Polkit, incidentally, is another good reason to avoid all SUID binaries on your system; see CVE-2011-1485 and then CVE-2013-4288 and friends, which are all race conditions involving tricking polkit into thinking that you're root by execing a SUID binary right before it checks who you are. http://seclists.org/oss-sec/2013/q3/624 http://seclists.org/oss-sec/2013/q3/624 This is indisputably a security bug in polkit, but the fact that it's so non-obvious how to do the right thing (and whether the right thing is even possible) is a good reason not to tempt fate by having SUID binaries around. It's not going to be the only system that's vulnerable to this.
- catern 10y ago
  >most of the other use cases (mount, pt-chown, dbus-daemon-launch-helper, etc.) Those listed examples could also be could be achieved with IPC to a privileged daemon. But unfortunately not all setuid executables can be replaced that way. The major design flaw with setuid executables is that they run in an execution environment (view of the root filesystem, Linux namespace, etc.) provided by the caller. A good explanation of this is http://maxsi.org/blog/setuid-bit-considered-harmful/ http://maxsi.org/blog/setuid-bit-considered-harmful/ Unfortunately this is why not all setuid executables can be replaced with IPC. setuid binaries like "nsenter" allow you to run processes with a selective modification to one part of your execution environment, while still inheriting everything else unmodified. You can't achieve this with IPC to a privileged daemon because that daemon cannot inherit your current execution environment when starting new processes. Nevertheless, getting rid of setuid is very important. Kind of baffling to me that it hasn't been done yet. The closest we've come is replacing setuid with filesystem capabilities, but those have the same fundamental design problem. I've been working on genuinely getting rid of setuid some myself. Happy to collaborate with anyone else who wants to work on this.
catern 10y ago
I don't have the inclination to write a long comment about this. But reading sections like this: >This state can then be verified whenever a process performs an action requiring a permissions check. For example, when a process requests that a file be opened, the kernel now calls out to the hypervisor. The hypervisor is then able to examine the process state and ensure that it remains consistent with its internal representation of process state. Makes me very uneasy. You're making the hypervisor more complicated, duplicating the logic in the kernel, because you feel the kernel is insecure. The only argument for doing this, AFAIK, is that the hypervisor is still simpler overall, and therefore more secure. But when you start introducing these kind of complications, that argument becomes kind of farcical... Adding another layer on top of the kernel and duplicating your permissions checks in there, I do not think that will end well.
- mjg59 10y ago
  You can take a look at the patch - there's very little additional complexity in either the kernel or the hypervisor. The kernel permission checks remain unmodified, with the hypervisor checks as a backup. The argument for this isn't that the hypervisor is simpler, it's that the hypervisor is running at a separate privilege level to the rest of the stack. The hardware enforces an additional level of separation. This just adds validation on the other side of that boundary. (Disclaimer: author of the code)
  - pranavkpr 10y ago
    Yup, completely agree with you.
  - daveguy 10y ago
    Best disclaimer ever.
  - xkxx 10y ago
    Hey, author, thanks for commenting. AFAIR, you also are the guy who criticizes Ubuntu's policy regarding derivative works. Anyway, I have a question for you. If I understand rkt's architecture correctly, the hypervisor runs multiple Linux kernel instances, every kernel runs several processes. If somebody exploits one of those processes and creates a root-owned process, then rkt detects such an attempt and restarts the kernel. But what if the exploit doesn't create any root-owned processes? You can get rights equal to those of the superuser without getting uid=0.
benmmurphy 10y ago
i think this is useful in that against unsophisticated attackers and people who don't know it is there the protection will trigger. however, it sounds like it only is going to be triggering when a process forks. so if you have some exploit that raises your creds then just don't fork and you don't trigger the detection. you can do everything evil in the system you want to do without forking. and if you want the convenience of a shell you can just chmod suid a binary and exec that. anyway, i think this generally idea is interesting because you can do a lot with it and if you keep what you are doing a secret (because you are running a public cloud platform for example) then you can have quite effective protection against attackers that don't have insider information. but if you are running a public cloud probably one of your biggest threats are insiders. so meh.
- Matt3o12_ 10y ago
  Well, I don't think your idea is very good either. You are basically doing security through obscurity by not reveling your mitigation methods. Sure they can throw off many attackers but any dedicated one will eventually find vulnerabilities (especially if they are state actors). If you have some mitigation techniques available, you will be able to block many attacks while providing a good guideline what your software does not do and encourage "white hacks" to come up with better mitigations.
- mjg59 10y ago
  It validates on every syscall that has a permission check. If you modify your process status without forking and then make a syscall, it'll still trigger. (disclaimer: author of this feature)
  - benmmurphy 10y ago
    Sorry. I completely misread the original post. That seems a lot more sound :)

111 points by manacit 10y ago

Who is using rkt in production? I'm working on a system from scratch and planning on using k8s and just assumed docker would be the container engine of choice, but the more I hear/ read about rkt the more it seems like the better choice.

Perceptes 10y ago
We will be as soon as it has the same level of support in Kubernetes.
- rwvhp 10y ago
  Should be around 1.6 or 1.7 from a talk given in December by one of the core rkt team. Logging changes and being able to 'exec' into a container seem to be the main bugbears at the moment (may be other things but this was my take-away from the talk). I looked into this on the 1.4 kubernetes release but ended up with file descriptors not being freed up leading to a very unstable kubernetes cluster nodes. We are really enjoying using Ceph under rkt-fly[1] upon CoreOS though. [1] https://coreos.com/rkt/docs/latest/running-fly-stage1.html https://coreos.com/rkt/docs/latest/running-fly-stage1.html
rwvhp 10y ago
BlaBlaCar have been using rkt since some really early version. Simon Lallemand gave a talk in Berlin at the CoreOS meetup about it. I don't recall it being recorded, though there is a blog entry covering some of the details[1]. Interesting how they built a lot of tooling for the container building and management before other ways appeared, and the approach they took to it. [1] http://blablatech.com/blog/why-and-how-blablacar-went-full-containers http://blablatech.com/blog/why-and-how-blablacar-went-full-c...
- robszumski 10y ago
  Video of a talk from BlaBlaCar in Paris on rkt: https://www.youtube.com/watch?v=dW7U2PZ16ek https://www.youtube.com/watch?v=dW7U2PZ16ek And a blog post on their site: http://blablatech.com/blog/why-and-how-blablacar-went-full-containers http://blablatech.com/blog/why-and-how-blablacar-went-full-c...
tyingq 10y ago
This list of known issues with rkt on k8s might be helpful: http://kubernetes.io/docs/getting-started-guides/rkt/notes/ http://kubernetes.io/docs/getting-started-guides/rkt/notes/

lima 10y ago

Glad to see that they acknowledge GRSecurity.

technofiend 10y ago
GRSecurity has some interesting ideas and if systemd is any indication some Linux vendors will happily work with iconoclasts. However it seems like much Theo De Raadt can rub people the wrong way resulting in OpenBSD, GR Security's alleged refusal to submit patches in a manner Linus will accept has resulted in their rejection from the kernel mainline. Considering for some applications Linux needs all the hardening it can get, this is truly unfortunate.
- geofft 10y ago
  There's good news here: the Kernel Self Protection Project is a team of Linux devs who have been successfully upstreaming good, compatible ideas from grsecurity. https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Pr... https://lwn.net/Articles/698827/ https://lwn.net/Articles/698827/ (note the admission in the comments by all sides that KSPP is essentially about upstreaming all the upstreamable ideas in grsecurity, to bypass the sort of yelling that you see in those comments) The trouble is that the incompatible ideas run contrary to Linux's philosophy of compatibility / "don't break userspace" being the highest virtue, but that at least isn't about personalities, just differing design goals.
- SEJeff 10y ago
  You've got it a bit wrong. Brad Spengler (aka spender) and much of the grsecurity team's goal is making Linux more secure by working on grsecurity. They're not interested in totally rewriting large portions of Linux technical debt to make Linux more secure, but upstream demands they do it. Is spender often a bit of a prick, yes, but is he virtually 100% right on security things, also yes. It wasn't really a rejection from kernel mainline so much as they don't have any interest in submitting it. You can't have rejected what you never submitted in the first place. Fast forward to now and you have kernel heavyweights like Kees Cook (former Canonical kernel/security lead now chromeos security badass extraordinaire) working on bringing these features into mainline via the aforementioned KSP projects. In the end, Linux is getting more secure, and it is largely in part to the hard work of the entire GRSec team, even if they're a bit abrasive to work with.

geertj 10y ago

Sounds very familiar to VMware's VMsafe [1], which apparently they EOL'd [2]. [1] http://www.gabesvirtualworld.com/vmsafe-what-is-it-exactly/ http://www.gabesvirtualworld.com/vmsafe-what-is-it-exactly/ [2] https://kb.vmware.com/kb/2058911 https://kb.vmware.com/kb/2058911

geofft 10y ago

How close are we to a world without SUID binaries? It seems like you could just replace sudo with RPC to some privileged daemon (which could be as simple as the existing sshd and "ssh root@localhost", but another daemon would be fine), and most of the other use cases (mount, pt-chown, dbus-daemon-launch-helper, etc.) have been slowly disappearing. That would avoid the weird special case for SUID binaries, which looks like the sort of thing that someone is going to discover an attack on in a year or two. (And it's a special case in lots of systems, not just this one.)

SEJeff 10y ago
You mean like Polkit (the software formerly known as Policy Kit): https://en.wikipedia.org/wiki/Polkit https://en.wikipedia.org/wiki/Polkit
- geofft 10y ago
  pkexec is still SUID, though. If it asked polkitd to actually execute the process, then it would work. (Something like systemd-run, which backed by polkit, is closer, except it's not set up for interactive commands. You need to proxy I/O as well as the terminal state and terminal-generated signals - and SSH already does all of that.) Polkit, incidentally, is another good reason to avoid all SUID binaries on your system; see CVE-2011-1485 and then CVE-2013-4288 and friends, which are all race conditions involving tricking polkit into thinking that you're root by execing a SUID binary right before it checks who you are. http://seclists.org/oss-sec/2013/q3/624 http://seclists.org/oss-sec/2013/q3/624 This is indisputably a security bug in polkit, but the fact that it's so non-obvious how to do the right thing (and whether the right thing is even possible) is a good reason not to tempt fate by having SUID binaries around. It's not going to be the only system that's vulnerable to this.
catern 10y ago
>most of the other use cases (mount, pt-chown, dbus-daemon-launch-helper, etc.) Those listed examples could also be could be achieved with IPC to a privileged daemon. But unfortunately not all setuid executables can be replaced that way. The major design flaw with setuid executables is that they run in an execution environment (view of the root filesystem, Linux namespace, etc.) provided by the caller. A good explanation of this is http://maxsi.org/blog/setuid-bit-considered-harmful/ http://maxsi.org/blog/setuid-bit-considered-harmful/ Unfortunately this is why not all setuid executables can be replaced with IPC. setuid binaries like "nsenter" allow you to run processes with a selective modification to one part of your execution environment, while still inheriting everything else unmodified. You can't achieve this with IPC to a privileged daemon because that daemon cannot inherit your current execution environment when starting new processes. Nevertheless, getting rid of setuid is very important. Kind of baffling to me that it hasn't been done yet. The closest we've come is replacing setuid with filesystem capabilities, but those have the same fundamental design problem. I've been working on genuinely getting rid of setuid some myself. Happy to collaborate with anyone else who wants to work on this.

catern 10y ago

I don't have the inclination to write a long comment about this. But reading sections like this: >This state can then be verified whenever a process performs an action requiring a permissions check. For example, when a process requests that a file be opened, the kernel now calls out to the hypervisor. The hypervisor is then able to examine the process state and ensure that it remains consistent with its internal representation of process state. Makes me very uneasy. You're making the hypervisor more complicated, duplicating the logic in the kernel, because you feel the kernel is insecure. The only argument for doing this, AFAIK, is that the hypervisor is still simpler overall, and therefore more secure. But when you start introducing these kind of complications, that argument becomes kind of farcical... Adding another layer on top of the kernel and duplicating your permissions checks in there, I do not think that will end well.

mjg59 10y ago
You can take a look at the patch - there's very little additional complexity in either the kernel or the hypervisor. The kernel permission checks remain unmodified, with the hypervisor checks as a backup. The argument for this isn't that the hypervisor is simpler, it's that the hypervisor is running at a separate privilege level to the rest of the stack. The hardware enforces an additional level of separation. This just adds validation on the other side of that boundary. (Disclaimer: author of the code)
- pranavkpr 10y ago
  Yup, completely agree with you.
- daveguy 10y ago
  Best disclaimer ever.
- xkxx 10y ago
  Hey, author, thanks for commenting. AFAIR, you also are the guy who criticizes Ubuntu's policy regarding derivative works. Anyway, I have a question for you. If I understand rkt's architecture correctly, the hypervisor runs multiple Linux kernel instances, every kernel runs several processes. If somebody exploits one of those processes and creates a root-owned process, then rkt detects such an attempt and restarts the kernel. But what if the exploit doesn't create any root-owned processes? You can get rights equal to those of the superuser without getting uid=0.

benmmurphy 10y ago

i think this is useful in that against unsophisticated attackers and people who don't know it is there the protection will trigger. however, it sounds like it only is going to be triggering when a process forks. so if you have some exploit that raises your creds then just don't fork and you don't trigger the detection. you can do everything evil in the system you want to do without forking. and if you want the convenience of a shell you can just chmod suid a binary and exec that. anyway, i think this generally idea is interesting because you can do a lot with it and if you keep what you are doing a secret (because you are running a public cloud platform for example) then you can have quite effective protection against attackers that don't have insider information. but if you are running a public cloud probably one of your biggest threats are insiders. so meh.

Matt3o12_ 10y ago
Well, I don't think your idea is very good either. You are basically doing security through obscurity by not reveling your mitigation methods. Sure they can throw off many attackers but any dedicated one will eventually find vulnerabilities (especially if they are state actors). If you have some mitigation techniques available, you will be able to block many attacks while providing a good guideline what your software does not do and encourage "white hacks" to come up with better mitigations.
mjg59 10y ago
It validates on every syscall that has a permission check. If you modify your process status without forking and then make a syscall, it'll still trigger. (disclaimer: author of this feature)
- benmmurphy 10y ago
  Sorry. I completely misread the original post. That seems a lot more sound :)