3 ms·

Projects like this reiterate just how important it is, from a security perspective, to ensure your production services are running in containers without an incl

Projects like this reiterate just how important it is, from a security perspective, to ensure your production services are running in containers without an included shell. If an attacker can get a shell, they can do pretty much anything. Debug containers are now a stable feature in Kubernetes. It honestly boggles my mind how companies will throw so much time, money, and effort into the cybersecurity product du jour when they can get the vast majority of the value by moving everything into distroless, shell-less containers running on managed VMs that are optimized for container workloads.

robinhoodexe 3y ago
Out of curiosity, how would you run Python or R workloads in kubernetes without a distro or shell?
- jonhohle 3y ago
  Python needs an ld.so and libc (minimally) but not a shell or other external utilities. Shebang scripts are loaded by ld.so, not the shell.
  - khrbtxyz 3y ago
    Shebang scripts are supported directly by the kernel via the exec family of system calls, so ld.so shouldn't be involved. https://github.com/torvalds/linux/blob/master/fs/Kconfig.binfmt#L100 https://github.com/torvalds/linux/blob/master/fs/Kconfig.bin... config BINFMT_SCRIPT tristate "Kernel support for scripts starting with #!" default y help Say Y here if you want to execute interpreted scripts starting with #! followed by the path to an interpreter.
  - thecodedmessage 3y ago
    Yeah, I remember reading the code in the kernel that handles shebang a long time ago. ld.so is not involved.
  - usr1106 3y ago
    Python with batteries included, doesn't that mean exploit tools included? No personal attack intended, I am wondering this about my own embedded product which contains Python.
- ttymck 3y ago
  https://stackoverflow.com/questions/62581924/is-there-a-way-to-compile-a-python-program-to-binary-and-use-it-with-a-scratch-d https://stackoverflow.com/questions/62581924/is-there-a-way-...
- oneshtein 3y ago
  Python is better than shell, so intruder will use it first.
riddley 3y ago
Please do a write-up of these debug features. I'd love to learn about them.
- deathanatos 3y ago
  Have you read the docs? https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container https://kubernetes.io/docs/tasks/debug/debug-application/deb...
swozey 3y ago
I work on multi-tenant k8s clusters at CDNs and used to work at Rancher and have seen just about every multi-tenant / federated deployment there is, nasa, meta, etc. Stuff where even the hardware vfio paths mounting the nic or gpu channels keep users apart from one another, and the entire path out of the cluster are completely apart from k8s and any userspace and would be something like multus as a shim- what exactly are you referring to that can cgroup hop via a shell that we're not currently mitigating? It's the hardware being infected by something we worry about at this level. A lot of the CDNs even use tools like kubevirt where the segmentation is even further. And then we have gvisor, firecracker, etc. I admittedly haven't touched k8s code since 1.18 but I can't think of anything like you're referring to and I definitely would like to know about it. Thanks.
kiririn 3y ago
Also can go even simpler and use apparmor and/or systemd hardening instead of containers
deathanatos 3y ago
Debug containers have been a bit of a let down in UX. It's rather difficult to do sort of basic stuff, and a lot of stuff is hidden behind flags. E.g., if you just sort of roll with the defaults, you're dropped into a pod in a very confused state: `ps -ef` says nothing in your debugee is running, and the filesystem of the debugee is nowhere to be found. You can work around both of those (the first is --target, but the latter requires an intricate SO answer¹) but its the sort of thing that would be nicer out of the box? The node debugging mode is a bit better: by default, puts you in at least the host pidns, and mounts the host FS. ¹https://stackoverflow.com/questions/73355970/how-to-get-access-to-filesystem-with-kubectl-debug-ephemeral-containers https://stackoverflow.com/questions/73355970/how-to-get-acce...
- bravetraveler 3y ago
  As someone who doesn't do much (any) k8s... Seems only marginally better than using nsenter on a privileged container to just go muck with the host
- freedomben 3y ago
  I've also been disappointed with debug containers. They are often not useful for debugging trick production-only issues because so many of those issues are related to container state, which can be (often is) different inside the container. Certain languages/platforms and developer discipline are better about this than others, like if you're using functional/immutable languages then it's less of an issue. For applications that aren't super high security, I've been really appreciating using immutable hosts (that get regularly updated/rotated), along with CI/CD that is constantly rebuilding from source, applying latest software updates, and deploying the latest version of the app. Combined with other tools like scanners, and de-bloating your images, it really raises the height of the fruit.
- solatic 3y ago
  > UX I would agree, but so much of day-to-day Kubernetes is arcane CLI commands to begin with. Other stuff that is non-trivial to do on the CLI but comes up in most reasonable production deployments: * Rotating secrets without exposing the secret to the shell history file (hint: kubectl apply -f can take - to signify atdin, but not kubectl patch!) * Ensuring your edits to a ConfigMap pass application-level validation (i.e. your configuration changes won't crash your app, not just that it's a valid ConfigMap) * Anything to do with user auth or RBAC * Scaling the default persistent volume size of a StatefulSet The truth is that Kubernetes is a platform, and just like how most people don't want to run a bare copy of Bash or VIM on their laptop, people will figure out aliases, one-liners, and other functions to help make them effective. So some of working effectively with Kubernetes means, yes, building your own custom debug containers, and writing your own helper shell stuff.

robinhoodexe 3y ago

Out of curiosity, how would you run Python or R workloads in kubernetes without a distro or shell?

jonhohle 3y ago
Python needs an ld.so and libc (minimally) but not a shell or other external utilities. Shebang scripts are loaded by ld.so, not the shell.
- khrbtxyz 3y ago
  Shebang scripts are supported directly by the kernel via the exec family of system calls, so ld.so shouldn't be involved. https://github.com/torvalds/linux/blob/master/fs/Kconfig.binfmt#L100 https://github.com/torvalds/linux/blob/master/fs/Kconfig.bin... config BINFMT_SCRIPT tristate "Kernel support for scripts starting with #!" default y help Say Y here if you want to execute interpreted scripts starting with #! followed by the path to an interpreter.
- thecodedmessage 3y ago
  Yeah, I remember reading the code in the kernel that handles shebang a long time ago. ld.so is not involved.
- usr1106 3y ago
  Python with batteries included, doesn't that mean exploit tools included? No personal attack intended, I am wondering this about my own embedded product which contains Python.
ttymck 3y ago
https://stackoverflow.com/questions/62581924/is-there-a-way-to-compile-a-python-program-to-binary-and-use-it-with-a-scratch-d https://stackoverflow.com/questions/62581924/is-there-a-way-...
oneshtein 3y ago
Python is better than shell, so intruder will use it first.

riddley 3y ago

Please do a write-up of these debug features. I'd love to learn about them.

deathanatos 3y ago
Have you read the docs? https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container https://kubernetes.io/docs/tasks/debug/debug-application/deb...

swozey 3y ago

I work on multi-tenant k8s clusters at CDNs and used to work at Rancher and have seen just about every multi-tenant / federated deployment there is, nasa, meta, etc. Stuff where even the hardware vfio paths mounting the nic or gpu channels keep users apart from one another, and the entire path out of the cluster are completely apart from k8s and any userspace and would be something like multus as a shim- what exactly are you referring to that can cgroup hop via a shell that we're not currently mitigating? It's the hardware being infected by something we worry about at this level. A lot of the CDNs even use tools like kubevirt where the segmentation is even further. And then we have gvisor, firecracker, etc. I admittedly haven't touched k8s code since 1.18 but I can't think of anything like you're referring to and I definitely would like to know about it. Thanks.

kiririn 3y ago

Also can go even simpler and use apparmor and/or systemd hardening instead of containers

deathanatos 3y ago

Debug containers have been a bit of a let down in UX. It's rather difficult to do sort of basic stuff, and a lot of stuff is hidden behind flags. E.g., if you just sort of roll with the defaults, you're dropped into a pod in a very confused state: `ps -ef` says nothing in your debugee is running, and the filesystem of the debugee is nowhere to be found. You can work around both of those (the first is --target, but the latter requires an intricate SO answer¹) but its the sort of thing that would be nicer out of the box? The node debugging mode is a bit better: by default, puts you in at least the host pidns, and mounts the host FS. ¹https://stackoverflow.com/questions/73355970/how-to-get-access-to-filesystem-with-kubectl-debug-ephemeral-containers https://stackoverflow.com/questions/73355970/how-to-get-acce...

bravetraveler 3y ago
As someone who doesn't do much (any) k8s... Seems only marginally better than using nsenter on a privileged container to just go muck with the host
freedomben 3y ago
I've also been disappointed with debug containers. They are often not useful for debugging trick production-only issues because so many of those issues are related to container state, which can be (often is) different inside the container. Certain languages/platforms and developer discipline are better about this than others, like if you're using functional/immutable languages then it's less of an issue. For applications that aren't super high security, I've been really appreciating using immutable hosts (that get regularly updated/rotated), along with CI/CD that is constantly rebuilding from source, applying latest software updates, and deploying the latest version of the app. Combined with other tools like scanners, and de-bloating your images, it really raises the height of the fruit.
solatic 3y ago
> UX I would agree, but so much of day-to-day Kubernetes is arcane CLI commands to begin with. Other stuff that is non-trivial to do on the CLI but comes up in most reasonable production deployments: * Rotating secrets without exposing the secret to the shell history file (hint: kubectl apply -f can take - to signify atdin, but not kubectl patch!) * Ensuring your edits to a ConfigMap pass application-level validation (i.e. your configuration changes won't crash your app, not just that it's a valid ConfigMap) * Anything to do with user auth or RBAC * Scaling the default persistent volume size of a StatefulSet The truth is that Kubernetes is a platform, and just like how most people don't want to run a bare copy of Bash or VIM on their laptop, people will figure out aliases, one-liners, and other functions to help make them effective. So some of working effectively with Kubernetes means, yes, building your own custom debug containers, and writing your own helper shell stuff.