5 ms·

For those unaware: Kubernetes has long had horrible default security. RBAC isn't even enabled by default. Nearly every cluster I've seen allows regular users an

For those unaware: Kubernetes has long had horrible default security. RBAC isn't even enabled by default. Nearly every cluster I've seen allows regular users and applications to breakout into the host nodes by default. Network ACLs, if they are ever used, almost never block either intra-cluster or extra-cluster attacks. Unencrypted API ports are standard. The distinctions between "cluster-level" resources and "everything else" unnecessarily complicates creating fine-grained rules. Default access even with RBAC is too wide-open. I won't even get into ABAC... Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default. I don't think it has ever been easier to take over cloud computing environments just by exploiting a dinky web app.

znpy 3y ago
> Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default. There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant. It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production. Also, it doesn’t help that the kubernetes documentation is awful. I remember my time working with kubernetes, it’s a patchwork of stuff… it does require time and a fairly high pain threshold to properly master it.
- orthoxerox 3y ago
  Well, there are two opposing approaches: "up and running" defaults or production defaults. You choose "up-and-running" defaults if you want people to be able to immediately run your software. So this means no encryption, no auth (or "the first user is the admin" kind of auth), storing your state in a bundled SQLite DB, etc. This helps with growth, because random devs can download the container image, run it and check out your features in five minutes. And then run it in production. With production defaults everything is locked down. Go issue some certificates and make sure your OS (or JRE) trusts them, because your nodes won't talk to each other without TLS. No one has any rights to anything by default, can't even log in until you go and edit authN/Z configuration. The container will kill itself if the env var for the database password is still "change_this!". It's a pain to deploy this, but a pleasure to run. Ideally, all products should come with two sets of defaults. Your AIO quickstart image should be easy to run. Your production-grade image or the configs that come with the distribution should be locked down tighter than Fort Knox. K8s is a piece of infrastructure that doesn't really benefit from an "up and running" approach, but it has this kind of defaults for some reason instead of having every default policy resolve to "not allowed".
  - hedora 3y ago
    OpenBSD is an existence proof of a third route: Carefully manage complexity, and write a good manual. OpenBSD is about 22M lines of code; kubernetes is 58M, so the code bases are of comparable complexity (OpenBSD uses many more programming languages than kubernetes does). I'd argue that the base OpenBSD install provides far more utility than Kubernetes, but market share numbers disagree. Sources: https://openhub.net/p/kubernetes https://openhub.net/p/kubernetes https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_lines_of_code_does_openbsd_65_contain/?rdt=50542 https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_li... Edit: Using open hub for both says OpenBSD is 75M lines: https://openhub.net/p/openbsd https://openhub.net/p/openbsd
    - lakomen 3y ago
      I don't follow. OpenBSD is an OS, k8s is a set of services running on an OS.
      - code_biologist 3y ago
        Both are things deployed to host network visible services.
  - mulmen 3y ago
    > or "the first user is the admin" kind of auth My mind immediately went to Eddie Izzard’s “do you have a flag” bit.
- deleted 3y ago
  [deleted]
- ryukafalz 3y ago
  > There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant. > It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production. Disagree. I can install an app to my Sandstorm instance and reasonably expect it to be secure, functional, and performant (inasmuch as the app is performant anyway). The tools we use make things difficult, but I don't think there's as much irreducible complexity here as you're implying. We can and should build better tools.
  - mplewis 3y ago
    Sandstorm is a very different category of system from k8s.
    - znpy 3y ago
      I had the same feeling, but frankly I didn’t bother replying because somebody putting sandstorm and kubernetes on the same level shows their complete lack of any understanding whatsoever. Kinda proves my point: some random amateur sysadmin thinking their experience with sandstorm is enough to put kubernetes in production. But thanks for replying, anyway.
      - ryukafalz 3y ago
        I worked with K8s professionally for several years but thanks for assuming I have no idea what I'm talking about. Yes, they're obviously different classes of systems, but not in the ways that matter for this discussion. If the question is whether or not you can design a hosting platform with secure defaults and managed/tightly-scoped communication between applications, the answer is plainly yes. Is Sandstorm as scalable and fault tolerant as Kubernetes? Obviously not.
- raesene9 3y ago
  Out of curiosity, which bits of the k8s documentation (if there are specific bits) did you find lacking? If it's security related stuff, I can open some issues to see if things can get improved :)
  - znpy 3y ago
    K8s documentation falls essentially in two categories: either reference page for k8s components (like the kubelet) that lists all the options woth a very very very brief description … or conceptual tutorial-like that explain some concepts, but without much details. Nothing in the middle. There is little documentation on how to put the various pieces together. That’s all on various random webpages outside of the main kubernetes.io website. The setup tutorial only describes some basic kubeadm-based setup. There’s a lot to improve, if you ask me.
benatkin 3y ago
Horrible scaling to tiny clusters as well. I'm glad they kept the design pure though. Unencrypted means you get to choose the encryption. Redis didn't have built in encrypted tcp servers for a long time either. When they added it there was a good discussion about the overhead: https://github.com/redis/redis/issues/7595 https://github.com/redis/redis/issues/7595
k8sfellow 3y ago
RBAC is now enabled by default. But I wonder how many of these clusters that are exploitable are Openshift for example. Openshift had secure by default settings from as long as I can remember. Non-root pods as default, api-server and even metric collection ports are protected by authentication. Openshift also had much strict RBAC settings from very beginning(like even in Openshift 3.x days, when k8s had no RBAC by default and you had to opt-in).
- vbezhenar 3y ago
  What I don't like about standard kubernetes (kubeadm) is how hard it is to configure pods being secure by default, so developers couldn't just mount host directories into the pods. You would need complicated configs for that and nobody does that.
  - raesene9 3y ago
    For less complex options in modern k8s, you could configure Pod Security Admission on a namespace basis. It's not too complex as you just add some annotations to a namespace to activate it. The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.
    - vbezhenar 3y ago
      I'm aware about that, but the problem is: it's just some kind of good will. First of all: you need to actually add those annotations to the namespace. If someone adds his own namespace, it won't be there. Second: anyone can just remove those annotations (without complicated RBAC rules) and do hostpath and other things. I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking. One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it. There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it. https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/ https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about. It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.
ransom1538 3y ago
"Default access even with RBAC is too wide-open." Not following this. Isn't a k8s cluster behind a network firewall? Only exposing 443? To me RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. That sounds awful.
- alephnerd 3y ago
  Jeff in Engineering might have deployed an app spanning vCenter as well as EKS, and to continue to allow the app to function wrote subpar ingress/egress rules.
- mulmen 3y ago
  443 is just a port. If we assume that implies HTTPS then that’s still only TLS. If I exploit the service (“securely”!) I am inside your network. I can then pivot to your management layer. Game over.
- vbezhenar 3y ago
  > Isn't a k8s cluster behind a network firewall? Of course not, unless you configure it this way. > RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. What? How comes no one uses service accounts? They're used extensively for many things.
- deleted 3y ago
  [deleted]
cratermoon 3y ago
I've seen non-tech corporations adopting Kubernetes without realizing that it's just one part of Google's infrastructure, not the entire operations side. There's an entirely separate chunk of infrastructure that provides the kind of security you mention. K8s is a "do one thing and do it well" that does workload orchestration. Google's BeyondCorp system enforces a zero-trust policy. It's not left to each part of the infrastructure to independently implement.
- alfalfasprout 3y ago
  Well, Google uses Borg internally which isn't exactly Kubernetes. But yes, the idea is that for it to be useful as infra it has to come without a lot of magic automatically attached.
raesene9 3y ago
Whilst Kubernetes defaults are indeed not great, they aren't that relevant unless you're building your clusters from scratch instead of using a k8s distribution, and I don't think I've seen anyone do that in a long time. RBAC is enabled by default on just about every distribution I've seen (with the notable exception of microk8s) Lack of default admission control to prevent someone with pod creation access breakout to the host is indeed pretty common Unencrypted APIs hasn't really been a problem in years, the insecure API port and read-only Kubelet were a thing but they've been disabled for quite a few versions now. I'd definitely agree that early k8s versions had very bad security, but things have been improving gradually over the versions :)
- 0xbadcafebee 3y ago
  Well first of all, there are still thousands of clusters running with the old insecure defaults, so that's bad. Both GKE and EKS currently have insecure defaults. There's a half dozen ways to take over the entire cluster with the default settings. The unencrypted API port is only this year being deprecated on GKE, I don't know about EKS. Since you can only grant permissions, not deny, most people end up granting too many permissions, defeating the purpose of least privilege. Default permissions for users on GKE/EKS are way too wide open. You can do things you just shouldn't need to do as a regular dev. You shouldn't be able to just deploy a helm chart to a random cluster and then all of a sudden there's an unauthenticated public service on the internet. But that's what every single distribution of K8s gives users by default. Regardless of whether you think that's fine or not, this is how companies get exposed on a regular basis, so it is a real-world persistent security problem that is perpetuated by the defaults.
  - raesene9 3y ago
    People not upgrading is indeed a problem but not one the distributions or k8s can particularly address, unless they force upgrade people which with deprecation/removal of APIs doesn't tend to be popular :) The support lifecycle is pretty short, compared to general enterprise software, so unless orgs are happy running unsupported versions of k8s they're on 1.23 for EKS or 1.24 for GKE by this point. I wouldn't have said that EKS, in particular, had very bad defaults, IME it's the best of the managed distributions for security. Any references for 1/2 dozen ways of taking over a cluster from defaults settings? I'd genuinely be interested as I collect information about Kubernetes security, so I'm always on the look-out for more references. for unencrypted API are you thinking of the Insecure API port or read-only Kubelet? For the insecure API port I don't think that's been a thing for a really long time on managed distros. The only time I remember seeing it was on ACS (the predecessor to AKS) where it was exposed to the container network. read-only Kubelet was more of a thing as that was on worker nodes, but it was less risky.

znpy 3y ago

> Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default. There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant. It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production. Also, it doesn’t help that the kubernetes documentation is awful. I remember my time working with kubernetes, it’s a patchwork of stuff… it does require time and a fairly high pain threshold to properly master it.

orthoxerox 3y ago
Well, there are two opposing approaches: "up and running" defaults or production defaults. You choose "up-and-running" defaults if you want people to be able to immediately run your software. So this means no encryption, no auth (or "the first user is the admin" kind of auth), storing your state in a bundled SQLite DB, etc. This helps with growth, because random devs can download the container image, run it and check out your features in five minutes. And then run it in production. With production defaults everything is locked down. Go issue some certificates and make sure your OS (or JRE) trusts them, because your nodes won't talk to each other without TLS. No one has any rights to anything by default, can't even log in until you go and edit authN/Z configuration. The container will kill itself if the env var for the database password is still "change_this!". It's a pain to deploy this, but a pleasure to run. Ideally, all products should come with two sets of defaults. Your AIO quickstart image should be easy to run. Your production-grade image or the configs that come with the distribution should be locked down tighter than Fort Knox. K8s is a piece of infrastructure that doesn't really benefit from an "up and running" approach, but it has this kind of defaults for some reason instead of having every default policy resolve to "not allowed".
- hedora 3y ago
  OpenBSD is an existence proof of a third route: Carefully manage complexity, and write a good manual. OpenBSD is about 22M lines of code; kubernetes is 58M, so the code bases are of comparable complexity (OpenBSD uses many more programming languages than kubernetes does). I'd argue that the base OpenBSD install provides far more utility than Kubernetes, but market share numbers disagree. Sources: https://openhub.net/p/kubernetes https://openhub.net/p/kubernetes https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_lines_of_code_does_openbsd_65_contain/?rdt=50542 https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_li... Edit: Using open hub for both says OpenBSD is 75M lines: https://openhub.net/p/openbsd https://openhub.net/p/openbsd
  - lakomen 3y ago
    I don't follow. OpenBSD is an OS, k8s is a set of services running on an OS.
    - code_biologist 3y ago
      Both are things deployed to host network visible services.
- mulmen 3y ago
  > or "the first user is the admin" kind of auth My mind immediately went to Eddie Izzard’s “do you have a flag” bit.
deleted 3y ago
[deleted]
ryukafalz 3y ago
> There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant. > It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production. Disagree. I can install an app to my Sandstorm instance and reasonably expect it to be secure, functional, and performant (inasmuch as the app is performant anyway). The tools we use make things difficult, but I don't think there's as much irreducible complexity here as you're implying. We can and should build better tools.
- mplewis 3y ago
  Sandstorm is a very different category of system from k8s.
  - znpy 3y ago
    I had the same feeling, but frankly I didn’t bother replying because somebody putting sandstorm and kubernetes on the same level shows their complete lack of any understanding whatsoever. Kinda proves my point: some random amateur sysadmin thinking their experience with sandstorm is enough to put kubernetes in production. But thanks for replying, anyway.
    - ryukafalz 3y ago
      I worked with K8s professionally for several years but thanks for assuming I have no idea what I'm talking about. Yes, they're obviously different classes of systems, but not in the ways that matter for this discussion. If the question is whether or not you can design a hosting platform with secure defaults and managed/tightly-scoped communication between applications, the answer is plainly yes. Is Sandstorm as scalable and fault tolerant as Kubernetes? Obviously not.
raesene9 3y ago
Out of curiosity, which bits of the k8s documentation (if there are specific bits) did you find lacking? If it's security related stuff, I can open some issues to see if things can get improved :)
- znpy 3y ago
  K8s documentation falls essentially in two categories: either reference page for k8s components (like the kubelet) that lists all the options woth a very very very brief description … or conceptual tutorial-like that explain some concepts, but without much details. Nothing in the middle. There is little documentation on how to put the various pieces together. That’s all on various random webpages outside of the main kubernetes.io website. The setup tutorial only describes some basic kubeadm-based setup. There’s a lot to improve, if you ask me.

benatkin 3y ago

Horrible scaling to tiny clusters as well. I'm glad they kept the design pure though. Unencrypted means you get to choose the encryption. Redis didn't have built in encrypted tcp servers for a long time either. When they added it there was a good discussion about the overhead: https://github.com/redis/redis/issues/7595 https://github.com/redis/redis/issues/7595

k8sfellow 3y ago

RBAC is now enabled by default. But I wonder how many of these clusters that are exploitable are Openshift for example. Openshift had secure by default settings from as long as I can remember. Non-root pods as default, api-server and even metric collection ports are protected by authentication. Openshift also had much strict RBAC settings from very beginning(like even in Openshift 3.x days, when k8s had no RBAC by default and you had to opt-in).

vbezhenar 3y ago
What I don't like about standard kubernetes (kubeadm) is how hard it is to configure pods being secure by default, so developers couldn't just mount host directories into the pods. You would need complicated configs for that and nobody does that.
- raesene9 3y ago
  For less complex options in modern k8s, you could configure Pod Security Admission on a namespace basis. It's not too complex as you just add some annotations to a namespace to activate it. The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.
  - vbezhenar 3y ago
    I'm aware about that, but the problem is: it's just some kind of good will. First of all: you need to actually add those annotations to the namespace. If someone adds his own namespace, it won't be there. Second: anyone can just remove those annotations (without complicated RBAC rules) and do hostpath and other things. I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking. One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it. There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it. https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/ https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about. It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.

ransom1538 3y ago

"Default access even with RBAC is too wide-open." Not following this. Isn't a k8s cluster behind a network firewall? Only exposing 443? To me RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. That sounds awful.

alephnerd 3y ago
Jeff in Engineering might have deployed an app spanning vCenter as well as EKS, and to continue to allow the app to function wrote subpar ingress/egress rules.
mulmen 3y ago
443 is just a port. If we assume that implies HTTPS then that’s still only TLS. If I exploit the service (“securely”!) I am inside your network. I can then pivot to your management layer. Game over.
vbezhenar 3y ago
> Isn't a k8s cluster behind a network firewall? Of course not, unless you configure it this way. > RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. What? How comes no one uses service accounts? They're used extensively for many things.
deleted 3y ago
[deleted]

cratermoon 3y ago

I've seen non-tech corporations adopting Kubernetes without realizing that it's just one part of Google's infrastructure, not the entire operations side. There's an entirely separate chunk of infrastructure that provides the kind of security you mention. K8s is a "do one thing and do it well" that does workload orchestration. Google's BeyondCorp system enforces a zero-trust policy. It's not left to each part of the infrastructure to independently implement.

alfalfasprout 3y ago
Well, Google uses Borg internally which isn't exactly Kubernetes. But yes, the idea is that for it to be useful as infra it has to come without a lot of magic automatically attached.

raesene9 3y ago

Whilst Kubernetes defaults are indeed not great, they aren't that relevant unless you're building your clusters from scratch instead of using a k8s distribution, and I don't think I've seen anyone do that in a long time. RBAC is enabled by default on just about every distribution I've seen (with the notable exception of microk8s) Lack of default admission control to prevent someone with pod creation access breakout to the host is indeed pretty common Unencrypted APIs hasn't really been a problem in years, the insecure API port and read-only Kubelet were a thing but they've been disabled for quite a few versions now. I'd definitely agree that early k8s versions had very bad security, but things have been improving gradually over the versions :)

0xbadcafebee 3y ago
Well first of all, there are still thousands of clusters running with the old insecure defaults, so that's bad. Both GKE and EKS currently have insecure defaults. There's a half dozen ways to take over the entire cluster with the default settings. The unencrypted API port is only this year being deprecated on GKE, I don't know about EKS. Since you can only grant permissions, not deny, most people end up granting too many permissions, defeating the purpose of least privilege. Default permissions for users on GKE/EKS are way too wide open. You can do things you just shouldn't need to do as a regular dev. You shouldn't be able to just deploy a helm chart to a random cluster and then all of a sudden there's an unauthenticated public service on the internet. But that's what every single distribution of K8s gives users by default. Regardless of whether you think that's fine or not, this is how companies get exposed on a regular basis, so it is a real-world persistent security problem that is perpetuated by the defaults.
- raesene9 3y ago
  People not upgrading is indeed a problem but not one the distributions or k8s can particularly address, unless they force upgrade people which with deprecation/removal of APIs doesn't tend to be popular :) The support lifecycle is pretty short, compared to general enterprise software, so unless orgs are happy running unsupported versions of k8s they're on 1.23 for EKS or 1.24 for GKE by this point. I wouldn't have said that EKS, in particular, had very bad defaults, IME it's the best of the managed distributions for security. Any references for 1/2 dozen ways of taking over a cluster from defaults settings? I'd genuinely be interested as I collect information about Kubernetes security, so I'm always on the look-out for more references. for unencrypted API are you thinking of the Insecure API port or read-only Kubelet? For the insecure API port I don't think that's been a thing for a really long time on managed distros. The only time I remember seeing it was on ACS (the predecessor to AKS) where it was exposed to the container network. read-only Kubelet was more of a thing as that was on worker nodes, but it was less risky.