4 ms·

This is really bad. They are running a single K8s cluster and expecting hard multi-tenancy guarantees? All the major clouds use vm boundaries and separate K8s

This is really bad. They are running a single K8s cluster and expecting hard multi-tenancy guarantees? All the major clouds use vm boundaries and separate K8s clusters between customers. Microsoft was similarly bitten a few years ago with one of their function products that expected K8s to be the primary security boundary.

bilekas 2y ago
> They are running a single K8s cluster and expecting hard multi-tenancy guarantees? Maybe I missed something in the article but where are they expecting any hard guarantees. If there is a model being trained for example (running arbitrary code) where does a multi K8 tenency play? The main issue I see is all internal network communication was trusted once behind the proxy/firewall (Istio) but I probably just don't understand k8 clusters too well.
- robertlagrant 2y ago
  Istio is point to point between services. It's not a boundary in the sense you're thinking.
  - bilekas 2y ago
    I will admit I don't know a lot about Kubernets at all, but as I see the Istio is supposed to be the proxy layer between services ? https://istio.io/latest/docs/ops/deployment/architecture/ https://istio.io/latest/docs/ops/deployment/architecture/ Being able to run as the Istio user (1337) renders the proxy itself moot right ?
    - gbrayut 2y ago
      There are a lot of other ways to bypass the Istio sidecar proxy, which is not designed to be a general egress boundary/firewall. See https://blog.howardjohn.info/posts/bypass-egress/ https://blog.howardjohn.info/posts/bypass-egress/
Arbortheus 2y ago
K8S done right is literally designed for multi tenancy. A separate cluster per customer would be insanely costly and terrible for the planet. Maybe in premium products where security is paramount, but a separate cluster per customer is basically setting your money on fire.
- blincoln 2y ago
  Multi-tenant Kubernetes[1] is a strcpy-level footgun IMO. It's perfectly fine as long as everyone involved does everything correctly and never makes a mistake. Kubernetes itself is very complex. The "who needs a UI when you have configuration files and an API?" approach makes it even more opaque to the people who often end up responsible for it. The landscape changes very rapidly.[2] I'd trust Kelsey Hightower to set up a secure multi-tenant deployment, but probably not anyone else. Is it not practical to deploy clusters on top of virtualization? That should make efficient use of hardware while still giving each tenant their own cluster, therefore providing stronger isolation than the typical Kubernetes configuration tends to. [1] I am specifically referring to a Kubernetes deployment where different customers are running custom code on the same underlying hosts. Using Kubernetes to host a service that is multi-tenant at a higher level is not something I would recommend, but it's not as immediately dangerous as a model where customers run container-level custom code. [2] This is not surprising for a relatively new technology, especially one that's as paradigm-shifting as Kubernetes was. But most people are not going to rearchitect and redeploy everything every six months just because the Kubernetes developers decided to replace a pod security or network security model with a non-backward-compatible alternative again.
- outworlder 2y ago
  > K8S done right is literally designed for multi tenancy No it is not. Not in the way they are using it. There are two main use-cases. One is multiple teams, in which case they are bound by their company's policies and guardrails The second is multiple customers. But that also assumes they have no direct access to the cluster. A vendor would, instead, deploy multiple instances of a workload; the customers would not. Straight from the horse's mouth: https://kubernetes.io/docs/concepts/security/multi-tenancy/ https://kubernetes.io/docs/concepts/security/multi-tenancy/ There's also nothing that says that multiple clusters need to be expensive, if they are sized right. They can be as small as you need both in number of instances and instance size. The overhead here is the control plane but, for small clusters, the resource requirements are similarly small. That said, if hard multi tenancy is what you need, then you need to use things like this: https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster https://github.com/kubernetes-sigs/cluster-api-provider-nest... (for the control plane - you still need to worry about the data plane)
outworlder 2y ago
Hard multi tenancy can't realistically be achieved in the same logical K8s cluster. And it is a moving target, which makes trying to secure it by admission controllers... not a great plan. One needs to look into things like VirtualClusters to even begin to consider hard multi-tenancy with potential hostile tenants(https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster https://github.com/kubernetes-sigs/cluster-api-provider-nest...). That is just about the control plane. It doesn't even touch the data plane. How secure that is even with the extra layer, I do not know. Even in the VM land we have seen crazy VM escape exploits over the years.T

bilekas 2y ago

> They are running a single K8s cluster and expecting hard multi-tenancy guarantees? Maybe I missed something in the article but where are they expecting any hard guarantees. If there is a model being trained for example (running arbitrary code) where does a multi K8 tenency play? The main issue I see is all internal network communication was trusted once behind the proxy/firewall (Istio) but I probably just don't understand k8 clusters too well.

robertlagrant 2y ago
Istio is point to point between services. It's not a boundary in the sense you're thinking.
- bilekas 2y ago
  I will admit I don't know a lot about Kubernets at all, but as I see the Istio is supposed to be the proxy layer between services ? https://istio.io/latest/docs/ops/deployment/architecture/ https://istio.io/latest/docs/ops/deployment/architecture/ Being able to run as the Istio user (1337) renders the proxy itself moot right ?
  - gbrayut 2y ago
    There are a lot of other ways to bypass the Istio sidecar proxy, which is not designed to be a general egress boundary/firewall. See https://blog.howardjohn.info/posts/bypass-egress/ https://blog.howardjohn.info/posts/bypass-egress/

Arbortheus 2y ago

K8S done right is literally designed for multi tenancy. A separate cluster per customer would be insanely costly and terrible for the planet. Maybe in premium products where security is paramount, but a separate cluster per customer is basically setting your money on fire.

blincoln 2y ago
Multi-tenant Kubernetes[1] is a strcpy-level footgun IMO. It's perfectly fine as long as everyone involved does everything correctly and never makes a mistake. Kubernetes itself is very complex. The "who needs a UI when you have configuration files and an API?" approach makes it even more opaque to the people who often end up responsible for it. The landscape changes very rapidly.[2] I'd trust Kelsey Hightower to set up a secure multi-tenant deployment, but probably not anyone else. Is it not practical to deploy clusters on top of virtualization? That should make efficient use of hardware while still giving each tenant their own cluster, therefore providing stronger isolation than the typical Kubernetes configuration tends to. [1] I am specifically referring to a Kubernetes deployment where different customers are running custom code on the same underlying hosts. Using Kubernetes to host a service that is multi-tenant at a higher level is not something I would recommend, but it's not as immediately dangerous as a model where customers run container-level custom code. [2] This is not surprising for a relatively new technology, especially one that's as paradigm-shifting as Kubernetes was. But most people are not going to rearchitect and redeploy everything every six months just because the Kubernetes developers decided to replace a pod security or network security model with a non-backward-compatible alternative again.
outworlder 2y ago
> K8S done right is literally designed for multi tenancy No it is not. Not in the way they are using it. There are two main use-cases. One is multiple teams, in which case they are bound by their company's policies and guardrails The second is multiple customers. But that also assumes they have no direct access to the cluster. A vendor would, instead, deploy multiple instances of a workload; the customers would not. Straight from the horse's mouth: https://kubernetes.io/docs/concepts/security/multi-tenancy/ https://kubernetes.io/docs/concepts/security/multi-tenancy/ There's also nothing that says that multiple clusters need to be expensive, if they are sized right. They can be as small as you need both in number of instances and instance size. The overhead here is the control plane but, for small clusters, the resource requirements are similarly small. That said, if hard multi tenancy is what you need, then you need to use things like this: https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster https://github.com/kubernetes-sigs/cluster-api-provider-nest... (for the control plane - you still need to worry about the data plane)

outworlder 2y ago

Hard multi tenancy can't realistically be achieved in the same logical K8s cluster. And it is a moving target, which makes trying to secure it by admission controllers... not a great plan. One needs to look into things like VirtualClusters to even begin to consider hard multi-tenancy with potential hostile tenants(https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster https://github.com/kubernetes-sigs/cluster-api-provider-nest...). That is just about the control plane. It doesn't even touch the data plane. How secure that is even with the extra layer, I do not know. Even in the VM land we have seen crazy VM escape exploits over the years.T