4 ms·

What's the story of multitenancy in Kubernetes? Is it true that it does not provide sufficient isolation between services by default? Is this something that cou

What's the story of multitenancy in Kubernetes? Is it true that it does not provide sufficient isolation between services by default? Is this something that could be handled by configuration only or is it a bigger issue with Kubernetes design? What's the best way to make Kubernetes the layer that isolates tenants in a multitenant platform?

gtirloni 7y ago
What's sufficient isolation? It has pods, pod security policies, network policies, maybe even hypervisor instead of Docker, etc. It's pretty much built for multitenancy.
ptrincr 7y ago
From a service point of view, you should be able to provide isolation between services using namespaces, istio and Calico. You can read about "Soft multi-tenancy" here: https://istio.io/blog/2018/soft-multitenancy/ https://istio.io/blog/2018/soft-multitenancy/ That works well to isolate services/apps/teams from each other, but also allows services to be shared where required. I think if you wanted true multitenancy it would be simpler to spin up a second cluster, especially if you are in gcp and controlling everything using a infrastructure as code solution such as terraform.
nielsole 7y ago
File descriptors and inotify watches can still be exhausted. In somewhat older versions also pids. If you allocate memory fast enough(if you're a best effort pod), oomkiller instead of kubelet evictions happen, which can be kind of unpredictable. Similarly you can fill up storage. You don't want to be colocated with bad pods.
meddlepal 7y ago
It provides soft multi tenancy via namespaces. You can then use NetworkPolicy if the CNI supports it to manage what other stuff can communicate with the Pods in that namespace. You can also schedule Pods on Nodes that are for a specific purpose (eg only used by a specific team or product) by using various scheduler stuff like taints and affinity/anti-affinity. Or you can write a custom scheduler. In general I feel like all the tools are there in Kubernetes to build a good multitenant experience but it's also just easier to run a number of clusters, especially if you're cloud hosting where the control plane is usually free and you pay only for the workers. I would though ask, what is sufficient isolation?
- rtempaccount1 7y ago
  I'd say the amount of isolation to be sufficient, depends on the threat model. If you want some isolation but moderately trusted workloads (e.g. different departments in the same company) that's easier than the isolation required for "arbitrary customers can run any workload on a given cluster" Multi-tenant, single cluster is tricky. You either need node pools per tenant, at which point why not run one cluster per tenant, or you need really good isolation of the workloads from the nodes, which requires PodSecurityPolicy or similar to prevent a breakout to the underlying node. Also RBAC can be tricky in a multi-tenant setup as you need to be very careful with anything that would allow breakout to the underlying node or any cluster level rights.
smarterclayton 7y ago
As some of the other comments have mentioned, kubernetes is “multitenant capable”, but the questions come down to: 1. Do you understand all the primitives and how they interact 2. Do your use cases require some or all of these primitives be bypassed (host network, root containers, custom extensions / operators), which may invalidate most security 3. Are the human costs to manage 1 and 2 less than the human/infra costs to manage multiple clusters? Many applications have a few assumptions that violate “safety on a single machine” - databases use raw block devices, processes run as root by default. The harder it becomes to isolate an app on a single machine, the harder it is to isolate that workload on a cluster. In addition, the moment you install something above kube (helm, ingress controllers, operators that can read or write secrets across namespaces) you start to blur the security boundaries at a cluster level and thus make it harder to reason about who can do what. It’s a lot like the Linux kernel / userspace separation - everything secure is in the kernel, but if a driver or subsystem has a vulnerability that tends to compromise the whole kernel. I would suggest most people run multiple clusters until you get to the point where you have cluster sprawl (and are paying for a ton of redundant / idle infra), and then start either with an opinionated distro designed for picking all the defaults for you (OpenShift, a few of the other vendors) or hire someone who can design a top to bottom policy (and even then you’ll want multiple clusters for failure isolation).
- alexellisuk 7y ago
  I would agree with this. Soft multi-tenancy can work for some setups, until the tenant has a business need for something which could introduce a vulnerability and compromise nodes.
- drondin 7y ago
  I mostly agree with you that using current tooling seems like patching holes constantly but it's getting much better with sandboxing and automation is key and of course, it's not for everyone but can provide lots of benefits with "low" effort. There is a Kubernetes Working Group where the definition and implementation of soft multi-tenancy and any feedback is welcome.
jacques_chester 7y ago
The answers you'll get will be some combination of "namespaces" plus an ever-expanding laundry list of configurations and knobs and dials and features and webhooks and options ... which you will need to apply uniformly and correctly. To all your workloads. Kubernetes was simply not designed as a multitenant system. Namespaces aren't the basis of multitenancy, for pete's sake, they're namespaces. Piling ever more extensions on top to try and plug all the holes just highlights it more completely. The concept given above is where I expect Kubernetes will go, either explicitly (by creating a hard separation between scheduling and everything else) or by convention and tooling (we all adopt layers of indirection like Virtual Kubernetes). The latter seems most likely to me. This will, as others are pointing out, only get more and more necessary as CRDs and Operators continue their long march to gather all the knowledge of your systems into their embrace. But it turns out that storing all the truths of dozens or hundreds of distinct systems, some of which conflict, many of which can't be fully walled off from others, means that your security posture may include a certain amount of humming loudly. Something like Virtual Kubernetes splits the difference: you can retain Kubernetes-as-scheduler, reaping all the benefits accruing from workload consolidation. But you also get hard tenancy boundaries between workspaces, between Kubernetes masters being used as a skin over etcd. Kubernetes is amazing. But this is one thing it got wrong. Just wrong and we'll all be wrestling with it for years. It would be like setting out to write a single-user OS, database or filesystem. Disclosure: I work for Pivotal, we do some Kubernetes stuff, but these are personal views.
- polskibus 7y ago
  What's the best alternative to Kubernetes if I really need to achieve multitenancy, perhaps at some cost of something else.that Kubernetes got right?
  - zaat 7y ago
    I'll go with separate Kubernetes clusters. The cost is more systems to manage, but they will be simpler and pretty much identical, and some low but not zero resource waste with each cluster you add. Ill add that there are few well established companies that have adopted this approach, AWS, Google and Azure to name a few.

gtirloni 7y ago

What's sufficient isolation? It has pods, pod security policies, network policies, maybe even hypervisor instead of Docker, etc. It's pretty much built for multitenancy.

ptrincr 7y ago

From a service point of view, you should be able to provide isolation between services using namespaces, istio and Calico. You can read about "Soft multi-tenancy" here: https://istio.io/blog/2018/soft-multitenancy/ https://istio.io/blog/2018/soft-multitenancy/ That works well to isolate services/apps/teams from each other, but also allows services to be shared where required. I think if you wanted true multitenancy it would be simpler to spin up a second cluster, especially if you are in gcp and controlling everything using a infrastructure as code solution such as terraform.

nielsole 7y ago

File descriptors and inotify watches can still be exhausted. In somewhat older versions also pids. If you allocate memory fast enough(if you're a best effort pod), oomkiller instead of kubelet evictions happen, which can be kind of unpredictable. Similarly you can fill up storage. You don't want to be colocated with bad pods.

meddlepal 7y ago

It provides soft multi tenancy via namespaces. You can then use NetworkPolicy if the CNI supports it to manage what other stuff can communicate with the Pods in that namespace. You can also schedule Pods on Nodes that are for a specific purpose (eg only used by a specific team or product) by using various scheduler stuff like taints and affinity/anti-affinity. Or you can write a custom scheduler. In general I feel like all the tools are there in Kubernetes to build a good multitenant experience but it's also just easier to run a number of clusters, especially if you're cloud hosting where the control plane is usually free and you pay only for the workers. I would though ask, what is sufficient isolation?

rtempaccount1 7y ago
I'd say the amount of isolation to be sufficient, depends on the threat model. If you want some isolation but moderately trusted workloads (e.g. different departments in the same company) that's easier than the isolation required for "arbitrary customers can run any workload on a given cluster" Multi-tenant, single cluster is tricky. You either need node pools per tenant, at which point why not run one cluster per tenant, or you need really good isolation of the workloads from the nodes, which requires PodSecurityPolicy or similar to prevent a breakout to the underlying node. Also RBAC can be tricky in a multi-tenant setup as you need to be very careful with anything that would allow breakout to the underlying node or any cluster level rights.

smarterclayton 7y ago

As some of the other comments have mentioned, kubernetes is “multitenant capable”, but the questions come down to: 1. Do you understand all the primitives and how they interact 2. Do your use cases require some or all of these primitives be bypassed (host network, root containers, custom extensions / operators), which may invalidate most security 3. Are the human costs to manage 1 and 2 less than the human/infra costs to manage multiple clusters? Many applications have a few assumptions that violate “safety on a single machine” - databases use raw block devices, processes run as root by default. The harder it becomes to isolate an app on a single machine, the harder it is to isolate that workload on a cluster. In addition, the moment you install something above kube (helm, ingress controllers, operators that can read or write secrets across namespaces) you start to blur the security boundaries at a cluster level and thus make it harder to reason about who can do what. It’s a lot like the Linux kernel / userspace separation - everything secure is in the kernel, but if a driver or subsystem has a vulnerability that tends to compromise the whole kernel. I would suggest most people run multiple clusters until you get to the point where you have cluster sprawl (and are paying for a ton of redundant / idle infra), and then start either with an opinionated distro designed for picking all the defaults for you (OpenShift, a few of the other vendors) or hire someone who can design a top to bottom policy (and even then you’ll want multiple clusters for failure isolation).

alexellisuk 7y ago
I would agree with this. Soft multi-tenancy can work for some setups, until the tenant has a business need for something which could introduce a vulnerability and compromise nodes.
drondin 7y ago
I mostly agree with you that using current tooling seems like patching holes constantly but it's getting much better with sandboxing and automation is key and of course, it's not for everyone but can provide lots of benefits with "low" effort. There is a Kubernetes Working Group where the definition and implementation of soft multi-tenancy and any feedback is welcome.

jacques_chester 7y ago

The answers you'll get will be some combination of "namespaces" plus an ever-expanding laundry list of configurations and knobs and dials and features and webhooks and options ... which you will need to apply uniformly and correctly. To all your workloads. Kubernetes was simply not designed as a multitenant system. Namespaces aren't the basis of multitenancy, for pete's sake, they're namespaces. Piling ever more extensions on top to try and plug all the holes just highlights it more completely. The concept given above is where I expect Kubernetes will go, either explicitly (by creating a hard separation between scheduling and everything else) or by convention and tooling (we all adopt layers of indirection like Virtual Kubernetes). The latter seems most likely to me. This will, as others are pointing out, only get more and more necessary as CRDs and Operators continue their long march to gather all the knowledge of your systems into their embrace. But it turns out that storing all the truths of dozens or hundreds of distinct systems, some of which conflict, many of which can't be fully walled off from others, means that your security posture may include a certain amount of humming loudly. Something like Virtual Kubernetes splits the difference: you can retain Kubernetes-as-scheduler, reaping all the benefits accruing from workload consolidation. But you also get hard tenancy boundaries between workspaces, between Kubernetes masters being used as a skin over etcd. Kubernetes is amazing. But this is one thing it got wrong. Just wrong and we'll all be wrestling with it for years. It would be like setting out to write a single-user OS, database or filesystem. Disclosure: I work for Pivotal, we do some Kubernetes stuff, but these are personal views.

polskibus 7y ago
What's the best alternative to Kubernetes if I really need to achieve multitenancy, perhaps at some cost of something else.that Kubernetes got right?
- zaat 7y ago
  I'll go with separate Kubernetes clusters. The cost is more systems to manage, but they will be simpler and pretty much identical, and some low but not zero resource waste with each cluster you add. Ill add that there are few well established companies that have adopted this approach, AWS, Google and Azure to name a few.