5 ms·

What's the best alternative to Kubernetes if I really need to achieve multitenancy, perhaps at some cost of something else.that Kubernetes got right?

zaat 7y ago
I'll go with separate Kubernetes clusters. The cost is more systems to manage, but they will be simpler and pretty much identical, and some low but not zero resource waste with each cluster you add. Ill add that there are few well established companies that have adopted this approach, AWS, Google and Azure to name a few.
- polskibus 7y ago
  In this approach, is a single VM node reused by multiple kubernetes clusters or is a node exclusive to a single cluster? If only the latter is possible then what is the best way to maximize resource utilization in cluster-per-tenant scenario? What happens if new customer wants his tenant - is automating of provisioning of an entire new cluster feasible?
  - zaat 7y ago
    Each vm belongs to a single cluster. The best way to maximimize resource utilization is to size the vms according to the tenant load, generally you want to minimize the number of vms, as each of them add some overhead, but you don't want to waste unnecessary resources. You must have a minimum set of vms in each cluster to ensure high availability. Automating the provisioning of an entire new cluster is very easy, at least for some people with the required resources. But even for a small shop it is possible to automate it using stuff like Jenkins and Ansible. It is still relatively sophisticated stuff. It is not rocket science but not an elementary school project level either.
    - polskibus 7y ago
      Thanks, that was informative. Do you have a reference that you'd recommend about the details of how the big names that you've mentioned are doing multitenant Kubernetes? I'd love to read more about it.
      - zaat 7y ago
        Oh, sorry, that was tongue in the cheek. They are cloud providers, and as such they provide costumers with separate cluster. The aks-engine repo let you peep to what goes behind the scenes, but this is not the kind of beast you will grow at home - it is a public cloud provider with global scale, with the expected level of complexity and engineering. It is available on: https://github.com/Azure/aks-engine https://github.com/Azure/aks-engine
      - merb 7y ago
        btw. in hard multitenant k8s you can even go as wide as using a "master" cluster that contains the control plane of another cluster. actually you need the right network cni for that.
  - jacques_chester 7y ago
    > What happens if new customer wants his tenant - is automating of provisioning of an entire new cluster feasible? This is doable through various provider/vendor-specific APIs. Pivotal's PKS has an API. GKE has one, EKS, AKS and so on have them too. In the longer run there's meant to be a full Cluster API, which I presume will take over for that role: https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api
jacques_chester 7y ago
For right now? Serious answer? Cloud Foundry. Pretty much everything you need to do today is done. Multitenancy was designed in from the start and is available in several levels of hardness (orgs/spaces gives you user-level multitenancy, isolation segments give you component-level multitenancy, multi-foundation gives you the equivalent of multi-cluster multitenancy). As time goes on it will adopt more and more of Kubernetes, Istio and other systems as they emerge and become able to take over from its own original components. But you'll get the same experience throughout. And Kubernetes will converge upwards as people realise that "It Just Works" is actually, you know, a pretty nice developer experience. But I am, having worked on and around Cloud Foundry for several years, hilariously biased.
- Terretta 7y ago
  Instead of K8s or Cloud Foundry, a practical “alternative, to really achieve multi-tenancy” might be: - Since essentially no container platform was designed for hard multi-tenancy, use automation to stand up separate copies of a platform at scale, using a platform with low overhead in controller to worker nodes. Consider what would be the “simplest thing that could possibly work” e.g. offering tenants an AWS autoscale group (with appropriate IAM account, user, role, security group perms) behind an AWS PrivateLink endpoint: https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-service.html https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-se... I say that, because most of these platforms weren’t intended for and are not marketed for hard multi-tenancy, until their engineers started getting asked the question by enterprises attempting it for regulatory/compliance/security reasons. > Cloud Foundry. Multitenancy was designed in from the start... This comment gives me pause, as CloudFoundry’s security model, approach, and engineering lacked hardness as of 3 - 4 years ago. We explored the multi-tenancy design, and found no guarantees, or even strong assurances, were brought by the platform. The larger issue was we found CF’s security engineering assertions rather overconfident (flavor of the comment above). Warranted relative to other solutions without security features, but not relative to being hardened enough to trust in the face of bad actors, particularly malicious insiders. My opinion is also “biased”, in the sense that unless a platform is doing work outside of “the server”, I don’t believe it can offer hard multi-tenancy. Here’s an example of work outside the server: https://perspectives.mvdirona.com/2019/02/aws-nitro-system/ https://perspectives.mvdirona.com/2019/02/aws-nitro-system/
tedmiston 7y ago
Additional Kubernetes clusters.
auspex 7y ago
1) Separate clusters 2) Service mocrosegmentation 3) 2 but using kubernetes network policies

zaat 7y ago

I'll go with separate Kubernetes clusters. The cost is more systems to manage, but they will be simpler and pretty much identical, and some low but not zero resource waste with each cluster you add. Ill add that there are few well established companies that have adopted this approach, AWS, Google and Azure to name a few.

polskibus 7y ago
In this approach, is a single VM node reused by multiple kubernetes clusters or is a node exclusive to a single cluster? If only the latter is possible then what is the best way to maximize resource utilization in cluster-per-tenant scenario? What happens if new customer wants his tenant - is automating of provisioning of an entire new cluster feasible?
- zaat 7y ago
  Each vm belongs to a single cluster. The best way to maximimize resource utilization is to size the vms according to the tenant load, generally you want to minimize the number of vms, as each of them add some overhead, but you don't want to waste unnecessary resources. You must have a minimum set of vms in each cluster to ensure high availability. Automating the provisioning of an entire new cluster is very easy, at least for some people with the required resources. But even for a small shop it is possible to automate it using stuff like Jenkins and Ansible. It is still relatively sophisticated stuff. It is not rocket science but not an elementary school project level either.
  - polskibus 7y ago
    Thanks, that was informative. Do you have a reference that you'd recommend about the details of how the big names that you've mentioned are doing multitenant Kubernetes? I'd love to read more about it.
    - zaat 7y ago
      Oh, sorry, that was tongue in the cheek. They are cloud providers, and as such they provide costumers with separate cluster. The aks-engine repo let you peep to what goes behind the scenes, but this is not the kind of beast you will grow at home - it is a public cloud provider with global scale, with the expected level of complexity and engineering. It is available on: https://github.com/Azure/aks-engine https://github.com/Azure/aks-engine
    - merb 7y ago
      btw. in hard multitenant k8s you can even go as wide as using a "master" cluster that contains the control plane of another cluster. actually you need the right network cni for that.
- jacques_chester 7y ago
  > What happens if new customer wants his tenant - is automating of provisioning of an entire new cluster feasible? This is doable through various provider/vendor-specific APIs. Pivotal's PKS has an API. GKE has one, EKS, AKS and so on have them too. In the longer run there's meant to be a full Cluster API, which I presume will take over for that role: https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api

jacques_chester 7y ago

For right now? Serious answer? Cloud Foundry. Pretty much everything you need to do today is done. Multitenancy was designed in from the start and is available in several levels of hardness (orgs/spaces gives you user-level multitenancy, isolation segments give you component-level multitenancy, multi-foundation gives you the equivalent of multi-cluster multitenancy). As time goes on it will adopt more and more of Kubernetes, Istio and other systems as they emerge and become able to take over from its own original components. But you'll get the same experience throughout. And Kubernetes will converge upwards as people realise that "It Just Works" is actually, you know, a pretty nice developer experience. But I am, having worked on and around Cloud Foundry for several years, hilariously biased.

Terretta 7y ago
Instead of K8s or Cloud Foundry, a practical “alternative, to really achieve multi-tenancy” might be: - Since essentially no container platform was designed for hard multi-tenancy, use automation to stand up separate copies of a platform at scale, using a platform with low overhead in controller to worker nodes. Consider what would be the “simplest thing that could possibly work” e.g. offering tenants an AWS autoscale group (with appropriate IAM account, user, role, security group perms) behind an AWS PrivateLink endpoint: https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-service.html https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-se... I say that, because most of these platforms weren’t intended for and are not marketed for hard multi-tenancy, until their engineers started getting asked the question by enterprises attempting it for regulatory/compliance/security reasons. > Cloud Foundry. Multitenancy was designed in from the start... This comment gives me pause, as CloudFoundry’s security model, approach, and engineering lacked hardness as of 3 - 4 years ago. We explored the multi-tenancy design, and found no guarantees, or even strong assurances, were brought by the platform. The larger issue was we found CF’s security engineering assertions rather overconfident (flavor of the comment above). Warranted relative to other solutions without security features, but not relative to being hardened enough to trust in the face of bad actors, particularly malicious insiders. My opinion is also “biased”, in the sense that unless a platform is doing work outside of “the server”, I don’t believe it can offer hard multi-tenancy. Here’s an example of work outside the server: https://perspectives.mvdirona.com/2019/02/aws-nitro-system/ https://perspectives.mvdirona.com/2019/02/aws-nitro-system/

tedmiston 7y ago

Additional Kubernetes clusters.

auspex 7y ago

1) Separate clusters 2) Service mocrosegmentation 3) 2 but using kubernetes network policies