4 ms·
Agreed. Separate prod clusters per product usually fixes an organizational problem (lack of trust that running shared workloads would/could be safe, versus givi
by solatic 3y ago
Agreed. Separate prod clusters per product usually fixes an organizational problem (lack of trust that running shared workloads would/could be safe, versus giving each team their own servers) first. A lot of organizations, sadly, prefer to pay for separate clusters than to set up RBAC, ResourceQuotas, etc.
Hypothetically, since the scaling limit for Kubernetes clusters is ~10,000 nodes (last I checked), you could have multiple product lines that took up 10k+ nodes each. Then there's no reason why not to split by product line. But in the beginning it should be fine.
There's also edge cases - Kubernetes doesn't support setting resource requests or limits for networking or I/O, which you usually solve by setting up taints/tolerations/affinity to manually schedule those workloads onto nodes where you've manually run the numbers. But still not usually a reason to prefer separate production clusters.
- p_l 3y agoArguably, the design of kubernetes is for clusters to be based on physical clusters - to the point of a cluster per aisle (or multiple aisles), and resources of those clusters then being used to deploy applications even across clusters. Or in the small, like Chick-Fil-A, with kubernetes running locally at every restaurant.
- solatic 3y agoYup, this is how you scale an application past the 10k node soft limit.
- p_l 3y agoNot just past 10k node limit - it's also because the k8s design has unsaid assumption about being used to arrange physical sets of machines, similar to how Borg paper would talk about cluster being equivalent to an aisle in datacenter.
- solatic 3y agoAre you just referring to how you assign labels and taints to nodes, which in a physical datacenter could include labels like which aisle the baremetal server is in? Of course that's feasible, to tell the Kubernetes scheduler that you only want to schedule the workload on a very specific server, but that's not really best-practice...
- p_l 3y agoNo, I'm referring to bundling machines into clusters across physical lines, with bare metal deployments.