4 ms·

Is Kubernetes really scalable to a meaningful extent? I feel like if I was going to set up a Kubernetes system, I'd need to plan from the beginning to have mult

Is Kubernetes really scalable to a meaningful extent? I feel like if I was going to set up a Kubernetes system, I'd need to plan from the beginning to have multiple clusters anyway, and then the utility of all the scheduling features would be considerably diminished since I'd have to plan to load balance applications across the clusters in some custom way. Yuck. The Kubernetes website (1) currently claims that it supports clusters of up to 5000 nodes, which is a decent amount but not enough to avoid having multiple clusters. Does anyone have experience operating multiple production clusters in a single territory as partitions for scaling reasons? What's the experience like? (1) https://kubernetes.io/docs/admin/cluster-large/ https://kubernetes.io/docs/admin/cluster-large/

jrs95 8y ago
If you have a need for more than 5000 nodes in a cluster, it should take a trivial amount of your resources to solve that problem IMO. The vast majority of Kuberentes users probably won’t even need a 500 node cluster, let alone a 5000 node cluster. It makes sense for them to prioritize optimizing for other things like developer experience to help address some of the concerns mentioned in this article over supporting even larger clusters right now.
rickycook 8y ago
nodes meaning compute nodes; not containers in case there was confusion
manigandham 8y ago
5000 nodes has proven to be a high enough ceiling for basically all workloads, especially given the size of individual servers available now in clouds. Easier to run smaller numbers of bigger servers, as always. Also Kubernetes does support federation for cross-cluster deployments (now named multi-cluster). Some cloud services even support ingress load balancing across these, or you can do that part yourself by simple sticking with the same ports, but it all works fine today. Nothing custom needed.
geerlingguy 8y ago
The other important distinction there is the current (soft-ish) limit of 110 pods per node; if you want to run hundreds or thousands of pods on a few giant (beefy CPU/RAM) nodes, you need to rethink that strategy and go with many smaller nodes, with the pods evenly spread across them.
ec109685 8y ago
Assuming C5 reserved instances, 5000 of those nodes is: $3 * 24 * 365 * 5000 * .5 (reserved discount) or $68M a year per aws region and 72 * 5000 or 360k cores. That is pretty scalable.

jrs95 8y ago

If you have a need for more than 5000 nodes in a cluster, it should take a trivial amount of your resources to solve that problem IMO. The vast majority of Kuberentes users probably won’t even need a 500 node cluster, let alone a 5000 node cluster. It makes sense for them to prioritize optimizing for other things like developer experience to help address some of the concerns mentioned in this article over supporting even larger clusters right now.

rickycook 8y ago

nodes meaning compute nodes; not containers in case there was confusion

manigandham 8y ago

5000 nodes has proven to be a high enough ceiling for basically all workloads, especially given the size of individual servers available now in clouds. Easier to run smaller numbers of bigger servers, as always. Also Kubernetes does support federation for cross-cluster deployments (now named multi-cluster). Some cloud services even support ingress load balancing across these, or you can do that part yourself by simple sticking with the same ports, but it all works fine today. Nothing custom needed.

geerlingguy 8y ago

The other important distinction there is the current (soft-ish) limit of 110 pods per node; if you want to run hundreds or thousands of pods on a few giant (beefy CPU/RAM) nodes, you need to rethink that strategy and go with many smaller nodes, with the pods evenly spread across them.

ec109685 8y ago

Assuming C5 reserved instances, 5000 of those nodes is: $3 * 24 * 365 * 5000 * .5 (reserved discount) or $68M a year per aws region and 72 * 5000 or 360k cores. That is pretty scalable.