4 ms·

Has Kubernetes grown past its limit of ~100 underlying servers? What's the largest production Kubernetes cluster running today?

by flyt 11y ago

philips 11y ago
Yes, the community is working hard to make k8s scale to 1000s of machines and beyond. There is a great blog post about the engineering effort from the various communities (k8s, etcd) to make this happen: http://blog.kubernetes.io/2015/09/kubernetes-performance-measurements-and.html http://blog.kubernetes.io/2015/09/kubernetes-performance-mea... In fact someone recently made a video showing a 1000 machine cluster; let me find that. UPDATE: here is the video https://www.youtube.com/watch?v=sgWv9sVTIYQ https://www.youtube.com/watch?v=sgWv9sVTIYQ
- flyt 11y ago
  The last time that I saw somebody from Google give a presentation about k8 it didn't seem like "larger cluster sizes" was a real priority at this point, and they were more focused on features. Good to hear this is getting some attention, since it was the major issue that made our team dismiss it as useful in production.
  - vruiz 11y ago
    May I ask at which scale do you operate? Keep in mind that is not necessary (and I believe not advised[1]) to keep all your infrastructure in one kubernetes cluster. [1] https://youtu.be/-8aUxpVrD40?t=3105 https://youtu.be/-8aUxpVrD40?t=3105
  - jcastro 11y ago
    I've heard googlers mention that same thing, but now that I think about it I believe they were meaning to say that "for 1.0" they were targeting ~100 nodes, and once they got out there door they would start tackling the large cluster case, which appears to be progressing nicely.
- merb 11y ago
  why don't they add something like master slaves to make that happen? i mean 1000s machines for a single master is way more than most software will handle anyway, still kubernetes makes it easy to manage it, however even 50 machines are working really fine on kubernetes and that even with a slower master server. currently we run with 20s at production with a master at 4 gigs which works really really well.
  - robszumski 11y ago
    There is an open PR to this project to document an HA control plane. Basically, you add the kube-podmaster onto each master machine, which does a leader election for the API, controller manager, etc so that only one instance is mutating the cluster. If that does down, another leader election happens.
  - bob_wise 11y ago
    The "master" in kubernetes is a collection of components that can be distributed. The kubernetes component that is the most loaded in a 1000 node system is the api server. The 1000 node cluster config in that video is running 10 api servers. I'm the author of that video, BTW, so I know that for sure. :-)
- bob_wise 11y ago
  Here's an updated video with a voiceover... https://www.youtube.com/watch?v=1bHUsBhPL20 https://www.youtube.com/watch?v=1bHUsBhPL20
Sym3tri 11y ago
Support for 250 nodes is scheduled for October. https://github.com/kubernetes/kubernetes/blob/master/docs/roadmap.md#kubernetes-11 https://github.com/kubernetes/kubernetes/blob/master/docs/ro...

Has Kubernetes grown past its limit of ~100 underlying servers? What's the largest production Kubernetes cluster running today?

by flyt 11y ago

Has Kubernetes grown past its limit of ~100 underlying servers? What's the largest production Kubernetes cluster running today?

philips 11y ago

Yes, the community is working hard to make k8s scale to 1000s of machines and beyond. There is a great blog post about the engineering effort from the various communities (k8s, etcd) to make this happen: http://blog.kubernetes.io/2015/09/kubernetes-performance-measurements-and.html http://blog.kubernetes.io/2015/09/kubernetes-performance-mea... In fact someone recently made a video showing a 1000 machine cluster; let me find that. UPDATE: here is the video https://www.youtube.com/watch?v=sgWv9sVTIYQ https://www.youtube.com/watch?v=sgWv9sVTIYQ

flyt 11y ago
The last time that I saw somebody from Google give a presentation about k8 it didn't seem like "larger cluster sizes" was a real priority at this point, and they were more focused on features. Good to hear this is getting some attention, since it was the major issue that made our team dismiss it as useful in production.
- vruiz 11y ago
  May I ask at which scale do you operate? Keep in mind that is not necessary (and I believe not advised[1]) to keep all your infrastructure in one kubernetes cluster. [1] https://youtu.be/-8aUxpVrD40?t=3105 https://youtu.be/-8aUxpVrD40?t=3105
- jcastro 11y ago
  I've heard googlers mention that same thing, but now that I think about it I believe they were meaning to say that "for 1.0" they were targeting ~100 nodes, and once they got out there door they would start tackling the large cluster case, which appears to be progressing nicely.
merb 11y ago
why don't they add something like master slaves to make that happen? i mean 1000s machines for a single master is way more than most software will handle anyway, still kubernetes makes it easy to manage it, however even 50 machines are working really fine on kubernetes and that even with a slower master server. currently we run with 20s at production with a master at 4 gigs which works really really well.
- robszumski 11y ago
  There is an open PR to this project to document an HA control plane. Basically, you add the kube-podmaster onto each master machine, which does a leader election for the API, controller manager, etc so that only one instance is mutating the cluster. If that does down, another leader election happens.
- bob_wise 11y ago
  The "master" in kubernetes is a collection of components that can be distributed. The kubernetes component that is the most loaded in a 1000 node system is the api server. The 1000 node cluster config in that video is running 10 api servers. I'm the author of that video, BTW, so I know that for sure. :-)
bob_wise 11y ago
Here's an updated video with a voiceover... https://www.youtube.com/watch?v=1bHUsBhPL20 https://www.youtube.com/watch?v=1bHUsBhPL20

Sym3tri 11y ago

Support for 250 nodes is scheduled for October. https://github.com/kubernetes/kubernetes/blob/master/docs/roadmap.md#kubernetes-11 https://github.com/kubernetes/kubernetes/blob/master/docs/ro...