5 ms·

Kubernetes can't (currently) scale to Google sizes. It's designed for small- or medium-sized businesses, which might have 50,000 VMs or fewer. There are entire

Kubernetes can't (currently) scale to Google sizes. It's designed for small- or medium-sized businesses, which might have 50,000 VMs or fewer. There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter.

hpkuarg 4y ago
> There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter. Forget a whole datacenter, even just one rack is an unimaginable amount of computing power, these days!
- dublin 4y ago
  Fact: An well-equipped Raspberry Pi 4 has more memory, more compute power, more storage and vastly faster networking than the Cray supercomputer I worked with at a major oil company in the 1990's! With the exception of the craziness around click-tracking (web ads and marketing have warped compute use even more than crypto), the data required to run even enterprise-scale businesses today is not really all that large. For almost all purposes, we don't really need thousands of containers running on unimaginably fast computers, coordinated by AI-driven automation systems. What we need is software that is not morbidly obese.
  - gh02t 4y ago
    I mean a single V100 GPU (~= 100 TFLOPS) has FLOPS throughputs similar to top end mid-00's supercomputers like Blue Gene, at least superficially. And you can squeeze like 4+ in a 1U if you have enough cooling and power. https://en.wikipedia.org/wiki/History_of_supercomputing https://en.wikipedia.org/wiki/History_of_supercomputing
blue_cookeh 4y ago
> small- or medium-sized businesses, which might have 50,000 VMs or fewer Holy shit, is this considered small to medium enterprise now?
- dijit 4y ago
  a single cluster supports approx 5,000 nodes, 110 pods per node~ The estimated maximum single cluster is 300,000 containers. That's pretty medium, I've ran more than a million processes before, and nomad has 1million containers as its challenge https://www.hashicorp.com/c1m https://www.hashicorp.com/c1m borg can handle this easily.
  - dlp211 4y ago
    I don't know anyone running 5k node K8S clusters. That said, borg (as an end user) appears to just keep scaling, but it uses a different model and makes different assumptions than K8S.
    - dijit 4y ago
      Sure. Most people have more clusters before they hit 5k nodes on a single cluster. But I’ve been in situations where it would have been worthwhile. I’ve been in situations with 30,000 machines that needed to be controlled. Splitting them out into very many clusters would be a lot of wasted overhead in configuration, administration and because you lose nodes to masters.
ec109685 4y ago
That’s a bit out of date. K8s can do 5,000 nodes and 300k VMs within its performance envelope: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg...
- jmillikin 4y ago
  A VM would be a node. A pod isn't a VM, it's a process tree.
drdaeman 4y ago
I'm afraid this is why people pick Kubernetes. They believe small business needs those tens of thousands of VMs distributed across thousand of nodes and so on. With some exceptions, I believe that's a few orders of magnitude above what a small business can run on. Nowadays people just start their day by drinking some K2l-aid and spinning up a "basic" 6-node cluster for a development prototype. Maybe I'm wrong, of course.
bg24 4y ago
Your scale (50,000 VMs) is way too high for small and medium sized business :-) Is my observation correct that unicorns start to see that scale?

hpkuarg 4y ago

> There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter. Forget a whole datacenter, even just one rack is an unimaginable amount of computing power, these days!

dublin 4y ago
Fact: An well-equipped Raspberry Pi 4 has more memory, more compute power, more storage and vastly faster networking than the Cray supercomputer I worked with at a major oil company in the 1990's! With the exception of the craziness around click-tracking (web ads and marketing have warped compute use even more than crypto), the data required to run even enterprise-scale businesses today is not really all that large. For almost all purposes, we don't really need thousands of containers running on unimaginably fast computers, coordinated by AI-driven automation systems. What we need is software that is not morbidly obese.
- gh02t 4y ago
  I mean a single V100 GPU (~= 100 TFLOPS) has FLOPS throughputs similar to top end mid-00's supercomputers like Blue Gene, at least superficially. And you can squeeze like 4+ in a 1U if you have enough cooling and power. https://en.wikipedia.org/wiki/History_of_supercomputing https://en.wikipedia.org/wiki/History_of_supercomputing

blue_cookeh 4y ago

> small- or medium-sized businesses, which might have 50,000 VMs or fewer Holy shit, is this considered small to medium enterprise now?

dijit 4y ago
a single cluster supports approx 5,000 nodes, 110 pods per node~ The estimated maximum single cluster is 300,000 containers. That's pretty medium, I've ran more than a million processes before, and nomad has 1million containers as its challenge https://www.hashicorp.com/c1m https://www.hashicorp.com/c1m borg can handle this easily.
- dlp211 4y ago
  I don't know anyone running 5k node K8S clusters. That said, borg (as an end user) appears to just keep scaling, but it uses a different model and makes different assumptions than K8S.
  - dijit 4y ago
    Sure. Most people have more clusters before they hit 5k nodes on a single cluster. But I’ve been in situations where it would have been worthwhile. I’ve been in situations with 30,000 machines that needed to be controlled. Splitting them out into very many clusters would be a lot of wasted overhead in configuration, administration and because you lose nodes to masters.

ec109685 4y ago

That’s a bit out of date. K8s can do 5,000 nodes and 300k VMs within its performance envelope: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg...

jmillikin 4y ago
A VM would be a node. A pod isn't a VM, it's a process tree.

drdaeman 4y ago

I'm afraid this is why people pick Kubernetes. They believe small business needs those tens of thousands of VMs distributed across thousand of nodes and so on. With some exceptions, I believe that's a few orders of magnitude above what a small business can run on. Nowadays people just start their day by drinking some K2l-aid and spinning up a "basic" 6-node cluster for a development prototype. Maybe I'm wrong, of course.

bg24 4y ago

Your scale (50,000 VMs) is way too high for small and medium sized business :-) Is my observation correct that unicorns start to see that scale?