5 ms·

Kubernetes doesn't get patches from hyperscale companies because kubernetes is not really hyperscale software. If I recall correctly (probably don't), clusters

by likpok 5y ago

Kubernetes doesn't get patches from hyperscale companies because kubernetes is not really hyperscale software. If I recall correctly (probably don't), clusters can handle tens of thousands of container hosts. That's not manageable at large scale unless you're selling people individual kubernetes instances directly (which, to be fair, google and amazon are). In my experience the thing that kills open source at big companies is the divergence of needs: The company needs massive scale software, but the community wants ease-of-use and features. Most open source models are a big drag on productivity (you can't just refactor all the callsites to do a migration!), and it's hard to change directions when the company's need does (since the other contributors will likely want the original software). The tailscale people have been arguing this as well (with more data and better reasoning), but fundamentally you need different software when you're a scrappy startup or a medium size company or a giant behemoth.

busterarm 5y ago
Having operated all the way up to "hyperscale", I agree with your point completely.
- zozbot234 5y ago
  What's preventing k8s from being useful in these larger scale deployments? GP was not very clear about that, they just said that it's not scaling enough.
  - busterarm 5y ago
    K8s doesn't do anything to help you with managing the fleet of k8s hosts. At 10^5 host machines you have a much bigger problem on your hands than the one that k8s is solving for. If you have and are solving that hyperscale infrastructure problem then whether or not you're running kubernetes doesn't actually make a damn bit of difference to how you allocate your human/financial resources. Or summarized, at the scale where Kubernetes is advertised to make the biggest difference, it's actually irrelevant/an implementation detail.
    - zozbot234 5y ago
      Is that a proper job for k8s? Isn't it supposed to be strictly for orchestrating workloads on top of that fleet?
      - busterarm 5y ago
        No but that's not the point. The scale of the underlying operational problem is so much bigger that whether you're running kubernetes or some other solution on top of it doesn't actually make any material difference (kubernetes or any other option are effectively equivalent). Thus hyperscale companies have no reason to contribute to Kubernetes. It does not affect their bottom line. Hence all of the memes about companies chasing Kubernetes are being distracted from their real goals.
        pas 5y ago
        what do hyperscale companies use to implement these functions?
    - lupire 5y ago
      10^5 machines cost $100M, at which point you are a huge enterprise who can afford to pay people to customize your system.
      - busterarm 5y ago
        That's exactly what I'm saying. And not "can". Have to. Kubernetes or not-Kubernetes isn't the problem that huge enterprises ever have.
      - likpok 5y ago
        10^5 is also on the low end of hyperscale. I'd bet that google/amazon have high single-digit millions of machines. So yes, big companies can afford to pay people to customize their systems, and do so. The infra teams at these places are enormous, even just counting job orchestration. The argument upthread (mine) is that the companies do so, and that kubernetes is the wrong choice for them. As a result, they don't really contribute to it.
  - maherbeg 5y ago
    A lot of k8s is just not optimized for beyond a relatively small scale (10k services etc.). In particular many of the components rely on a primary node to handle all the traffic rather than sharding work across a group of nodes.
    - busterarm 5y ago
      Completely true. Woe unto those who learned this the hard way and have suddenly found themselves in the fulltime position of "tuning etcd for kubernetes".
faizshah 5y ago
Maybe I’m misunderstanding but why would you want to run one giant kubernetes cluster at hyper scale? I would think at hyper scale you would be running micro services with one kubernetes cluster per service per datacenter/region.
- GauntletWizard 5y ago
  At hyper scale, efficiency matters a little. Small gains multiplied by big numbers become big gains. The overhead of running a cluster is not small, but more importantly - the overhead of managing nodes between services is real. No, you would not want to run a cluster per service; Kubernetes is somewhat multitenant. You'd get better efficiency from running one cluster per datacenter, with all those teams in their own namespaces. Kubernetes can stack many containers per physical host, with widely varied workloads cooperatively sharing.
  - busterarm 5y ago
    > Kubernetes can stack many containers per physical host, with widely varied workloads cooperatively sharing. Until you learn the downsides of this approach at hyper scale, namely that containers and a shared kernel mean that all of your workloads are sharing the same kernel parameters, including things like network limits and timeouts and file handle limits. Multitenancy and containers actually ends up working against you and creates new problems and knobs in your individual jobs that you have to configure -- to the point that it's almost worth just having different types of jobs run on different isolated node pools and eliminating your multitenancy issue anyway. Companies that scaled on KVM never had to learn about these limitations and just focused on what their hardware was capable of in aggregate. At hyper scale and with multitenancy, microVMs are always going to be the end state -- and while there's k8s support for this, it's far from the default or even most convenient option.
    - GauntletWizard 5y ago
      Network limits and timeouts aren't different between kubernetes hosts and non-Kubernetes hosts. Network resources are a real resource, and you may need to implement quality of service or custom resources (a new feature [1], and one that is late to the party). File handle limits are something no sane workload ever encounters. They are technically a shared resource, but in a sensible kubernetes configuration, it is impossible to hit because the ulimits on each process are low enough. A very small number of teams may need an exception, with good reason, and will typically be cordoned on to their own node classes that are specially tainted. Yes, fleet Management via taints offers nothing over the fleet Management that you've already got. This is a good thing. Fleet Management tools are a damage to your reliability. They mean that your machines are non-fungible. Kubernetes great innovation is making machines, units of compute, fungible. There are workloads and architectures that will never be suitable for kubernetes. There are HPC Clusters that heavily rely on things like rack-locality that Kubernetes views as damage. Getting rid of them is a net win for humanity. [1] https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#extended-resources https://kubernetes.io/docs/concepts/configuration/manage-res...

by likpok 5y ago

busterarm 5y ago

Having operated all the way up to "hyperscale", I agree with your point completely.

zozbot234 5y ago
What's preventing k8s from being useful in these larger scale deployments? GP was not very clear about that, they just said that it's not scaling enough.
- busterarm 5y ago
  K8s doesn't do anything to help you with managing the fleet of k8s hosts. At 10^5 host machines you have a much bigger problem on your hands than the one that k8s is solving for. If you have and are solving that hyperscale infrastructure problem then whether or not you're running kubernetes doesn't actually make a damn bit of difference to how you allocate your human/financial resources. Or summarized, at the scale where Kubernetes is advertised to make the biggest difference, it's actually irrelevant/an implementation detail.
  - zozbot234 5y ago
    Is that a proper job for k8s? Isn't it supposed to be strictly for orchestrating workloads on top of that fleet?
    - busterarm 5y ago
      No but that's not the point. The scale of the underlying operational problem is so much bigger that whether you're running kubernetes or some other solution on top of it doesn't actually make any material difference (kubernetes or any other option are effectively equivalent). Thus hyperscale companies have no reason to contribute to Kubernetes. It does not affect their bottom line. Hence all of the memes about companies chasing Kubernetes are being distracted from their real goals.
      - pas 5y ago
        what do hyperscale companies use to implement these functions?
  - lupire 5y ago
    10^5 machines cost $100M, at which point you are a huge enterprise who can afford to pay people to customize your system.
    - busterarm 5y ago
      That's exactly what I'm saying. And not "can". Have to. Kubernetes or not-Kubernetes isn't the problem that huge enterprises ever have.
    - likpok 5y ago
      10^5 is also on the low end of hyperscale. I'd bet that google/amazon have high single-digit millions of machines. So yes, big companies can afford to pay people to customize their systems, and do so. The infra teams at these places are enormous, even just counting job orchestration. The argument upthread (mine) is that the companies do so, and that kubernetes is the wrong choice for them. As a result, they don't really contribute to it.
- maherbeg 5y ago
  A lot of k8s is just not optimized for beyond a relatively small scale (10k services etc.). In particular many of the components rely on a primary node to handle all the traffic rather than sharding work across a group of nodes.
  - busterarm 5y ago
    Completely true. Woe unto those who learned this the hard way and have suddenly found themselves in the fulltime position of "tuning etcd for kubernetes".

faizshah 5y ago

Maybe I’m misunderstanding but why would you want to run one giant kubernetes cluster at hyper scale? I would think at hyper scale you would be running micro services with one kubernetes cluster per service per datacenter/region.

GauntletWizard 5y ago
At hyper scale, efficiency matters a little. Small gains multiplied by big numbers become big gains. The overhead of running a cluster is not small, but more importantly - the overhead of managing nodes between services is real. No, you would not want to run a cluster per service; Kubernetes is somewhat multitenant. You'd get better efficiency from running one cluster per datacenter, with all those teams in their own namespaces. Kubernetes can stack many containers per physical host, with widely varied workloads cooperatively sharing.
- busterarm 5y ago
  > Kubernetes can stack many containers per physical host, with widely varied workloads cooperatively sharing. Until you learn the downsides of this approach at hyper scale, namely that containers and a shared kernel mean that all of your workloads are sharing the same kernel parameters, including things like network limits and timeouts and file handle limits. Multitenancy and containers actually ends up working against you and creates new problems and knobs in your individual jobs that you have to configure -- to the point that it's almost worth just having different types of jobs run on different isolated node pools and eliminating your multitenancy issue anyway. Companies that scaled on KVM never had to learn about these limitations and just focused on what their hardware was capable of in aggregate. At hyper scale and with multitenancy, microVMs are always going to be the end state -- and while there's k8s support for this, it's far from the default or even most convenient option.
  - GauntletWizard 5y ago
    Network limits and timeouts aren't different between kubernetes hosts and non-Kubernetes hosts. Network resources are a real resource, and you may need to implement quality of service or custom resources (a new feature [1], and one that is late to the party). File handle limits are something no sane workload ever encounters. They are technically a shared resource, but in a sensible kubernetes configuration, it is impossible to hit because the ulimits on each process are low enough. A very small number of teams may need an exception, with good reason, and will typically be cordoned on to their own node classes that are specially tainted. Yes, fleet Management via taints offers nothing over the fleet Management that you've already got. This is a good thing. Fleet Management tools are a damage to your reliability. They mean that your machines are non-fungible. Kubernetes great innovation is making machines, units of compute, fungible. There are workloads and architectures that will never be suitable for kubernetes. There are HPC Clusters that heavily rely on things like rack-locality that Kubernetes views as damage. Getting rid of them is a net win for humanity. [1] https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#extended-resources https://kubernetes.io/docs/concepts/configuration/manage-res...