4 ms·

Just a funny thought exercise I did recently - I looked at our sizeable kubernetes cluster of 30ish VMs, and added up the core counts. It added up to 98. You ca

Just a funny thought exercise I did recently - I looked at our sizeable kubernetes cluster of 30ish VMs, and added up the core counts. It added up to 98. You can easily get a box that has this many (or more) cores. I wouldn't be surprised if our cloud provider analyzed the topology of the connections, and decided to put the whole thing on the same server.

jordanthoms 5y ago
I guess it's all relative, our largest cluster scales up to around 3000 VCPU and there's plenty of much larger ones out there. You might be able to get better utilisation of your VMs by running a smaller number with more cores on each - I'd try 16vcpu nodes.
- randomswede 5y ago
  At my previous workplace, one of the clusters I herded was 60-odd (physical) machines, 72 or 96 cores (honestly can't recall at this point) and 384 GB RAM (IIRC) per physical server. But, that place was a wee bit strange, in that we ran all our kubernetes in-house and started on bare metal. And, yes, as a general recommendation, I would tend towards bigger, rather than smaller, nodes. There's a trade-off to be had, but you want each node to be at LEASt as large as the largest pod you intend to deploy, and you probably want at least some spare, on top of that.
darkwater 5y ago
So if that host goes down you are screwed? Would not be a wise decision from your cloud provider. Although thinking about it twice, usually cloud providers response in such a case would be a shrug and that's all...
- KptMarchewa 5y ago
  I very doubt that would happen. Even if, at the very least, control plane would be distributed, so host going down would just cause another underlying VM to spawn and everything being recreated.
rhplus 5y ago
This would all depend on your risk tolerance and fail-over plan. On Azure, for example, K8S node pools run on VM Scale Sets. By enabling Availability Zone distribution for those VMs, you could at least keep part of your node pool up if one zone in the region went down: https://docs.microsoft.com/en-us/azure/aks/availability-zones https://docs.microsoft.com/en-us/azure/aks/availability-zone...
deleted 5y ago
[deleted]
sgtfrankieboy 5y ago
Probably need less cores for the same performance. Its never entirely clear what a "vCPU" is.
- KptMarchewa 5y ago
  It's one hyperthread core on x86 and full core on ARM.
nijave 5y ago
Sure, but then you can't, say, restart the server with no app downtime (for patches, maintenance, etc). You'd also probably have to setup cgroups yourself if you don't want to risk a rogue process taking up all the resources and starving other processes. For AWS and GCP, there's zone-specific VM scaling groups that ensure you're in multiple AZs so you'd know for sure you're not on a single physical server.
- glintik 5y ago
  > you can't, say, restart the server with no app downtime You can, on properly configured server. Cloud named cloud due to it allows to do such things - VM migration from one host to another one flawlessly.
- bsagdiyev 5y ago
  What is that nonsense? I literally restart apps in a rolling fashion on VMs all the time, no Kubernetes needed and no downtime in sight. This includes a full reboot of the VM itself.
  - anamexis 5y ago
    They are talking about restarting the single server that all of your VMs are running on.
    - bsagdiyev 5y ago
      Which can also be done with zero downtime to the VM.
      - panzagl 5y ago
        It's just like flying- throw yourself at the ground and miss...
        bsagdiyev 5y ago
        I responded above, either I missed the word "single" or it's been edited (but doesn't show it as being edited?) -- live migration between hypervisors is a thing.
hpjules 5y ago
In AWS EC2 you can avoid that not just by spreading across different zones but also using placement groups so nodes are placed in different racks.
antishatter 5y ago
I would be astonished if what you say is true
numbsafari 5y ago
Kubernetes has the ability to allow you to prevent this. At the very least, to tell your operator to not do this. You can use inter-pod affinity and anti-affinity[1] to basically say "don't run the instances of this scaling group on the same physical nodes". Then, when the physical VMs are upgraded for whatever reason, the cluster knows not to bin all the Pods hosting your blog onto the same physical node. Combined with how, e.g., Google handles node upgrades, means that I can keep services up and running pretty seamlessly without a lot of work. In fact, I haven't had downtime due to an upgrade of the underlying nodes or VMs in over 6 years, despite Google regularly updating the underlying nodes to newer host OS and kubelet versions. You can also use node selectors[2] and related configurations to run specific pods on specific nodes or types of nodes (e.g., only run this pod on a node with a fast disk). We do this with all of our Jobs. They are assigned to nodes in an autoscaling nodepool that scales down to zero. So, those nodes are only ever in use when we have jobs running, and jobs coming and going doesn't create any thrashing amongst our long running services. Of course, your cluster operator could ignore all that somehow, but then when you have unexpected downtime during a machine or VM host upgrade, you'll be pretty peeved. You do bring up a good point regarding node core counts. If you are running "30ish VMs ~= 98 cores", then you are probably running 2 and/or 4 core nodes. You may actually want to run "fatter" nodes with higher core counts. Keep in mind that each node is going to have dedicated resources for the kubelet, and any daemonset pods, which can add up pretty quickly sometimes. You may actually be wasting quite a bit of resources to overly redundant overhead. You may also suffer from slower scale-ups because you have to add a lot more individual nodes when things start to ramp up. There's a balance to find, not just with perf and overhead but with resiliency, but it's something I see a lot of folks overlook because they think they want to have super fine-grained node scaling. [1] https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity https://kubernetes.io/docs/concepts/scheduling-eviction/assi... [2] https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector https://kubernetes.io/docs/concepts/scheduling-eviction/assi...
honkycat 5y ago
And managing dozens of apps on a giant VM using raw chroots and cgroups and your own scripting is desirable to Kubernetes? I don't see what you win here. I don't want to have to write my own rolling updates, canary deployments, networking/traffic/load balancing scripts, configuration/secret management, dependency management, log aggregation, user management, etc. I just spin up a Kubernetes cluster, create service accounts for my users, and let them use the k8s docs and declarative primitives to ship their code. If I'm lucky they are already familiar with the system from their last job, so they can bring expertise with them.
deleted 5y ago
[deleted]
talideon 5y ago
If your cloud provider is scheduling everything on the same server, they're doing something very, _very_ wrong.

jordanthoms 5y ago

I guess it's all relative, our largest cluster scales up to around 3000 VCPU and there's plenty of much larger ones out there. You might be able to get better utilisation of your VMs by running a smaller number with more cores on each - I'd try 16vcpu nodes.

randomswede 5y ago
At my previous workplace, one of the clusters I herded was 60-odd (physical) machines, 72 or 96 cores (honestly can't recall at this point) and 384 GB RAM (IIRC) per physical server. But, that place was a wee bit strange, in that we ran all our kubernetes in-house and started on bare metal. And, yes, as a general recommendation, I would tend towards bigger, rather than smaller, nodes. There's a trade-off to be had, but you want each node to be at LEASt as large as the largest pod you intend to deploy, and you probably want at least some spare, on top of that.

darkwater 5y ago

So if that host goes down you are screwed? Would not be a wise decision from your cloud provider. Although thinking about it twice, usually cloud providers response in such a case would be a shrug and that's all...

KptMarchewa 5y ago
I very doubt that would happen. Even if, at the very least, control plane would be distributed, so host going down would just cause another underlying VM to spawn and everything being recreated.

rhplus 5y ago

This would all depend on your risk tolerance and fail-over plan. On Azure, for example, K8S node pools run on VM Scale Sets. By enabling Availability Zone distribution for those VMs, you could at least keep part of your node pool up if one zone in the region went down: https://docs.microsoft.com/en-us/azure/aks/availability-zones https://docs.microsoft.com/en-us/azure/aks/availability-zone...

deleted 5y ago

[deleted]

sgtfrankieboy 5y ago

Probably need less cores for the same performance. Its never entirely clear what a "vCPU" is.

KptMarchewa 5y ago
It's one hyperthread core on x86 and full core on ARM.

nijave 5y ago

Sure, but then you can't, say, restart the server with no app downtime (for patches, maintenance, etc). You'd also probably have to setup cgroups yourself if you don't want to risk a rogue process taking up all the resources and starving other processes. For AWS and GCP, there's zone-specific VM scaling groups that ensure you're in multiple AZs so you'd know for sure you're not on a single physical server.

glintik 5y ago
> you can't, say, restart the server with no app downtime You can, on properly configured server. Cloud named cloud due to it allows to do such things - VM migration from one host to another one flawlessly.
bsagdiyev 5y ago
What is that nonsense? I literally restart apps in a rolling fashion on VMs all the time, no Kubernetes needed and no downtime in sight. This includes a full reboot of the VM itself.
- anamexis 5y ago
  They are talking about restarting the single server that all of your VMs are running on.
  - bsagdiyev 5y ago
    Which can also be done with zero downtime to the VM.
    - panzagl 5y ago
      It's just like flying- throw yourself at the ground and miss...
      - bsagdiyev 5y ago
        I responded above, either I missed the word "single" or it's been edited (but doesn't show it as being edited?) -- live migration between hypervisors is a thing.

hpjules 5y ago

In AWS EC2 you can avoid that not just by spreading across different zones but also using placement groups so nodes are placed in different racks.

antishatter 5y ago

I would be astonished if what you say is true

numbsafari 5y ago

Kubernetes has the ability to allow you to prevent this. At the very least, to tell your operator to not do this. You can use inter-pod affinity and anti-affinity[1] to basically say "don't run the instances of this scaling group on the same physical nodes". Then, when the physical VMs are upgraded for whatever reason, the cluster knows not to bin all the Pods hosting your blog onto the same physical node. Combined with how, e.g., Google handles node upgrades, means that I can keep services up and running pretty seamlessly without a lot of work. In fact, I haven't had downtime due to an upgrade of the underlying nodes or VMs in over 6 years, despite Google regularly updating the underlying nodes to newer host OS and kubelet versions. You can also use node selectors[2] and related configurations to run specific pods on specific nodes or types of nodes (e.g., only run this pod on a node with a fast disk). We do this with all of our Jobs. They are assigned to nodes in an autoscaling nodepool that scales down to zero. So, those nodes are only ever in use when we have jobs running, and jobs coming and going doesn't create any thrashing amongst our long running services. Of course, your cluster operator could ignore all that somehow, but then when you have unexpected downtime during a machine or VM host upgrade, you'll be pretty peeved. You do bring up a good point regarding node core counts. If you are running "30ish VMs ~= 98 cores", then you are probably running 2 and/or 4 core nodes. You may actually want to run "fatter" nodes with higher core counts. Keep in mind that each node is going to have dedicated resources for the kubelet, and any daemonset pods, which can add up pretty quickly sometimes. You may actually be wasting quite a bit of resources to overly redundant overhead. You may also suffer from slower scale-ups because you have to add a lot more individual nodes when things start to ramp up. There's a balance to find, not just with perf and overhead but with resiliency, but it's something I see a lot of folks overlook because they think they want to have super fine-grained node scaling. [1] https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity https://kubernetes.io/docs/concepts/scheduling-eviction/assi... [2] https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector https://kubernetes.io/docs/concepts/scheduling-eviction/assi...

honkycat 5y ago

And managing dozens of apps on a giant VM using raw chroots and cgroups and your own scripting is desirable to Kubernetes? I don't see what you win here. I don't want to have to write my own rolling updates, canary deployments, networking/traffic/load balancing scripts, configuration/secret management, dependency management, log aggregation, user management, etc. I just spin up a Kubernetes cluster, create service accounts for my users, and let them use the k8s docs and declarative primitives to ship their code. If I'm lucky they are already familiar with the system from their last job, so they can bring expertise with them.

deleted 5y ago

[deleted]

talideon 5y ago

If your cloud provider is scheduling everything on the same server, they're doing something very, _very_ wrong.