3 ms·
Kubernetes has the ability to allow you to prevent this. At the very least, to tell your operator to not do this. You can use inter-pod affinity and anti-affin
by numbsafari 5y ago
Kubernetes has the ability to allow you to prevent this. At the very least, to tell your operator to not do this.
You can use inter-pod affinity and anti-affinity[1] to basically say "don't run the instances of this scaling group on the same physical nodes". Then, when the physical VMs are upgraded for whatever reason, the cluster knows not to bin all the Pods hosting your blog onto the same physical node.
Combined with how, e.g., Google handles node upgrades, means that I can keep services up and running pretty seamlessly without a lot of work. In fact, I haven't had downtime due to an upgrade of the underlying nodes or VMs in over 6 years, despite Google regularly updating the underlying nodes to newer host OS and kubelet versions.
You can also use node selectors[2] and related configurations to run specific pods on specific nodes or types of nodes (e.g., only run this pod on a node with a fast disk).
We do this with all of our Jobs. They are assigned to nodes in an autoscaling nodepool that scales down to zero. So, those nodes are only ever in use when we have jobs running, and jobs coming and going doesn't create any thrashing amongst our long running services.
Of course, your cluster operator could ignore all that somehow, but then when you have unexpected downtime during a machine or VM host upgrade, you'll be pretty peeved.
You do bring up a good point regarding node core counts. If you are running "30ish VMs ~= 98 cores", then you are probably running 2 and/or 4 core nodes. You may actually want to run "fatter" nodes with higher core counts. Keep in mind that each node is going to have dedicated resources for the kubelet, and any daemonset pods, which can add up pretty quickly sometimes. You may actually be wasting quite a bit of resources to overly redundant overhead. You may also suffer from slower scale-ups because you have to add a lot more individual nodes when things start to ramp up. There's a balance to find, not just with perf and overhead but with resiliency, but it's something I see a lot of folks overlook because they think they want to have super fine-grained node scaling.
[1] https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity https://kubernetes.io/docs/concepts/scheduling-eviction/assi...
[2] https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector https://kubernetes.io/docs/concepts/scheduling-eviction/assi...