4 ms·

It is difficult to determine what exactly the 'mistake' was in this post.

Bombthecat 7y ago
I didn't read the post (don't Medium and most of the time it's useless stuff) but I'm not surprised. Gke / kubernetes has a ton of dark stuff going on. In, there are dozens of parameters and if you don't set them. They are default which can... Lead to strange situations :) Inspect all the stuff and generated configs! That's how I started.
edaemon 7y ago
They were using high-resource nodes and made the mistake of shifting to low-resource nodes for fine-grained control over the total amount of resources provisioned -- i.e. a single 96-unit node vs 96 1-unit nodes. That was a problem because the low-resource nodes were much less efficient at processing the actual load; a portion of the resources for each node were allocated to k8s system processes, but also any idle pod consumed a higher proportion of the node's resources to do nothing. As a result the autoscaling functions provisioned even more resources than they were using with the high-resource nodes. The solution was to use medium-resource nodes that offered somewhat granular control but made efficient use of the available resources.
rumanator 7y ago
My take was that the mistake was using a myriad of low resource nodes to run deployments that required an increasing amount of computational resources to run without problems. This led to launching more nodes to accommodate peaks which then just stayed idling. The Kubernetes cluster was configured with horizontal pod autoscaling and cluster autoscaling, and to avoid problems the cpu limits were set to 0.5cpu. The end result was Kubernetes creating a myriad of nodes running 70% idle to accommodate the result of the cluster's autoscaling policy because a 1vCPU node does not have much headroom to accommodate peaks. For example, if you have 3 or 4 pods with 500m cpu limit running on a single vCPU node and it so happens that two peak at the same time, resource limits will be hit and cluster scaling will kick in to create yet another node just to meet that demand. In practice this mean that for each and every 1vCPU node to accommodate the peak demand of a single pod without kicking in cluster autoscaling, it needs to run at most at 50% idle. This problem is mitigated by replacing 1vCPU nodes with higher vCPU count (the author switched to nodes with 16 vCPUs) because they have enough resources to scale up deployments without having to launch new nodes.

I didn't read the post (don't Medium and most of the time it's useless stuff) but I'm not surprised. Gke / kubernetes has a ton of dark stuff going on. In, there are dozens of parameters and if you don't set them. They are default which can... Lead to strange situations :) Inspect all the stuff and generated configs! That's how I started.

edaemon 7y ago

They were using high-resource nodes and made the mistake of shifting to low-resource nodes for fine-grained control over the total amount of resources provisioned -- i.e. a single 96-unit node vs 96 1-unit nodes. That was a problem because the low-resource nodes were much less efficient at processing the actual load; a portion of the resources for each node were allocated to k8s system processes, but also any idle pod consumed a higher proportion of the node's resources to do nothing. As a result the autoscaling functions provisioned even more resources than they were using with the high-resource nodes. The solution was to use medium-resource nodes that offered somewhat granular control but made efficient use of the available resources.

rumanator 7y ago

My take was that the mistake was using a myriad of low resource nodes to run deployments that required an increasing amount of computational resources to run without problems. This led to launching more nodes to accommodate peaks which then just stayed idling. The Kubernetes cluster was configured with horizontal pod autoscaling and cluster autoscaling, and to avoid problems the cpu limits were set to 0.5cpu. The end result was Kubernetes creating a myriad of nodes running 70% idle to accommodate the result of the cluster's autoscaling policy because a 1vCPU node does not have much headroom to accommodate peaks. For example, if you have 3 or 4 pods with 500m cpu limit running on a single vCPU node and it so happens that two peak at the same time, resource limits will be hit and cluster scaling will kick in to create yet another node just to meet that demand. In practice this mean that for each and every 1vCPU node to accommodate the peak demand of a single pod without kicking in cluster autoscaling, it needs to run at most at 50% idle. This problem is mitigated by replacing 1vCPU nodes with higher vCPU count (the author switched to nodes with 16 vCPUs) because they have enough resources to scale up deployments without having to launch new nodes.