3 ms·

I think the main issue this author ran into was caused by setting the CPU requests so far below the limits. I get that he was trying to communicate the spiky na

I think the main issue this author ran into was caused by setting the CPU requests so far below the limits. I get that he was trying to communicate the spiky nature of the workload to the control plane, but I think it would have been better to reserve CPU to cover the spikes by setting the requests higher, especially on a 1 core node. It's important to grok the underlying systems here, imo. CPU requests map to the cpushares property of the container's cpu,cpuacct cgroup. A key thing about cpushares is that it guarantees a minimum number of 1/1024 shares of a core, but doesn't prevent a process from consuming more if they're available. The CPU limit uses the CPU bandwidth control scheduler, which specifies a number of slices per second (default 100k) and a share of those slices which the process cannot, afaik, exceed. So by setting the request to 20m and the limit to 200m the author left a lot of room for pods that look like they fit fine under normal operating conditions to spike up and consume all the CPU resources on the machine. K8s is supposed to reserve resources for the kubelet and other components on a per node basis but I'm not surprised it's possible to place these components under pressure using settings like these on a 1 core node.

bt848 7y ago
Another thing to keep in mind is that CPU rate is a compressible resource, unlike memory or disk space. It’s fine to briefly oversubscribe CPU. No terrible thing will happen.
brown9-2 7y ago
If I am understanding the article right, this would have led to fewer pods per node and a higher overall cost, wouldn’t it have? The author claims that the extra headroom allowed per pod for spiky CPU usage helped absorb those spikes.
- markbnj 7y ago
  > If I am understanding the article right, this would have led to fewer pods per node and a higher overall cost, wouldn’t it have? To be clear I don't know exactly how the k8s scheduler weighs CPU requests vs. limits in fitting pods to nodes. It's something I've wanted to dig further into. I do know basically how the two underlying control systems function. The cpushares system (K8S CPU requests) cannot prevent a process from taking more shares. The CPU bandwidth control system (K8S CPU limits) will throttle a process at the upper limit, but processes will not be evicted by the kubelet for hitting this limit. So if you have a pod with requests set to 20m and limits set to 200m, those pods are able to take 200m. If the scheduler is using limits to fit pods then maybe you can get 3-4 of these on a 1 core node and leave enough CPU for the other system components. If it's using some weighted combination of limits and requests then it might place more than 3-4 pods on that node, each of which is then permitted to take up to 200m. Just a theory, and I am sure there are some folks here who know exactly how it works. Maybe we'll hear from someone.
  - brown9-2 7y ago
    The scheduler uses requests alone: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-requests-are-scheduled https://kubernetes.io/docs/concepts/configuration/manage-com...

bt848 7y ago

Another thing to keep in mind is that CPU rate is a compressible resource, unlike memory or disk space. It’s fine to briefly oversubscribe CPU. No terrible thing will happen.

brown9-2 7y ago

If I am understanding the article right, this would have led to fewer pods per node and a higher overall cost, wouldn’t it have? The author claims that the extra headroom allowed per pod for spiky CPU usage helped absorb those spikes.

markbnj 7y ago
> If I am understanding the article right, this would have led to fewer pods per node and a higher overall cost, wouldn’t it have? To be clear I don't know exactly how the k8s scheduler weighs CPU requests vs. limits in fitting pods to nodes. It's something I've wanted to dig further into. I do know basically how the two underlying control systems function. The cpushares system (K8S CPU requests) cannot prevent a process from taking more shares. The CPU bandwidth control system (K8S CPU limits) will throttle a process at the upper limit, but processes will not be evicted by the kubelet for hitting this limit. So if you have a pod with requests set to 20m and limits set to 200m, those pods are able to take 200m. If the scheduler is using limits to fit pods then maybe you can get 3-4 of these on a 1 core node and leave enough CPU for the other system components. If it's using some weighted combination of limits and requests then it might place more than 3-4 pods on that node, each of which is then permitted to take up to 200m. Just a theory, and I am sure there are some folks here who know exactly how it works. Maybe we'll hear from someone.
- brown9-2 7y ago
  The scheduler uses requests alone: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-requests-are-scheduled https://kubernetes.io/docs/concepts/configuration/manage-com...