3 ms·

I've seen CPU throttling occur when limits aren't exhausted even on 5.4 kernels, so I don't believe the underlying kernel bug is fixed. One option not mentione

I've seen CPU throttling occur when limits aren't exhausted even on 5.4 kernels, so I don't believe the underlying kernel bug is fixed. One option not mentioned in the post is to enable k8s' static CPU scheduler policy. With this option in place workloads in the "guaranteed" quality of service class that are allocated an integer CPU limit will be given exclusive use of their CPUs. I've found this also avoids the CFS bugs and eliminates CPU throttling, without removing CPU limits. One thing to keep in mind is that this bug mostly impacts workloads that spin up more threads then they have allocated CPUs. For golang workloads you can set GOMAXPROCS to be equal to your CPU allocation and eliminate most throttling that way too, without messing with limits or the static scheduler policy

solatic 6y ago
Enabling the static CPU scheduler policy currently requires setting a Kubelet flag, and that puts it out of reach of most people running managed Kubernetes distributions.
- rsanders 6y ago
  It looks possible on EKS now. https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-eks-managed-node-groups-now-support-ec2-launch-templates-custom-amis/ https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-ek...
  - solatic 6y ago
    Because EKS supports custom launch templates? Good luck trying to finagle that into supporting the exact Kubelet flags that you want to enable, while staying abreast of upstream updates so that your cluster doesn't break when AWS tries to keep it up-to-date. Not anywhere close to a simple "extra_kubelet_flags: array[text]" kind of field.