I'm very excited for the better CFS quota enforcement mechanism in 5.14 (https://lwn.net/Articles/844976/ https://lwn.net/Articles/844976/). Kubernetes impleme

I'm very excited for the better CFS quota enforcement mechanism in 5.14 (https://lwn.net/Articles/844976/ https://lwn.net/Articles/844976/). Kubernetes implements CPU and memory limits for containers (https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ https://kubernetes.io/docs/concepts/configuration/manage-res...). The memory limit works the way you'd expect: there's a memory cgroup, and Linux implements the memory-allocation algorithm based on the memory cgroup's limit and not the total system memory. Wildly-overboard allocations get denied; somewhat-overboard allocations are permitted if you have overcommit turned on, but cause an OOM kill if you try to use all of it. This is at least expected behavior for existing non-Kubernetes applications and (more importantly) for people used to non-Kubernetes application development/deployment: it's just like if you were running on a Linux system with the limited memory. CPU limits are much wackier. Because CPUs are a discrete resource and Kubernetes supports fractional CPU allocations, they can't use CPU pinning. Instead, they use the CFS quota controls, which aren't commonly used outside Kubernetes as far as I can tell. The quota is expressed as basically a fraction: for a process with, say, 0.5 vCPUs allocated, it gets to use 50 ms of CPU time every 100 ms of real time. The way it works at the kernel level, as best as I've been able to understand, is that after a process runs on the CPU for a bit, the kernel looks at the quota and sees if it has exceeded the quota. If it has, the process becomes unschedulable until it has "re-earned" sufficient quota. It does not slow down or proactively de-schedule the process, analogous to how it would have run on a slower or fewer-CPU computer. So, if you have multiple processes or threads running in your container, and there are free cores for them to run, you can easily use up multiple seconds of CPU time in those 100 ms. Then you have to wait to re-earn your quota in 50 ms chunks before your process can run. And all the while the process appears to be runnable, not stuck, it just seems like the kernel is never choosing to schedule it. We have seen, in production, processes stalled for minutes because they evidently ran so far past their quota. We have seen entire system deadlocks when such a stalled process is holding some kernel lock for network I/O, in our case, Ceph (if the kernel releases the process into interruptible sleep, then that process can still hold locks; it will release those locks if signaled and rescheduled, but in this case the process isn't being scheduled at all). And we're far from the only people who have seen awful behavior from CFS quotas. Tim Hockin, Kubernetes co-creator and tech lead, advises people to just never use the CPU limit feature! https://twitter.com/thockin/status/1134193838841401345 https://twitter.com/thockin/status/1134193838841401345 We have done as he advises and just turned it off cluster-wide with --cpu-cfs-quota=false - it helps the Kubernetes scheduler if you set your CPU/memory limits equal to your requests, so this gets us the right Kubernetes behavior without incurring the wacky kernel behavior. If you Google that flag you can find many others who have done the same in their public configurations. There's a Kubernetes feature to use CPU pinning instead, which mentions improved performance as an advantage https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/ https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-... , but it only works if you're scheduling whole numbers of CPUs always. A whole lot of applications on our Kubernetes clusters - lightly-accessed web servers, monitoring daemons, etc. - are best described as wanting 0.1 or 0.5 CPUs, in the sense that on (say) an 8-core machine, you could run many dozens of instances of these applications just fine. But that's not how the existing CFS quota mechanism interprets 0.1 or 0.5 CPUs. It seems like the "burst" mechanism here is likely to handle processes that, in a random 100 ms period, actually do run for more than 10 ms, but on an ongoing basis don't average out to using more than one tenth of a CPU. We might be able to turn --cpu-cfs-quota back on as a result.