8 ms·

For the love of god, stop using CPU limits on Kubernetes (updated)

47 points by ciceryadam 4y ago

ekimekim 4y ago
The argument against this is consistency. Without a limit set, you are only guarenteed up to your request's worth of cpu, but you will often be allowed to have more. This can create a false sense of security, as your application is working fine (even though it occasionally exceeds its request). Until one day, when a neighbor happens to get thirsty, and your application suddenly breaks. Limits front-load the brokenness so that it happens immediately instead of randomly.
- stingraycharles 4y ago
  It’s a trade-off that needs to be considered on a case-by-case basis. This is one of those cases where “one rule fits all” doesn’t work.
- zufallsheld 4y ago
  > Until one day, when a neighbor happens to get thirsty, and your application suddenly breaks. Limits front-load the brokenness so that it happens immediately instead of randomly. Can you expand on this? Why should the application break? It still has its guaranteed cpu requests. If it breaks with the defined requests, shouldn't it always break?
  - kobalsky 4y ago
    Not op, but it would break because your CPU request wasn't high enough from the start and the problem was hidden because you had CPU to spare in the node. Once you don't have CPU to spare, the app breaks. EDIT: the vertical pod autoscaler helps with this as it will adjust requirements on pods to make sure they don't over/under allocate.
    - dilyevsky 4y ago
      Solution - monitor your apps for when usage is consistently over request
  - sascha_sl 4y ago
    I've had this happen in the real world. In this case it was a GraphQL implementation that gradually grew in size, complexity and scope. The team maintaining it never adjusted the initial resource requirements within about 2 years because it has never been a problem - until a different (much larger by total CPU allocated) service started consuming all the bursting it could get and the GraphQL service just stopped processing requests. Before any autoscaling could kick in, it went into CrashLoopBackoff with failing liveness. You can and should catch such cases early with monitoring, but our platform team was extremely tiny, especially for how many developers it served.
skyde 4y ago
I don’t agree on the recommendation for memory « Always set your memory requests equal to your limits » you can layer high priority service and low priority service better if you use some buffer.
- clhodapp 4y ago
  My current model for memory on k8s tends to agree with the article. Would it be possible for you to explain "you can layer high priority service and low priority service better if you use some buffer" further?
  - skyde 4y ago
    I mean think of low priority service as services that are not latency sensitive (background jobs). you want those low priory cgroup to use the extra memory if it’s available (not used by a high priority cgroup) it a high priority cgroup need the memory it steal it from the low priority cgroup up to the minimum guaranteed to this low priority cgroup. this low priority cgroup memory contention will turns into IO pressure from page faults. Then IO limit on the low priority cgroup will cap how much IO it can generate by throttling it. Meanwhile the high priority cgroup use it’s guaranteed cpu, guaranteed IO and guaranteed memory with no hiccup. to learn more check « fbtax2 memory controller configuration » at https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.html https://facebookmicrosites.github.io/cgroup2/docs/memory-con...
    - skyde 4y ago
      In Linux kernel, Different memory cgroups have distinct lruvecs, for memcg reclaim. This mean global reclaim algorithm can look at all cgroup "memory.low", and reclaim first from cgroups that are using more than their "memory.low" config. You need to make sure that the sum of all cgroup "memory.low" is still below total memory available. Leaving 20% buffer for when the whole system is under stress is a good rule of thumb. In addition, any memory that's guaranteed but unused can be allocated to other processes, further optimizing memory utilization.
- 4y ago
birdyrooster 4y ago
I don't think this is a remotely compelling argument to never use limits.
- dhsysusbsjsi 4y ago
  The summary should be to use a request which satisfies the long term minimum cpu required at any given time. Then if you’re happy to allow it to burst then don’t set a limit. If you want consistency and constant runtime, set a limit. If you find that your container only works when it uses the “free” burst time, and breaks when it goes back to the request cpu, then your request is too low.
  - skyde 4y ago
    good summary!
988747 4y ago
The reason to never use CPU limits is different than those stated in the article. In short: Linux kernel SUCKS. More specifically, the "Completely Fair Scheduler" (CFS) sucks at enforcing those limits. Setting any limit at all causes CFS to waste like half of CPU cycles on enforcing it, and only the other half is available for any useful work.
- dilyevsky 4y ago
  The double accounting has been fixed a while ago. The real issue is poor defaults (100ms period is too long) and generally folks aren’t aware that limit actually sets cfs quota
- oogali 4y ago
  Can’t you use a different scheduler on your nodes?
rahen 4y ago
We use CPU limits at work for the simple reason we can't autoscale deployments without having them set. An HPA will deploy a new pod each time the CPU limit has been reached for more than 30 seconds. The whole point is to scale out, not up.
- sgarland 4y ago
  CPU utilization isn't a great metric to scale on. If the app is running at 90% CPU and all requests are being processed well within your SLIs, it's fine. KEDA [0] lets you scale on any number of things - basically if you scrape it with Prometheus, you can scale on it. [0] https://keda.sh https://keda.sh
- dilyevsky 4y ago
  That is incorrect - hpa works off cpu request not limit (and usage).
  - rahen 4y ago
    When you do a "kubectl get hpa", the 0% and 100% rules are set according to the CPU request and limit. The current usage will be set to "<unknown>" if either is missing, and the deployment won't autoscale.
    - dilyevsky 4y ago
      Again this is not correct - limits are not factored in at all. Even if you do set them it's important to understand this in case overcommit is setup. Here's the relevant offical documentation: > For per-pod resource metrics (like CPU), the controller fetches the metrics from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler. Then, if a target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod. > ... > Please note that if some of the Pod's containers do not have the relevant resource request set, CPU utilization for the Pod will not be defined and the autoscaler will not take any action for that metric. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ https://kubernetes.io/docs/tasks/run-application/horizontal-... Edit: here's the code: https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebcea5d75617da6accaf67f815e0cf/pkg/controller/podautoscaler/replica_calculator.go#L92 https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebce...
iknownothow 4y ago
This advice comes from tunnel vision and makes perfect sense if you know that you have exactly two pods running at any given time. But if you have exactly two pods, then why bother use k8s? IIRC one of the major selling points of K8s was on-demand scaling or auto scaling horizontally. Which means the number of pods you have in the cluster is dynamic. In the context of pods dynamically spinning up and spinning down, it's bad when a pod replica can't be allocated in the cluster "predictably" but there is nothing worse than when a new pod (new deployment) fails because "Marcus the pod" drank all the water and now I have to call DevOps and wait god knows how long before they spin up a new node to guarantee a spot for the new pod. Bin-packing is a already an np-hard problem. If you remove limits from CPU then you're adding probabilities into the mix. So, for the love of god, always use limits unless you have a very specific use case.
- sgarland 4y ago
  > I have to call DevOps and wait god knows how long before they spin up a new node to guarantee a spot for the new pod. Cluster autoscaler. No need to call anyone.
lazyant 4y ago
Looks to me the author hasn't run different workloads in different production clusters of any complexity. Advise is fine for a small predictable cluster but too simplistic for any real complex cluster.
- sgarland 4y ago
  We run hundreds of nodes (autoscaling cluster) across multiple clusters, with wildly varying workloads, and almost none of the apps have CPU limits set. No issues.
  - dilyevsky 4y ago
    Same here. We have eliminated all limits entirely. Same scale (maybe slightly larger). Diverse workloads. In principle we can bring cfs back with like 1ms period but we’d probably look into cpuset pinning first

47 points by ciceryadam 4y ago

The argument against this is consistency. Without a limit set, you are only guarenteed up to your request's worth of cpu, but you will often be allowed to have more. This can create a false sense of security, as your application is working fine (even though it occasionally exceeds its request). Until one day, when a neighbor happens to get thirsty, and your application suddenly breaks. Limits front-load the brokenness so that it happens immediately instead of randomly.

stingraycharles 4y ago
It’s a trade-off that needs to be considered on a case-by-case basis. This is one of those cases where “one rule fits all” doesn’t work.
zufallsheld 4y ago
> Until one day, when a neighbor happens to get thirsty, and your application suddenly breaks. Limits front-load the brokenness so that it happens immediately instead of randomly. Can you expand on this? Why should the application break? It still has its guaranteed cpu requests. If it breaks with the defined requests, shouldn't it always break?
- kobalsky 4y ago
  Not op, but it would break because your CPU request wasn't high enough from the start and the problem was hidden because you had CPU to spare in the node. Once you don't have CPU to spare, the app breaks. EDIT: the vertical pod autoscaler helps with this as it will adjust requirements on pods to make sure they don't over/under allocate.
  - dilyevsky 4y ago
    Solution - monitor your apps for when usage is consistently over request
- sascha_sl 4y ago
  I've had this happen in the real world. In this case it was a GraphQL implementation that gradually grew in size, complexity and scope. The team maintaining it never adjusted the initial resource requirements within about 2 years because it has never been a problem - until a different (much larger by total CPU allocated) service started consuming all the bursting it could get and the GraphQL service just stopped processing requests. Before any autoscaling could kick in, it went into CrashLoopBackoff with failing liveness. You can and should catch such cases early with monitoring, but our platform team was extremely tiny, especially for how many developers it served.

skyde 4y ago

I don’t agree on the recommendation for memory « Always set your memory requests equal to your limits » you can layer high priority service and low priority service better if you use some buffer.

clhodapp 4y ago
My current model for memory on k8s tends to agree with the article. Would it be possible for you to explain "you can layer high priority service and low priority service better if you use some buffer" further?
- skyde 4y ago
  I mean think of low priority service as services that are not latency sensitive (background jobs). you want those low priory cgroup to use the extra memory if it’s available (not used by a high priority cgroup) it a high priority cgroup need the memory it steal it from the low priority cgroup up to the minimum guaranteed to this low priority cgroup. this low priority cgroup memory contention will turns into IO pressure from page faults. Then IO limit on the low priority cgroup will cap how much IO it can generate by throttling it. Meanwhile the high priority cgroup use it’s guaranteed cpu, guaranteed IO and guaranteed memory with no hiccup. to learn more check « fbtax2 memory controller configuration » at https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.html https://facebookmicrosites.github.io/cgroup2/docs/memory-con...
  - skyde 4y ago
    In Linux kernel, Different memory cgroups have distinct lruvecs, for memcg reclaim. This mean global reclaim algorithm can look at all cgroup "memory.low", and reclaim first from cgroups that are using more than their "memory.low" config. You need to make sure that the sum of all cgroup "memory.low" is still below total memory available. Leaving 20% buffer for when the whole system is under stress is a good rule of thumb. In addition, any memory that's guaranteed but unused can be allocated to other processes, further optimizing memory utilization.
4y ago

birdyrooster 4y ago

I don't think this is a remotely compelling argument to never use limits.

dhsysusbsjsi 4y ago
The summary should be to use a request which satisfies the long term minimum cpu required at any given time. Then if you’re happy to allow it to burst then don’t set a limit. If you want consistency and constant runtime, set a limit. If you find that your container only works when it uses the “free” burst time, and breaks when it goes back to the request cpu, then your request is too low.
- skyde 4y ago
  good summary!

988747 4y ago

The reason to never use CPU limits is different than those stated in the article. In short: Linux kernel SUCKS. More specifically, the "Completely Fair Scheduler" (CFS) sucks at enforcing those limits. Setting any limit at all causes CFS to waste like half of CPU cycles on enforcing it, and only the other half is available for any useful work.

dilyevsky 4y ago
The double accounting has been fixed a while ago. The real issue is poor defaults (100ms period is too long) and generally folks aren’t aware that limit actually sets cfs quota
oogali 4y ago
Can’t you use a different scheduler on your nodes?

rahen 4y ago

We use CPU limits at work for the simple reason we can't autoscale deployments without having them set. An HPA will deploy a new pod each time the CPU limit has been reached for more than 30 seconds. The whole point is to scale out, not up.

sgarland 4y ago
CPU utilization isn't a great metric to scale on. If the app is running at 90% CPU and all requests are being processed well within your SLIs, it's fine. KEDA [0] lets you scale on any number of things - basically if you scrape it with Prometheus, you can scale on it. [0] https://keda.sh https://keda.sh
dilyevsky 4y ago
That is incorrect - hpa works off cpu request not limit (and usage).
- rahen 4y ago
  When you do a "kubectl get hpa", the 0% and 100% rules are set according to the CPU request and limit. The current usage will be set to "<unknown>" if either is missing, and the deployment won't autoscale.
  - dilyevsky 4y ago
    Again this is not correct - limits are not factored in at all. Even if you do set them it's important to understand this in case overcommit is setup. Here's the relevant offical documentation: > For per-pod resource metrics (like CPU), the controller fetches the metrics from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler. Then, if a target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod. > ... > Please note that if some of the Pod's containers do not have the relevant resource request set, CPU utilization for the Pod will not be defined and the autoscaler will not take any action for that metric. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ https://kubernetes.io/docs/tasks/run-application/horizontal-... Edit: here's the code: https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebcea5d75617da6accaf67f815e0cf/pkg/controller/podautoscaler/replica_calculator.go#L92 https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebce...

iknownothow 4y ago

This advice comes from tunnel vision and makes perfect sense if you know that you have exactly two pods running at any given time. But if you have exactly two pods, then why bother use k8s? IIRC one of the major selling points of K8s was on-demand scaling or auto scaling horizontally. Which means the number of pods you have in the cluster is dynamic. In the context of pods dynamically spinning up and spinning down, it's bad when a pod replica can't be allocated in the cluster "predictably" but there is nothing worse than when a new pod (new deployment) fails because "Marcus the pod" drank all the water and now I have to call DevOps and wait god knows how long before they spin up a new node to guarantee a spot for the new pod. Bin-packing is a already an np-hard problem. If you remove limits from CPU then you're adding probabilities into the mix. So, for the love of god, always use limits unless you have a very specific use case.

sgarland 4y ago
> I have to call DevOps and wait god knows how long before they spin up a new node to guarantee a spot for the new pod. Cluster autoscaler. No need to call anyone.

lazyant 4y ago

Looks to me the author hasn't run different workloads in different production clusters of any complexity. Advise is fine for a small predictable cluster but too simplistic for any real complex cluster.

sgarland 4y ago
We run hundreds of nodes (autoscaling cluster) across multiple clusters, with wildly varying workloads, and almost none of the apps have CPU limits set. No issues.
- dilyevsky 4y ago
  Same here. We have eliminated all limits entirely. Same scale (maybe slightly larger). Diverse workloads. In principle we can bring cfs back with like 1ms period but we’d probably look into cpuset pinning first