4 ms·

We use CPU limits at work for the simple reason we can't autoscale deployments without having them set. An HPA will deploy a new pod each time the CPU limit has

by rahen 4y ago

We use CPU limits at work for the simple reason we can't autoscale deployments without having them set. An HPA will deploy a new pod each time the CPU limit has been reached for more than 30 seconds. The whole point is to scale out, not up.

sgarland 4y ago
CPU utilization isn't a great metric to scale on. If the app is running at 90% CPU and all requests are being processed well within your SLIs, it's fine. KEDA [0] lets you scale on any number of things - basically if you scrape it with Prometheus, you can scale on it. [0] https://keda.sh https://keda.sh
dilyevsky 4y ago
That is incorrect - hpa works off cpu request not limit (and usage).
- rahen 4y ago
  When you do a "kubectl get hpa", the 0% and 100% rules are set according to the CPU request and limit. The current usage will be set to "<unknown>" if either is missing, and the deployment won't autoscale.
  - dilyevsky 4y ago
    Again this is not correct - limits are not factored in at all. Even if you do set them it's important to understand this in case overcommit is setup. Here's the relevant offical documentation: > For per-pod resource metrics (like CPU), the controller fetches the metrics from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler. Then, if a target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod. > ... > Please note that if some of the Pod's containers do not have the relevant resource request set, CPU utilization for the Pod will not be defined and the autoscaler will not take any action for that metric. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ https://kubernetes.io/docs/tasks/run-application/horizontal-... Edit: here's the code: https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebcea5d75617da6accaf67f815e0cf/pkg/controller/podautoscaler/replica_calculator.go#L92 https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebce...

by rahen 4y ago

sgarland 4y ago

CPU utilization isn't a great metric to scale on. If the app is running at 90% CPU and all requests are being processed well within your SLIs, it's fine. KEDA [0] lets you scale on any number of things - basically if you scrape it with Prometheus, you can scale on it. [0] https://keda.sh https://keda.sh

dilyevsky 4y ago

That is incorrect - hpa works off cpu request not limit (and usage).

rahen 4y ago
When you do a "kubectl get hpa", the 0% and 100% rules are set according to the CPU request and limit. The current usage will be set to "<unknown>" if either is missing, and the deployment won't autoscale.
- dilyevsky 4y ago
  Again this is not correct - limits are not factored in at all. Even if you do set them it's important to understand this in case overcommit is setup. Here's the relevant offical documentation: > For per-pod resource metrics (like CPU), the controller fetches the metrics from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler. Then, if a target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod. > ... > Please note that if some of the Pod's containers do not have the relevant resource request set, CPU utilization for the Pod will not be defined and the autoscaler will not take any action for that metric. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ https://kubernetes.io/docs/tasks/run-application/horizontal-... Edit: here's the code: https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebcea5d75617da6accaf67f815e0cf/pkg/controller/podautoscaler/replica_calculator.go#L92 https://github.com/kubernetes/kubernetes/blob/3ffdfbe286ebce...