4 ms·

This sort of tuning isn't necessary if you use CPU reservations instead of limits, as you should: https://home.robusta.dev/blog/stop-using-cpu-limits https://ho

This sort of tuning isn't necessary if you use CPU reservations instead of limits, as you should: https://home.robusta.dev/blog/stop-using-cpu-limits https://home.robusta.dev/blog/stop-using-cpu-limits CPU reservations are limits, just implicit ones and declared as guarantees. So let the Go runtime use all the CPUs available, and let the Linux scheduler throttle according to your declared reservations if the CPU is contended for.

yipbub 3y ago
Interesting. This is not true for Memory, correct? The OOMKiller might get you. You also cannot achieve a QoS class of Guaranteed without both CPU and Memory limits, so the pod might be evicted at some point.
- dilyevsky 3y ago
  Correct regarding memory - not true for memory because it's non-fungible unlike CPU shares > You also cannot achieve a QoS class of Guaranteed without both CPU and Memory limits, so the pod might be evicted at some point. Evicted due to node pressure - yes (but if all other pods also don't have limits it doesn't matter). For preemption QoS is not factored in the decision [0] [0] - https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#interactions-of-pod-priority-and-qos https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...
- iTokio 3y ago
  > Memory is different because it is non-compressible - once you give memory you can't take it away without killing the process
  - afr0ck 3y ago
    Swap (Disk, RDMA, Compression)? Page migration (NUMA, CXL)?
    - karolist 3y ago
      All of this is below K8s scheduler level, the K8s scheduler doesn't know how the underlying kernel handles memory, all it cares about whether it thinks there's enough free memory to give to a pod or not because it just keeps track of other pod requests and limits, the fact that giving this memory, which it thinks is unavailable, would actually not result in any swapping for the pod is unknown.
kubiton 3y ago
I run a few things on 128 core setups and I set CPU limits to much higher than request but still set them to make sure nothing runs ammok. I would be curious to see this discussed but your article only states that people think you need limit to ensure CPU for all pods.
ithkuil 3y ago
I don't set limits because I'm afraid of how a pod is going to affect other pods. I set limits because I don't want to get used to being able to tap on the excess CPU available because that's not guaranteed to be available. As the node fills up with more and more other pods, it's possible that a pod that was running just fine a moment ago is crawling to a halt. Limits allow me to simulate the same behavior and plan for it by doing the right capacity planning. They are not the only way to approach it! But they are the simplest way to so it.
- eloisant 3y ago
  That's why you use monitor and alerting, so you notice degraded performances before the pods is crawling to a halt. You need to do it anyway because a service might progressively need more resources as it's getting more traffic, even if you're not adding any other pod.
  - ithkuil 3y ago
    Sure you need monitoring and alerting and sure there are other reasons why you need to update your requests. But having _neighbours_ affecting the behaviour of your workload is precisely what creates the kind of fatigue that then results in people claiming that it's hard to run k8s workloads. K8s is highly dynamical, pods can get scheduled on a node by chance sometimes and on some clusters; pagers will ring, incidents will be created for conditions that may solve themselves because of another deployment (possibly of another team) happening. Overcommit/bursting is an advanced cost saving feature. Let me say it again: splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature! The problem is that the request/limits feature is presented in the configuration spec and in the documentation in a deceptively simple way and we're tricked to think it's a basic feature. Not all companies have ops teams that are well equipped to do more sophisticated things. My advice for those teams who cannot setup full automation around capacity management is to just not use this advanced features. An alternative is to just use smaller dedicated nodes and (anti)affinity rules, so you always understand which pods go with which other pods. It's clunky but it's actually easier to reason about what's going to happen. EDIT: typos
_cenw 3y ago
The Kubernetes community has this discussion every other week. This article isn't wrong per-se (and it's mostly content marketing, they have articles that are just plain wrong too, like https://home.robusta.dev/blog/containers-dont-use-chroot https://home.robusta.dev/blog/containers-dont-use-chroot - i caused the "updated" tag on that one), but it's sweeping in its assertions, ignoring many good reason to set limits. Workloads that use up all the bursting with little benefit, or prioritizing burst volume for an HTTP server instead of a cronjob that'll finish in designated time anyway. We even had a case where developers weren't updating their requests while their app grew in requirements, and then had an incident on their hand when spare CPU time suddenly was sparse.
eightnoteight 3y ago
Reservations are not limits, they are minimum guaranteed CPU usage constraints. while theoretically, those are minimum guaranteed resources, if there are other busy containers running on the host, you will still see your tail latencies and average latency increasing abnormally imagine you got 4 core ec2 instance, the latencies you see at 50% cpu utilisation and 90% cpu utilisation are quite different. with reservations similar thing happens i.e even though each container is guaranteed to their reservations, the relative CPU utilisation still increased very high when there are other busy processes on the same host

yipbub 3y ago

Interesting. This is not true for Memory, correct? The OOMKiller might get you. You also cannot achieve a QoS class of Guaranteed without both CPU and Memory limits, so the pod might be evicted at some point.

dilyevsky 3y ago
Correct regarding memory - not true for memory because it's non-fungible unlike CPU shares > You also cannot achieve a QoS class of Guaranteed without both CPU and Memory limits, so the pod might be evicted at some point. Evicted due to node pressure - yes (but if all other pods also don't have limits it doesn't matter). For preemption QoS is not factored in the decision [0] [0] - https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#interactions-of-pod-priority-and-qos https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...
iTokio 3y ago
> Memory is different because it is non-compressible - once you give memory you can't take it away without killing the process
- afr0ck 3y ago
  Swap (Disk, RDMA, Compression)? Page migration (NUMA, CXL)?
  - karolist 3y ago
    All of this is below K8s scheduler level, the K8s scheduler doesn't know how the underlying kernel handles memory, all it cares about whether it thinks there's enough free memory to give to a pod or not because it just keeps track of other pod requests and limits, the fact that giving this memory, which it thinks is unavailable, would actually not result in any swapping for the pod is unknown.

kubiton 3y ago

I run a few things on 128 core setups and I set CPU limits to much higher than request but still set them to make sure nothing runs ammok. I would be curious to see this discussed but your article only states that people think you need limit to ensure CPU for all pods.

ithkuil 3y ago

I don't set limits because I'm afraid of how a pod is going to affect other pods. I set limits because I don't want to get used to being able to tap on the excess CPU available because that's not guaranteed to be available. As the node fills up with more and more other pods, it's possible that a pod that was running just fine a moment ago is crawling to a halt. Limits allow me to simulate the same behavior and plan for it by doing the right capacity planning. They are not the only way to approach it! But they are the simplest way to so it.

eloisant 3y ago
That's why you use monitor and alerting, so you notice degraded performances before the pods is crawling to a halt. You need to do it anyway because a service might progressively need more resources as it's getting more traffic, even if you're not adding any other pod.
- ithkuil 3y ago
  Sure you need monitoring and alerting and sure there are other reasons why you need to update your requests. But having _neighbours_ affecting the behaviour of your workload is precisely what creates the kind of fatigue that then results in people claiming that it's hard to run k8s workloads. K8s is highly dynamical, pods can get scheduled on a node by chance sometimes and on some clusters; pagers will ring, incidents will be created for conditions that may solve themselves because of another deployment (possibly of another team) happening. Overcommit/bursting is an advanced cost saving feature. Let me say it again: splitting up a large machine into smaller parts that can use the unused capacity of other parts in order to reduce waste is an advanced feature! The problem is that the request/limits feature is presented in the configuration spec and in the documentation in a deceptively simple way and we're tricked to think it's a basic feature. Not all companies have ops teams that are well equipped to do more sophisticated things. My advice for those teams who cannot setup full automation around capacity management is to just not use this advanced features. An alternative is to just use smaller dedicated nodes and (anti)affinity rules, so you always understand which pods go with which other pods. It's clunky but it's actually easier to reason about what's going to happen. EDIT: typos

_cenw 3y ago

The Kubernetes community has this discussion every other week. This article isn't wrong per-se (and it's mostly content marketing, they have articles that are just plain wrong too, like https://home.robusta.dev/blog/containers-dont-use-chroot https://home.robusta.dev/blog/containers-dont-use-chroot - i caused the "updated" tag on that one), but it's sweeping in its assertions, ignoring many good reason to set limits. Workloads that use up all the bursting with little benefit, or prioritizing burst volume for an HTTP server instead of a cronjob that'll finish in designated time anyway. We even had a case where developers weren't updating their requests while their app grew in requirements, and then had an incident on their hand when spare CPU time suddenly was sparse.

eightnoteight 3y ago

Reservations are not limits, they are minimum guaranteed CPU usage constraints. while theoretically, those are minimum guaranteed resources, if there are other busy containers running on the host, you will still see your tail latencies and average latency increasing abnormally imagine you got 4 core ec2 instance, the latencies you see at 50% cpu utilisation and 90% cpu utilisation are quite different. with reservations similar thing happens i.e even though each container is guaranteed to their reservations, the relative CPU utilisation still increased very high when there are other busy processes on the same host