3 ms·

For the love of god - care about other pods on the node, especially in a multi-tenant setup. Sorry for the cheeky response. CPU Limits have a place, you don't

by websap 2mo ago

For the love of god - care about other pods on the node, especially in a multi-tenant setup. Sorry for the cheeky response. CPU Limits have a place, you don't want a bad change for 1 deployment object affect all neighbors by taking all the CPU. You need to be able to constrain the blast radius. This doc gives me strong AI vibes. Setting CPU limits isn't free. You still need to care about how the programming language that you use discovers those limits, and correctly handles them. For e.g. if you spin up a 100 Java threads, but only have 1 cpu as the limit, that's bad design.

inigyou 2mo ago
That's what CPU requests are for.
- websap 2mo ago
  CPU requests are cgroup weights.
  - ksbd-pls-finish 2mo ago
    But they also affect scheduling, right? If you set them too high, you will waste resources.
deleted 2mo ago
[deleted]
crymer11 2mo ago
I don’t think you understand how CPU limits and the Linux CPU scheduler work. CPU limits don’t protect you from something taking all the CPU; that’s what CPU requests do. Limits throttle your pods even if the CPU is idle/free to do work.
- websap 2mo ago
  Hahaha! Thanks for the laugh.
- da_chicken 2mo ago
  That's literally not what the documentation says. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits https://kubernetes.io/docs/concepts/configuration/manage-res...
  - acuteaura 2mo ago
    Where do you think the contradiction is? A CPU limit of 1 does not prevent an application from using 8 cores worth of compute at once, it just limits the compute to 1 core per 100ms slice on average, and usually that means getting unscheduled for 7/8 of the slice, if the app is using all available resources and nothing else contests them (weighted by requests).
    - da_chicken 2mo ago
      According to the documentation, CPU requests don't cap usage. They're not used for that. They're used to gauge how much CPU you say you will use. They're used for allocation and pod assignment. They never throttle you. They never cap you. According to the documentation, CPU limits are the only mechanism to prevent exceeding your resource allocation. You directly said the opposite of that. If that's not what they actually do, then FIX. THE. FUCKING. DOCUMENTATION. Either the documentation is wrong or incomplete, or you are wrong or incomplete. That's the contradiction. And, to be clear, I think you're both wrong and incomplete. It's partially a problem that how it's being documented is either really misleading or fundamentally incomplete. That may be because that's just not how Google imagined Kubernates was going to be used. If you need to understand the CPU scheduler to be able to use this option in the first place, then the documentation should explain that directly or by referring to more information elsewhere.
      - acuteaura 2mo ago
        I don't see the contradiction to what I said or what the prior commenter said. CPU requests still protect you from something using all the resources, because the weights correspond to your number vs all the other pods on that node, and the scheduler will not schedule pods so they exceed the total available resources. So more accurately, requests prevent exceeding limits under contestion, and limits always do.
- ozyschmozy 2mo ago
  From the docs another comment linked: > The CPU request typically defines a weighting. If several different containers (cgroups) want to run on a contended system, workloads with larger CPU requests are allocated more CPU time than workloads with small requests. So I guess limits _would_ protect other pods to a degree. Though I agree that it doesn't seem worth the tradeoff of your pod getting constantly interrupted while the rest of the box is sitting idle
mystifyingpoi 2mo ago
Exactly on point. Shit happens, performance bugs appear, someone messes up Kafka config and it starts consuming from the beginning of the world, etc. Limiting CPU is a must. I could see maybe if someone has a super good monitoring + oncall response team, then letting things go loose for a bit is a lesser evil than working out limits, but still.

by websap 2mo ago

inigyou 2mo ago

That's what CPU requests are for.

websap 2mo ago
CPU requests are cgroup weights.
- ksbd-pls-finish 2mo ago
  But they also affect scheduling, right? If you set them too high, you will waste resources.

deleted 2mo ago

[deleted]

crymer11 2mo ago

I don’t think you understand how CPU limits and the Linux CPU scheduler work. CPU limits don’t protect you from something taking all the CPU; that’s what CPU requests do. Limits throttle your pods even if the CPU is idle/free to do work.

websap 2mo ago
Hahaha! Thanks for the laugh.
da_chicken 2mo ago
That's literally not what the documentation says. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits https://kubernetes.io/docs/concepts/configuration/manage-res...
- acuteaura 2mo ago
  Where do you think the contradiction is? A CPU limit of 1 does not prevent an application from using 8 cores worth of compute at once, it just limits the compute to 1 core per 100ms slice on average, and usually that means getting unscheduled for 7/8 of the slice, if the app is using all available resources and nothing else contests them (weighted by requests).
  - da_chicken 2mo ago
    According to the documentation, CPU requests don't cap usage. They're not used for that. They're used to gauge how much CPU you say you will use. They're used for allocation and pod assignment. They never throttle you. They never cap you. According to the documentation, CPU limits are the only mechanism to prevent exceeding your resource allocation. You directly said the opposite of that. If that's not what they actually do, then FIX. THE. FUCKING. DOCUMENTATION. Either the documentation is wrong or incomplete, or you are wrong or incomplete. That's the contradiction. And, to be clear, I think you're both wrong and incomplete. It's partially a problem that how it's being documented is either really misleading or fundamentally incomplete. That may be because that's just not how Google imagined Kubernates was going to be used. If you need to understand the CPU scheduler to be able to use this option in the first place, then the documentation should explain that directly or by referring to more information elsewhere.
    - acuteaura 2mo ago
      I don't see the contradiction to what I said or what the prior commenter said. CPU requests still protect you from something using all the resources, because the weights correspond to your number vs all the other pods on that node, and the scheduler will not schedule pods so they exceed the total available resources. So more accurately, requests prevent exceeding limits under contestion, and limits always do.
ozyschmozy 2mo ago
From the docs another comment linked: > The CPU request typically defines a weighting. If several different containers (cgroups) want to run on a contended system, workloads with larger CPU requests are allocated more CPU time than workloads with small requests. So I guess limits _would_ protect other pods to a degree. Though I agree that it doesn't seem worth the tradeoff of your pod getting constantly interrupted while the rest of the box is sitting idle

mystifyingpoi 2mo ago

Exactly on point. Shit happens, performance bugs appear, someone messes up Kafka config and it starts consuming from the beginning of the world, etc. Limiting CPU is a must. I could see maybe if someone has a super good monitoring + oncall response team, then letting things go loose for a bit is a lesser evil than working out limits, but still.