4 ms·

I don’t agree on the recommendation for memory « Always set your memory requests equal to your limits » you can layer high priority service and low priority s

by skyde 4y ago

I don’t agree on the recommendation for memory « Always set your memory requests equal to your limits » you can layer high priority service and low priority service better if you use some buffer.

clhodapp 4y ago
My current model for memory on k8s tends to agree with the article. Would it be possible for you to explain "you can layer high priority service and low priority service better if you use some buffer" further?
- skyde 4y ago
  I mean think of low priority service as services that are not latency sensitive (background jobs). you want those low priory cgroup to use the extra memory if it’s available (not used by a high priority cgroup) it a high priority cgroup need the memory it steal it from the low priority cgroup up to the minimum guaranteed to this low priority cgroup. this low priority cgroup memory contention will turns into IO pressure from page faults. Then IO limit on the low priority cgroup will cap how much IO it can generate by throttling it. Meanwhile the high priority cgroup use it’s guaranteed cpu, guaranteed IO and guaranteed memory with no hiccup. to learn more check « fbtax2 memory controller configuration » at https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.html https://facebookmicrosites.github.io/cgroup2/docs/memory-con...
  - skyde 4y ago
    In Linux kernel, Different memory cgroups have distinct lruvecs, for memcg reclaim. This mean global reclaim algorithm can look at all cgroup "memory.low", and reclaim first from cgroups that are using more than their "memory.low" config. You need to make sure that the sum of all cgroup "memory.low" is still below total memory available. Leaving 20% buffer for when the whole system is under stress is a good rule of thumb. In addition, any memory that's guaranteed but unused can be allocated to other processes, further optimizing memory utilization.
dilyevsky 4y ago
The problem with that approach (overcommit) in k8s is when oom killer kicks in your containers will die immediately and thus you can easily corrupt some internal state if not careful. Also it’s easy to accidentally kill wrong process group bc oomkiller will calculate priority based on memory usage among other things so outsized processes can be killed before lower priority tiny processes

by skyde 4y ago

I don’t agree on the recommendation for memory « Always set your memory requests equal to your limits » you can layer high priority service and low priority service better if you use some buffer.

clhodapp 4y ago

My current model for memory on k8s tends to agree with the article. Would it be possible for you to explain "you can layer high priority service and low priority service better if you use some buffer" further?

skyde 4y ago
I mean think of low priority service as services that are not latency sensitive (background jobs). you want those low priory cgroup to use the extra memory if it’s available (not used by a high priority cgroup) it a high priority cgroup need the memory it steal it from the low priority cgroup up to the minimum guaranteed to this low priority cgroup. this low priority cgroup memory contention will turns into IO pressure from page faults. Then IO limit on the low priority cgroup will cap how much IO it can generate by throttling it. Meanwhile the high priority cgroup use it’s guaranteed cpu, guaranteed IO and guaranteed memory with no hiccup. to learn more check « fbtax2 memory controller configuration » at https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.html https://facebookmicrosites.github.io/cgroup2/docs/memory-con...
- skyde 4y ago
  In Linux kernel, Different memory cgroups have distinct lruvecs, for memcg reclaim. This mean global reclaim algorithm can look at all cgroup "memory.low", and reclaim first from cgroups that are using more than their "memory.low" config. You need to make sure that the sum of all cgroup "memory.low" is still below total memory available. Leaving 20% buffer for when the whole system is under stress is a good rule of thumb. In addition, any memory that's guaranteed but unused can be allocated to other processes, further optimizing memory utilization.

dilyevsky 4y ago

The problem with that approach (overcommit) in k8s is when oom killer kicks in your containers will die immediately and thus you can easily corrupt some internal state if not careful. Also it’s easy to accidentally kill wrong process group bc oomkiller will calculate priority based on memory usage among other things so outsized processes can be killed before lower priority tiny processes