3 ms·
> Monitoring container_memory_working_set_bytes still makes sense since it is the metric that kublet uses to kill the pod when it exceeds the limits. This is n
by jeffbee 2y ago
> Monitoring container_memory_working_set_bytes still makes sense since it is the metric that kublet uses to kill the pod when it exceeds the limits.
This is not a great way to describe it. When a container is out of memory then the kernel ends it, and the kubelet is not involved. This is the main way that most users will experience OOM conditions. Note that it is not possible for RSS to exceed limit, because the task will end the instant it tries to realize memory that would have put it over the limit.
When the node is under total memory pressure then the kubelet uses working set to rank pods for eviction. Working set is used for that in an attempt to attribute kernel resources like page caches to each control group. But eviction due to node memory pressure should be rare.
- zokier 2y agoI think this part really needs clarification, because it feels odd that containers would get scaled out or killed based on disk cache usage. Like what was actually the thing that triggered this investigation, its not quite clear? > And then the hell broke loose. I received a message from our devops team that the application was scaling like crazy - we had a memory leak! what sort of scaling are they referring to here
- jeffbee 2y agoPossibly they were using a pod autoscaler that was keyed off this metric.