3 ms·
Phantom Menance: memory leak that wasn't there
- JohnMakin 2y agoGreat writeup, have definitely been bit by container_memory_working_set_bytes before - in fact when I first read the description of the dashboard I was wondering if it was something like that. This is one of the most difficult parts of working in SRE or DevOps, often each side of the conversation lacks the necessary context to understand what the problem is. As author mentioned, this is still a worthwhile metric to monitor, but its meaning is not necessarily clear in this context. Kudos to this author for digging in - as a DevOps/SRE guy, I'd imagine this conversation often going in companies I have worked for like "Something is wrong with your dashboard" and my team being like "something is wrong with your application" and nothing gets done for months while managers point fingers and figure out whose problem it is.
- jeffbee 2y ago> Monitoring container_memory_working_set_bytes still makes sense since it is the metric that kublet uses to kill the pod when it exceeds the limits. This is not a great way to describe it. When a container is out of memory then the kernel ends it, and the kubelet is not involved. This is the main way that most users will experience OOM conditions. Note that it is not possible for RSS to exceed limit, because the task will end the instant it tries to realize memory that would have put it over the limit. When the node is under total memory pressure then the kubelet uses working set to rank pods for eviction. Working set is used for that in an attempt to attribute kernel resources like page caches to each control group. But eviction due to node memory pressure should be rare.
- zokier 2y agoI think this part really needs clarification, because it feels odd that containers would get scaled out or killed based on disk cache usage. Like what was actually the thing that triggered this investigation, its not quite clear? > And then the hell broke loose. I received a message from our devops team that the application was scaling like crazy - we had a memory leak! what sort of scaling are they referring to here
- jeffbee 2y agoPossibly they were using a pod autoscaler that was keyed off this metric.
- jmugan 2y agoLove the title!