3 ms·
>the AKS engineering team spent over a day helping identify that the user had over scheduled their nodes, by running applications without memory limit, resultin
by trhway 8y ago
>the AKS engineering team spent over a day helping identify that the user had over scheduled their nodes, by running applications without memory limit, resulting in the kernel oom (out of memory) killer terminating the Docker daemon and kubelet.
sounds like a bunch of people have just learned for the first time about OOM killer. I mean the production systems with overcommits and the running loose OOM killer and I bet without swap ... And they blame the customer. Sounds like a PaaS MVP quickly slapped together by an alpha state startup. You may want to look into man pages, in particular oom scoring and the code -17.
- praseodym 8y agoActually Kubelet should already be adjusting OOM scores to make sure that user pods (containers) get killed over Kubelet or the Docker daemon. Why didn't that work here?
- trhway 8y agoadjusting scores for other processes skews the chances yet doesn't guarantee. The way to guarantee it for a given process is to disable the killer for that particular process.
- nimos 8y agoInteresting. The kubelet seems to use varying negative OOMAdjusts to prioritise killing[0] but if I'm reading the kernel code right anything at -999/-998 would return 1 from the badness function and essentially be equally valid to kill unless it was using over 99.9% of available memory.[1] I see OOMScoreAdjust=-999 for kubelet being used but why not -1000. -999 seems like it would be equally likely to be evicted as -998 unless the for_each_process(p) macro always goes first to last processes? [0]https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/qos/policy.go https://github.com/kubernetes/kubernetes/blob/master/pkg/kub... [1]https://elixir.bootlin.com/linux/latest/source/mm/oom_kill.c#L195 https://elixir.bootlin.com/linux/latest/source/mm/oom_kill.c...
- trhway 8y agoseems that way to me too - everybody like kubelet and "guaranteed" containers gets 1. >unless the for_each_process(p) macro always goes first to last processes? It seems it would usually go first to the first processes - the macro below - i.e. it would get to the "top" processes like kubelet, docker, etc before the containers. #define next_task(p) \ list_entry_rcu((p)->tasks.next, struct task_struct, tasks) #define for_each_process(p) \ for (p = &init_task ; (p = next_task(p)) != &init_task ; ) Given that "chosen" is updated only "if (points > chosen_points)" it seems that the first listed process with score 1 will stay the "chosen" in that situation, ie. it will be one of the top processes like the [-999] kubelet, not a [-998] container. From a provider of an Azure class i'd have expected that they wouldn't rely on that machinery and would instead go the way of disabling the killer for the top processes outright.
- enitihas 8y agoInteresting, How does one disable the killer for a process? I thought on linux it was only possible to adjust oom scores.