2 ms·

Actually Kubelet should already be adjusting OOM scores to make sure that user pods (containers) get killed over Kubelet or the Docker daemon. Why didn't that w

Actually Kubelet should already be adjusting OOM scores to make sure that user pods (containers) get killed over Kubelet or the Docker daemon. Why didn't that work here?

trhway 8y ago
adjusting scores for other processes skews the chances yet doesn't guarantee. The way to guarantee it for a given process is to disable the killer for that particular process.
- nimos 8y ago
  Interesting. The kubelet seems to use varying negative OOMAdjusts to prioritise killing[0] but if I'm reading the kernel code right anything at -999/-998 would return 1 from the badness function and essentially be equally valid to kill unless it was using over 99.9% of available memory.[1] I see OOMScoreAdjust=-999 for kubelet being used but why not -1000. -999 seems like it would be equally likely to be evicted as -998 unless the for_each_process(p) macro always goes first to last processes? [0]https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/qos/policy.go https://github.com/kubernetes/kubernetes/blob/master/pkg/kub... [1]https://elixir.bootlin.com/linux/latest/source/mm/oom_kill.c#L195 https://elixir.bootlin.com/linux/latest/source/mm/oom_kill.c...
  - trhway 8y ago
    seems that way to me too - everybody like kubelet and "guaranteed" containers gets 1. >unless the for_each_process(p) macro always goes first to last processes? It seems it would usually go first to the first processes - the macro below - i.e. it would get to the "top" processes like kubelet, docker, etc before the containers. #define next_task(p) \ list_entry_rcu((p)->tasks.next, struct task_struct, tasks) #define for_each_process(p) \ for (p = &init_task ; (p = next_task(p)) != &init_task ; ) Given that "chosen" is updated only "if (points > chosen_points)" it seems that the first listed process with score 1 will stay the "chosen" in that situation, ie. it will be one of the top processes like the [-999] kubelet, not a [-998] container. From a provider of an Azure class i'd have expected that they wouldn't rely on that machinery and would instead go the way of disabling the killer for the top processes outright.
- enitihas 8y ago
  Interesting, How does one disable the killer for a process? I thought on linux it was only possible to adjust oom scores.

Actually Kubelet should already be adjusting OOM scores to make sure that user pods (containers) get killed over Kubelet or the Docker daemon. Why didn't that work here?

trhway 8y ago

adjusting scores for other processes skews the chances yet doesn't guarantee. The way to guarantee it for a given process is to disable the killer for that particular process.

nimos 8y ago
Interesting. The kubelet seems to use varying negative OOMAdjusts to prioritise killing[0] but if I'm reading the kernel code right anything at -999/-998 would return 1 from the badness function and essentially be equally valid to kill unless it was using over 99.9% of available memory.[1] I see OOMScoreAdjust=-999 for kubelet being used but why not -1000. -999 seems like it would be equally likely to be evicted as -998 unless the for_each_process(p) macro always goes first to last processes? [0]https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/qos/policy.go https://github.com/kubernetes/kubernetes/blob/master/pkg/kub... [1]https://elixir.bootlin.com/linux/latest/source/mm/oom_kill.c#L195 https://elixir.bootlin.com/linux/latest/source/mm/oom_kill.c...
- trhway 8y ago
  seems that way to me too - everybody like kubelet and "guaranteed" containers gets 1. >unless the for_each_process(p) macro always goes first to last processes? It seems it would usually go first to the first processes - the macro below - i.e. it would get to the "top" processes like kubelet, docker, etc before the containers. #define next_task(p) \ list_entry_rcu((p)->tasks.next, struct task_struct, tasks) #define for_each_process(p) \ for (p = &init_task ; (p = next_task(p)) != &init_task ; ) Given that "chosen" is updated only "if (points > chosen_points)" it seems that the first listed process with score 1 will stay the "chosen" in that situation, ie. it will be one of the top processes like the [-999] kubelet, not a [-998] container. From a provider of an Azure class i'd have expected that they wouldn't rely on that machinery and would instead go the way of disabling the killer for the top processes outright.
enitihas 8y ago
Interesting, How does one disable the killer for a process? I thought on linux it was only possible to adjust oom scores.