5 ms·
AKS engineering team spent over a day helping identify that the user had over scheduled their nodes, by running applications without memory limit, resulting in
by po 8y ago
AKS engineering team spent over a day helping identify that the user had over scheduled their nodes, by running applications without memory limit, resulting in the kernel oom (out of memory) killer terminating the Docker daemon and kubelet.
I'm a bit confused why the cluster nodes don't come configured like this out of the box... kubernetes users aren't supposed to have to worry about OOM of the underlying system killing ops-side processes are they?
- dilyevsky 8y agothey do if cluster admin didn't setup proper system-reserved and kube-reserved (both are kubelet flags) and configured enforcement.
- alpb 8y agoIn this case, the cluster admin would be whoever is provisioning the cluster nodes. In Google Kubernetes Engine, the "Capacity" and "Allocatable" info shown on the nodes are different (I see some mem/cpu reserved for probably system stuff). This makes me think GKE probably subtracts node capacity from what's allocated for the system automatically. P.S. I work at Google.
- dilyevsky 8y agocorrect, it should be provisioned by k8s provider (AKS in this case) and is what GKE is doing https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu https://cloud.google.com/kubernetes-engine/docs/concepts/clu... note, it also needs to match node configuration (how cgroups are setup specifically) so I doubt this works well on EKS which is BYO node. Maybe it's the issue with AKS too, I don't know enough about how it works...
- QiKe 8y agoAKS now reserves 20% of memory from each agent node and a very small amount of CPU to protect docker daemon and kubelet to function with misbehaving customer pods. However, that just means customer's pods will be evicted or no place to schedule when all resource is used up. This is something we see now in customer support cases.