6 ms·
This is a pretty bad mistake from the customer if this is true. If not done already it would probably be good to expose Prometheus metrics on CPU/Memory usage p
by specialp 8y ago
This is a pretty bad mistake from the customer if this is true. If not done already it would probably be good to expose Prometheus metrics on CPU/Memory usage per node.
- QiKe 8y agoWe are indeed working on more convenient container monitoring and logging on Azure portal.
- dilyevsky 8y agolol so aks forgot to provision enough resources and possibly setup enforcement and you are blaming the user? the user should be able to run as close to edge of "allocatable" as possible or even go over it and be oom kill'ed without bringing down the entire node. this functionality is even built into kubelet already. there's no way you can twist this to make it into user error.
- btown 8y agoMore generally I should be able to choose to run an interruptible workload that I know to leak memory. I should expect that if I don’t, one of my coworkers will, and the node will stay up. Not leaving enough RAM for the node’s core resources is a mistake, but far from the worst thing in the world.
- markbnj 8y agoYes, this is true on it's face: it's bad to deploy containers to k8s without appropriate resource limits. However, this should in no way affect the operation of the node, so the implied transfer of responsibility for this incident from AKS to the customer is invalid imo.
- sleepybrett 8y agoIt does if you don't have enough system reserved. It causes the kubelet to not function well and eventually the node goes sideways due to oom killing.
- markbnj 8y agoRight, I think we're saying the same thing. If the node is properly configured an end-user pod should not be able to take down the kubelet.