10 ms·

Guide to OOMKill Alerting in Kubernetes Clusters

65 points by draganm 6y ago

geoffbp 6y ago
This fails to load for me
- uturingmachine 6y ago
  Probably ran out of memory.
- yyyk 6y ago
  https://archive.is/DDHQT https://archive.is/DDHQT
- draganm 6y ago
  Sorry for that - seems I've under-estimated possible traffic. Scaled up the server a bit now.
  - hendry 6y ago
    Scaled up a blog... lol
    - draganm 6y ago
      When the blog is on a $5/month DO droplet and it does its own TLS termination that was necessary.
ec109685 6y ago
Another hidden issue is that as a container gets close to running out of memory, it furiously drops read only pages from memory, only to need to read some of them back into memory moments later. This pathological swapping behavior can impact other workloads on the system. cgroups2 has better protections against this behavior.
- throwii 6y ago
  Is there an issue on cgroups2 adoption for Kubernetes somewhere?
  - The_rationalist 6y ago
    https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/20191118-cgroups-v2.md https://github.com/kubernetes/enhancements/blob/master/keps/...
    - throwii 6y ago
      Thank you!
- segmondy 6y ago
  I think the issue is that your nodes have swaps. Why will you have swap on container nodes? IMO, the idea with container management is to get predictability with resources. If you have 8gb on a node, you know that the containers get 8gb. You might not be able to tell exactly how based on how it's configured, but you know once they collectively use 8gb, that's it. Swap is going to mess up things really bad in ways you can't even predict.
  - johncolanduoni 6y ago
    Even without swap enabled or any explicit memory mapping, read only pages from executables (code, read-only data) are mapped into the process’ address space and may be evicted. Unless you explicitly lock those into RAM they still behave somewhat like swapped memory does, except the pages don’t need to be written back.
- Technically 6y ago
  This is why, without specific concerns, you set memory alerts at a high water mark rather than at full memory.
- pflanze 6y ago
dilatedmind 6y ago
would it have been sufficient to alert on high memory usage? It might be reasonable to set an alert on say 70% rss. As long as the pod does not pass this threshold and die before a metric can be sampled. that "no such file or directory" looks to be coming from building a dynamic executable on debian and trying to run it on alpine.
- draganm 6y ago
  as for the first question - that wouldn't be enough. AFAIK mmap-ed pages are part of RSS and it's quite usual for them to use up everything up to the memory limit (databases kind of rely on this 'feature'). None of that would provoke an OOMKill. for the second comment - I've used images the author has published on Docker hub. Maybe there would've been a way to make it work, but if you take a look at the amount of code in missing-container-metrics, you will realise that I've used less time to write that than I would've spent debugging someone else's Docker build and golang code that is not really maintained.
- linsomniac 6y ago
  I mean that's fine if you're ok with 30% wasted memory... We just recently had to tune some JVM and monitoring settings because we do the initial and max heap allocation to around 90% memory. There's very little else going on.
linsomniac 6y ago
I have an Icinga2 monitor on all my hosts for "dmesg" output to include the OOM killer string, and alert on that. Very useful.
bpaliz 6y ago
At last year's KubeCon someone presented https://github.com/opsgenie/kubernetes-event-exporter https://github.com/opsgenie/kubernetes-event-exporter
- hagmonk 6y ago
  Not going to help you in this case; Kubernetes does not log an OOMKilled event. Issue tracking this has been open for some time: https://github.com/kubernetes/kubernetes/issues/69676 https://github.com/kubernetes/kubernetes/issues/69676 Tracking OOMKilled counts via the kubelet also had an open issue which was closed without being fixed: https://github.com/kubernetes/kubernetes/pull/87856 https://github.com/kubernetes/kubernetes/pull/87856
hendry 6y ago
Use serverless guys. k8s is a train wreck of needless complexity for 99% of developers.
- chokeartist 6y ago
  While I acknowledge you probably have solved for your use-case... I can't help but hardcore LOL at your somewhat terse perspective! Dude... K8S just got mature!
- outworlder 6y ago
  The hate on K8s is misplaced in this case. You should redirect it to containers. Other container orchestration mechanisms will encounter similar, if not the same, issues.
- gautamdivgi 6y ago
  k8s unfortunately is the only way to maintain sanity if you want to maintain a multi-cloud environment. I don't relish the idea of duplicating functionality but maintaining code across aws lambda's and azure functions.
- justincormack 6y ago
  Running out of memory vs wasting memory is also an issue on serverless although somewhat less extreme. Part of the issue is no one knows how much memory their code might use, and we don’t have frameworks that adapt to available memory much. Resource constrained computing is hard.
m1keil 6y ago
What about healthchecks?
- jacques_chester 6y ago
  I don't think it would work to report an OOMkill. The process that would answer the healthcheck probe would be gone.
  - m1keil 6y ago
    It's not clear to me what happens after the OOM. Does the init process restarts the daemon? I would argue that it shouldn't. If the process stops responding to a healthcheck, it's the scheduler's responsibility (k8s in this case) to handle it. Crashes in this case should be handled in a similar way, whether it's due to OOM or a bug. Maybe I'm missing another scenario here?
    - dharmab 6y ago
      > Does the init process restarts the daemon? In systemd, this depends on what the Restart option in the service unit is set to. The default is to not restart. https://www.freedesktop.org/software/systemd/man/systemd.service.html https://www.freedesktop.org/software/systemd/man/systemd.ser...
- zimbatm 6y ago
  They are complementary. If a sub-process gets OOMKilled and the container doesn't die, then it's most likely that the parent process didn't handle that scenario. In which case the health-check wouldn't cover that issue.
jeppesen-io 6y ago
Thank you so much!!!! I've been trying to find something like this

65 points by draganm 6y ago

geoffbp 6y ago

This fails to load for me

uturingmachine 6y ago
Probably ran out of memory.
yyyk 6y ago
https://archive.is/DDHQT https://archive.is/DDHQT
draganm 6y ago
Sorry for that - seems I've under-estimated possible traffic. Scaled up the server a bit now.
- hendry 6y ago
  Scaled up a blog... lol
  - draganm 6y ago
    When the blog is on a $5/month DO droplet and it does its own TLS termination that was necessary.

ec109685 6y ago

Another hidden issue is that as a container gets close to running out of memory, it furiously drops read only pages from memory, only to need to read some of them back into memory moments later. This pathological swapping behavior can impact other workloads on the system. cgroups2 has better protections against this behavior.

throwii 6y ago
Is there an issue on cgroups2 adoption for Kubernetes somewhere?
- The_rationalist 6y ago
  https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/20191118-cgroups-v2.md https://github.com/kubernetes/enhancements/blob/master/keps/...
  - throwii 6y ago
    Thank you!
segmondy 6y ago
I think the issue is that your nodes have swaps. Why will you have swap on container nodes? IMO, the idea with container management is to get predictability with resources. If you have 8gb on a node, you know that the containers get 8gb. You might not be able to tell exactly how based on how it's configured, but you know once they collectively use 8gb, that's it. Swap is going to mess up things really bad in ways you can't even predict.
- johncolanduoni 6y ago
  Even without swap enabled or any explicit memory mapping, read only pages from executables (code, read-only data) are mapped into the process’ address space and may be evicted. Unless you explicitly lock those into RAM they still behave somewhat like swapped memory does, except the pages don’t need to be written back.
Technically 6y ago
This is why, without specific concerns, you set memory alerts at a high water mark rather than at full memory.
pflanze 6y ago

dilatedmind 6y ago

would it have been sufficient to alert on high memory usage? It might be reasonable to set an alert on say 70% rss. As long as the pod does not pass this threshold and die before a metric can be sampled. that "no such file or directory" looks to be coming from building a dynamic executable on debian and trying to run it on alpine.

draganm 6y ago
as for the first question - that wouldn't be enough. AFAIK mmap-ed pages are part of RSS and it's quite usual for them to use up everything up to the memory limit (databases kind of rely on this 'feature'). None of that would provoke an OOMKill. for the second comment - I've used images the author has published on Docker hub. Maybe there would've been a way to make it work, but if you take a look at the amount of code in missing-container-metrics, you will realise that I've used less time to write that than I would've spent debugging someone else's Docker build and golang code that is not really maintained.
linsomniac 6y ago
I mean that's fine if you're ok with 30% wasted memory... We just recently had to tune some JVM and monitoring settings because we do the initial and max heap allocation to around 90% memory. There's very little else going on.

linsomniac 6y ago

I have an Icinga2 monitor on all my hosts for "dmesg" output to include the OOM killer string, and alert on that. Very useful.

bpaliz 6y ago

At last year's KubeCon someone presented https://github.com/opsgenie/kubernetes-event-exporter https://github.com/opsgenie/kubernetes-event-exporter

hagmonk 6y ago
Not going to help you in this case; Kubernetes does not log an OOMKilled event. Issue tracking this has been open for some time: https://github.com/kubernetes/kubernetes/issues/69676 https://github.com/kubernetes/kubernetes/issues/69676 Tracking OOMKilled counts via the kubelet also had an open issue which was closed without being fixed: https://github.com/kubernetes/kubernetes/pull/87856 https://github.com/kubernetes/kubernetes/pull/87856

hendry 6y ago

Use serverless guys. k8s is a train wreck of needless complexity for 99% of developers.

chokeartist 6y ago
While I acknowledge you probably have solved for your use-case... I can't help but hardcore LOL at your somewhat terse perspective! Dude... K8S just got mature!
outworlder 6y ago
The hate on K8s is misplaced in this case. You should redirect it to containers. Other container orchestration mechanisms will encounter similar, if not the same, issues.
gautamdivgi 6y ago
k8s unfortunately is the only way to maintain sanity if you want to maintain a multi-cloud environment. I don't relish the idea of duplicating functionality but maintaining code across aws lambda's and azure functions.
justincormack 6y ago
Running out of memory vs wasting memory is also an issue on serverless although somewhat less extreme. Part of the issue is no one knows how much memory their code might use, and we don’t have frameworks that adapt to available memory much. Resource constrained computing is hard.

m1keil 6y ago

What about healthchecks?

jacques_chester 6y ago
I don't think it would work to report an OOMkill. The process that would answer the healthcheck probe would be gone.
- m1keil 6y ago
  It's not clear to me what happens after the OOM. Does the init process restarts the daemon? I would argue that it shouldn't. If the process stops responding to a healthcheck, it's the scheduler's responsibility (k8s in this case) to handle it. Crashes in this case should be handled in a similar way, whether it's due to OOM or a bug. Maybe I'm missing another scenario here?
  - dharmab 6y ago
    > Does the init process restarts the daemon? In systemd, this depends on what the Restart option in the service unit is set to. The default is to not restart. https://www.freedesktop.org/software/systemd/man/systemd.service.html https://www.freedesktop.org/software/systemd/man/systemd.ser...
zimbatm 6y ago
They are complementary. If a sub-process gets OOMKilled and the container doesn't die, then it's most likely that the parent process didn't handle that scenario. In which case the health-check wouldn't cover that issue.

jeppesen-io 6y ago

Thank you so much!!!! I've been trying to find something like this