3 ms·
One of the things I am greatful for kubernetes and the killing of pods. Had a similar problem but memory wise with a pesky memory leak, and the short term solu
by coolgoose 2y ago
One of the things I am greatful for kubernetes and the killing of pods.
Had a similar problem but memory wise with a pesky memory leak, and the short term solution was to do nothing as instances would to do nothing.
- maximinus_thrax 2y agoDuring one of my past gigs, this exact feature hid a huge memory leak, in old code, always running on k8s which we found out only when we moved some instances to bare metal.
- esprehn 2y agoWe hit this in a past gig too. One of the big services had a leak, but deployed every 24 hours which was hiding it. When the holiday deploy freeze hit the pods lived much longer than normal and caused an OOM storm. At first I thought maybe we should add a "hack" to cycle all the pods over 24 hours old, but then I wondered if making holiday freezes behave like normal weeks was really a hack at all or just reasonable predictability. In the end folks managed to fix the leak and we didn't resolve the philosophical question though.