5 ms·

You can disable the oom killer if you think its heuristics are misfiring. > OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of g

You can disable the oom killer if you think its heuristics are misfiring. > OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of gigabytes) of unallocated memory because it can't evict the buffer cache quickly enough Is that due to dirty pages? Have you tried tweaking the dirty ratio to get dirty files flushed sooner? I believe there were also some improvements in the default behavior towards the end of the 4.x series that are supposed to result in more steady flushing.

wahern 7y ago
> You can disable the oom killer if you think its heuristics are misfiring. You can't completely disable the OOM killer. Some in-kernel allocations will trigger an OOM reap and kill regardless. Too many parts of Linux were written with the assumption of overcommit. Heck, as this issue shows even when there's technically free memory the OOM killer can still kick in. I had heard about these issues before but never saw that in practice--at least not often enough that I looked closely enough. I knew Google and Facebook and others have been relying on userspace OOM killers for years to improve reliability and consistency. But only after having recently joined a DevOps team at a large enterprise with large enterprise, big data customers did I begin to see the depth and breadth of the problem. Before getting into the hell-hole of helpless despair that is DevOps I primarily did systems programming. And, for the record, I always made sure to handle allocation failure in those languages where I could (C, Lua, etc) because as a software engineer it was always clear to me that, much as with security, an over reliance on policy-based mechanisms invariably resulted in the worst possible QoS. (Of course, in Linux even disabling overcommit is no guarantee that page fault won't trigger an OOM kill, but there are other techniques to improve reliability under load.) > Is that due to dirty pages? Yes, people have tweaked and turned those knobs and reduced incidence rate by maybe 20-30%. Yet nobody is sleeping better at night. Computing shouldn't be a black art. The irony is that overcommit and loose memory accounting more generally was originally intended to remove the necessity for professional system administrators to babysit a system to keep it humming along. But now we've come completely full circle. And it was entirely predictable. And before anyone says that cloud computing means you increase reliability by scaling horizontally, well "cloud scale" systems is exactly what are being run. But unless you massively over provision, and especially when certain tasks, even when split up across multiple nodes, can take minutes or even hours, an OOM kill incident can reverberate and cascade. Again, ironic because the whole idea of loose memory accounting is supposedly to obviate the need to over provision.

wahern 7y ago

> You can disable the oom killer if you think its heuristics are misfiring. You can't completely disable the OOM killer. Some in-kernel allocations will trigger an OOM reap and kill regardless. Too many parts of Linux were written with the assumption of overcommit. Heck, as this issue shows even when there's technically free memory the OOM killer can still kick in. I had heard about these issues before but never saw that in practice--at least not often enough that I looked closely enough. I knew Google and Facebook and others have been relying on userspace OOM killers for years to improve reliability and consistency. But only after having recently joined a DevOps team at a large enterprise with large enterprise, big data customers did I begin to see the depth and breadth of the problem. Before getting into the hell-hole of helpless despair that is DevOps I primarily did systems programming. And, for the record, I always made sure to handle allocation failure in those languages where I could (C, Lua, etc) because as a software engineer it was always clear to me that, much as with security, an over reliance on policy-based mechanisms invariably resulted in the worst possible QoS. (Of course, in Linux even disabling overcommit is no guarantee that page fault won't trigger an OOM kill, but there are other techniques to improve reliability under load.) > Is that due to dirty pages? Yes, people have tweaked and turned those knobs and reduced incidence rate by maybe 20-30%. Yet nobody is sleeping better at night. Computing shouldn't be a black art. The irony is that overcommit and loose memory accounting more generally was originally intended to remove the necessity for professional system administrators to babysit a system to keep it humming along. But now we've come completely full circle. And it was entirely predictable. And before anyone says that cloud computing means you increase reliability by scaling horizontally, well "cloud scale" systems is exactly what are being run. But unless you massively over provision, and especially when certain tasks, even when split up across multiple nodes, can take minutes or even hours, an OOM kill incident can reverberate and cascade. Again, ironic because the whole idea of loose memory accounting is supposedly to obviate the need to over provision.