3 ms·
It almost always happens at some point. Someone fat fingers a memory allocation. Memory leak from a long running process finally grows too big, someone installs
by lazide 4y ago
It almost always happens at some point. Someone fat fingers a memory allocation. Memory leak from a long running process finally grows too big, someone installs an update and it doubles memory usage on something, etc.
I had it happen recently that the backup/sync software for a NAS had a constant factor memory consumption based on the number of files it was syncing. Transferred in a bunch of data, and blam. Commercial NAS, and they used swap (shitty, never buy QNAP). Wedged so hard it took a hard power cycle to even get console, AND caused data corruption in the ZFS pools.
It’s what happens next that decides the stability of the system.
If it can grow into swap and keep going, things start crawling, load builds up, buffers expand, and the system eventually grinds to a halt. On Linux, often in a really wedged and irritating/impossible to fix way.
Or, if no swap, OOM killer shoots something in the head (hopefully the offender), and we’re back to normal (minus the thing it killed, which is usually the thing you didn’t want it to - but at least the system works and you know something is wrong).
In either case, caches have dropped to near zero awhile ago so performance is already getting bad. It’s if we enter a death spiral, or get death early enough the whole system doesn’t spiral.
- Dylan16807 4y ago> In either case, caches have dropped to near zero awhile ago so performance is already getting bad. Right. You can get pretty deep into a death spiral even without swap. I feel like the better-performing solution is to make OOM trigger earlier, and to then go ahead and have swap be on. Somewhat tangentially, I'd really like a setting for minimum disk cache, and that would do so much to help prevent thrashing.