4 ms·

Articles like this further cement my belief that Linux's low memory handling is deeply flawed. (macOS's too, btw.) My experience with Linux machines that have

Articles like this further cement my belief that Linux's low memory handling is deeply flawed. (macOS's too, btw.) My experience with Linux machines that have HDDs and swap is that once they start swapping, they slow down not only until memory is freed (ctrl-C gets through, OOM killer runs, whatever) but actually until reboot. My theory is that processes' VM space is now a minefield. Both anonymous memory and file-backed mmaps now have many gaps which get paged in on demand with inadequate readahead. With each seek taking ~10 ms, it doesn't take too much of that before you should give up and just reboot. Once memory is available, I'd like the system to proactively page back in both swap and file-backed pages to prevent this. If it does need to happen reactively, it should be in big blocks (each 10 ms of seeking should be follow by at least 10 ms of reading => ~1 MiB, not ~4 KiB or whatever it's doing now). By default. I'm not convinced the decision to use swap is ever a good one. E.g., from the article: > With swap: We can choose to swap out rarely-used anonymous memory that may only be used during a small part of the process lifecycle, allowing us to use this memory to improve cache hit rate, or do other optimisations. This is short-sighted. Programs can be written to tolerate their filesystem I/O being slow. They often take care to do I/O on a particular thread which is not handling network or GUI operations. They can't realistically be written to tolerate arbitrary memory accesses (including program text, stack, and heap) being that slow. It's a stupid idea to potentially slow down the latter to (try to) speed up the former. And programs shouldn't all have to mlockall, which iirc has other bad consequences. (maybe it allocates backing pages for all mappings immediately? even stuff like stack guard pages that's pure waste?) I have written programs to mlock (just) their program text; I'd prefer that weren't necessary either. I prefer that under memory pressure, a task dies promptly. Ideally the "right" task, but really it just has to be better than making the system unusable for a long time. Even if it kills the display server or pid 1 inappropriately, I was probably going to have to reboot anyway, so...

zozbot234 5y ago
> I prefer that under memory pressure, a task dies promptly. This is exactly what memory pressure indicators can be used for. Linux also supports volatile page ranges, that get wiped out under memory pressure even with no swap support.
IshKebab 5y ago
I agree. Another issue with Linux is that it provides no way to choose which tasks to kill when you start heavily swapping, and the OOM killer won't do anything because it still thinks everything is fine. You can't just use the normal UI to kill tasks at that point because it will have slowed to a crawl due to all the swapping. Back when I used Linux with spinning disks it was much better to disable swap and hope the OOM killer did something sane than to wait for 10 minutes for a terminal to open and eventually give up and hard restart. Windows handles it much better. You just press ctrl-alt-delete and because the GUI is properly integrated into the kernel it can always react and pause other tasks even if the system is heavily loaded.
- xxs 5y ago
  >no way to choose which tasks to kill when you start heavily swapping This is false, of course. oom_score_adj, or just read on oom_killer.
  - pas 5y ago
    That's not a choice, it's a predetermined bias. Anyway, the "real solution" is oomd + PSI (pressure stall information), which can detect when memory is running low and the system is affected.
ec109685 5y ago
The os will drop program text if it encounters memory pressure even if swap is disabled. Anything that isn’t anonymous memory is fair game and it does it for even well behaved programs on the system.
- scottlamb 5y ago
  I mentioned that, yes. It's also problematic, particularly on HDD systems. (SSDs are fast enough that it's mostly okay.) This is why I said "Linux's low memory handling is deeply flawed" rather than just "turning off swap fixes everything".
  - ec109685 5y ago
    Got it. Totally agreed that it's deeply flawed. Without swap, even with SSD's, the system gets into a very bad state when a cgroup is close to running out of memory. The behavior ends up being that the kernel ends up constantly swapping out everything but anonymous memory until it can't anymore and then it finally terminates the process when it can't page anything else out. That can unfortunately impact other process on the system during that time.
    - scottlamb 5y ago
      It seems like there are enough failure modes that everyone has a different story about bad behavior, and unfortunately I don't think there's one solution a sysadmin can adopt (e.g., swap or not, sysctl settings, cgroup settings) that reliably prevents them all. I've been guilty in the past of telling people turning off swap dramatically improves things, but in your scenario it sounds like the opposite is true. /shruggie The PSI-based oom daemon seems like an improvement, but I imagine there are still gaps there if it doesn't react quickly enough. Adding more parts can often paper over problems but rarely completely solve them...I think what's needed is a rethink: don't page these vital things out. Instead kill a process (chosen by userspace if possible, anything if not). Reserve enough for the killing rather than doing more reclamation to make it happen.
      - 5y ago

zozbot234 5y ago

> I prefer that under memory pressure, a task dies promptly. This is exactly what memory pressure indicators can be used for. Linux also supports volatile page ranges, that get wiped out under memory pressure even with no swap support.

IshKebab 5y ago

I agree. Another issue with Linux is that it provides no way to choose which tasks to kill when you start heavily swapping, and the OOM killer won't do anything because it still thinks everything is fine. You can't just use the normal UI to kill tasks at that point because it will have slowed to a crawl due to all the swapping. Back when I used Linux with spinning disks it was much better to disable swap and hope the OOM killer did something sane than to wait for 10 minutes for a terminal to open and eventually give up and hard restart. Windows handles it much better. You just press ctrl-alt-delete and because the GUI is properly integrated into the kernel it can always react and pause other tasks even if the system is heavily loaded.

xxs 5y ago
>no way to choose which tasks to kill when you start heavily swapping This is false, of course. oom_score_adj, or just read on oom_killer.
- pas 5y ago
  That's not a choice, it's a predetermined bias. Anyway, the "real solution" is oomd + PSI (pressure stall information), which can detect when memory is running low and the system is affected.

ec109685 5y ago

The os will drop program text if it encounters memory pressure even if swap is disabled. Anything that isn’t anonymous memory is fair game and it does it for even well behaved programs on the system.

scottlamb 5y ago
I mentioned that, yes. It's also problematic, particularly on HDD systems. (SSDs are fast enough that it's mostly okay.) This is why I said "Linux's low memory handling is deeply flawed" rather than just "turning off swap fixes everything".
- ec109685 5y ago
  Got it. Totally agreed that it's deeply flawed. Without swap, even with SSD's, the system gets into a very bad state when a cgroup is close to running out of memory. The behavior ends up being that the kernel ends up constantly swapping out everything but anonymous memory until it can't anymore and then it finally terminates the process when it can't page anything else out. That can unfortunately impact other process on the system during that time.
  - scottlamb 5y ago
    It seems like there are enough failure modes that everyone has a different story about bad behavior, and unfortunately I don't think there's one solution a sysadmin can adopt (e.g., swap or not, sysctl settings, cgroup settings) that reliably prevents them all. I've been guilty in the past of telling people turning off swap dramatically improves things, but in your scenario it sounds like the opposite is true. /shruggie The PSI-based oom daemon seems like an improvement, but I imagine there are still gaps there if it doesn't react quickly enough. Adding more parts can often paper over problems but rarely completely solve them...I think what's needed is a rethink: don't page these vital things out. Instead kill a process (chosen by userspace if possible, anything if not). Reserve enough for the killing rather than doing more reclamation to make it happen.
    - 5y ago