It seems like there are enough failure modes that everyone has a different story about bad behavior, and unfortunately I don't think there's one solution a sysa

It seems like there are enough failure modes that everyone has a different story about bad behavior, and unfortunately I don't think there's one solution a sysadmin can adopt (e.g., swap or not, sysctl settings, cgroup settings) that reliably prevents them all. I've been guilty in the past of telling people turning off swap dramatically improves things, but in your scenario it sounds like the opposite is true. /shruggie The PSI-based oom daemon seems like an improvement, but I imagine there are still gaps there if it doesn't react quickly enough. Adding more parts can often paper over problems but rarely completely solve them...I think what's needed is a rethink: don't page these vital things out. Instead kill a process (chosen by userspace if possible, anything if not). Reserve enough for the killing rather than doing more reclamation to make it happen.