8 ms·
Modernizing Linux swapping: introducing the swap table
- FooBarWidget 8mo agoOne pet peeve I have with virtual memory management on Linux is that, as memory usage approaches 100%, the kernel starts evicting executable pages because technically they're read-only and can be loaded from disk. Thus, the entire system grinds to a halt in a behavior that looks like swapping, because every program that wants to execute instructions has to load its instructions from disk again, only to have those instruction pages be evicted again when context switching to another program. This behavior is especially counter intuitive because disabling swap does not prevent this problem. There are no convenient settings for administrators for preventing this problem. It's good that we have better swapping now, but I wish they'd address the above. I'd rather have programs getting OOMKilled or throwing errors before the system grinds to a halt, where I can't even ssh in and run 'ps'.
- robinsonb5 8mo agoIndeed. I think what's really needed is some way to mark pages as "required for interactivity" so that nothing related to the user interface gets paged out, ever. That, I think, would go at least some way towards restoring the feeling of "having a computer's full attention" that we had thirty years ago.
- akdev1l 8mo agoSeems the applications can call mlockall() to do this
- direwolf20 8mo agoAn Electron app would mark its entire 2GB as required for interactivity. If you run 4 electron apps on an 8GB system you run out of memory.
- robinsonb5 8mo agoI don't mean interactivity within apps, per se - I mean the desktop and underlying OS, so that if an electron app goes unresponsive and eats all the free RAM the window manager can still kill it. Or you can still open a new terminal window, log in and kill it. Right now it can take several minutes to get a Linux system back under control once a swapstorm starts.
- M95D 8mo agoAlt+[SysRq,f] Or Alt+[SysRq,h] for help
- dingaling 8mo agoNo effect, captain. In 30 years of using desktop Linux I've never been able to interrupt a swapstorm. The only way out is long-press the power button.
- M95D 8mo agoIt always works for me, including on SBCs over TTL serial port. Always. Never had a situation where invoking the OOM killer by sysrq didn't solve the swap storm. On your system it probably doesn't work at all, not even when idle. It can only be 3 reasons: - Your kernel doesn't have it. You probably have a generic kernel provided by the package manager without this feature enabled. I can't really help you here. I always build my own kernel from source (+ patches). - Something hijacks your keyboard input. If you have a console already opened and logged in as root, you can "echo f > /proc/sysrq-trigger". Else, you can try setting up a permanent serial console and send the command from another computer. CTRL+2 then the command letter (f). The magic sysrq key over serial console is a separate kernel option that needs to be enabled. - You're doing it wrong. On laptop keyboards, keyboards less than 104/105 keys, sysrq is one of the first keys to be removed. Getting it pressed with Fn combinations... Good luck with that!
- SAI_Peregrinus 8mo agoLinux doesn't really have any distinction between the desktop and underlying OS components in userspace and anything else in userspace. Linux is quite userland-agnostic, and distros have traditionally mixed user software with distro-managed software. You shouldn't use `sudo` to install software by default, your package manager should allow installing software for just your user. Software installed for the system could then be the only software allowed to mark itself as required for interactivity. You could do that manually to other software if you had root access, but "normal" user software installs with the package manager couldn't do so since they wouldn't get root access. That'd require some new capabilities added, and some substantial shifts in how distro maintainers & users operate, so it's extremely unlikely. It's much closer to how things like Android operate, though still not quite as secure as giving each application its own user & dedicated storage for data.
- FooBarWidget 8mo agoThere is, mlock() or mlockall(), but it requires developer support. I wish there is an administrator knob that allows me to mark whole processes without needing to modify them.
- man8alexd 8mo agoThere is cgroup memory.min
- nolist_policy 8mo agoLinux swap has been fixed on Chromebooks for years thanks to MGLRU. It's upstream since Linux 6.1 and you can try it with echo y >/sys/kernel/mm/lru_gen/enabled
- tremon 8mo agoDocumentation links: https://docs.kernel.org/next/admin-guide/mm/multigen_lru.html https://docs.kernel.org/next/admin-guide/mm/multigen_lru.htm... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/mm/multigen_lru.rst https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- M95D 8mo agoI had nothing but problems since that was introduced in 6.1. It seems that the kernel prefers to compact/defrag memory, repeatedly, each time freezing everything 1-2 seconds, rather than releasing some disk cache memory or swapping out.
- man8alexd 8mo agoActively used executable pages are explicitly excluded from reclaim. And if they are not used, why should they stay in memory when the memory is constrained? It is not the first time I have heard complaints about executable pages, but it seems to be some kind of common misunderstanding. https://news.ycombinator.com/item?id=45369516 https://news.ycombinator.com/item?id=45369516
- FooBarWidget 8mo agoWhat is "actively used"? The bash session that I was using 2 seconds before system grinded to a halt sure didn't count.
- man8alexd 8mo agoYour bash session is most likely still in memory but the system is spending 99.9% of time waiting for swap I/O, trying to free a few pages of memory. Swap random access latency is 10^3 slower than RAM.
- FooBarWidget 8mo agoThe system has swap disabled.
- man8alexd 8mo agoThen see the link in my previous comment.
- fulafel 8mo agoThe set of possibly blocking operations for interactively using bash is big. Executable pages of the bash executable are far from the only thing that could be missing. If the machine is swap trashing, all i/o goes to the same congested queue. .bash_history read or write access, memory allocation, stuff your terminal program does, stuff your wayland compositor or X11 stack does, bash accessing data in its memory that has been swapped out, etc etc. And each of those could be waiting for a while to issue their IO request since the IO system is flooded by swap IO. There should be a tool that can show the interdependent graph of pending, blocking io operations.
- worldsavior 8mo agoProgram instructions size is small thus loading is fast, so no need to worry about that too much. I'd look on different things first.
- 112233 8mo agoIs there a way to make linux kernel schedule in a "batch friendly way"? Say I do "make -j" and get 200 gcc processes diong jobserver LTO link with 2GB RSS each. In my head, optimal way through such mess is get as many processes as can fit into RAM without swapping, run them to completion, and schedule additional processes as resources become available. A depth first, "infinite latency" mode. Any combination of cgroups, /proc flags and other forbidden knobs to get such behaviour?
- direwolf20 8mo agoIt's not possible for the kernel to predict the memory needs of a process unfortunately
- man8alexd 8mo agoIt is possible to measure process memory utilitsation and set appropriate cgroup limits.
- 112233 8mo agoBut how about not scheduling swapped out processes if there currently is no free ram for their current RSS? of course kernel cannot know that a new process will balloon to eat all RAM, but once it has done so, is there a way to let it run to completion without being swapped out to "improve responsivity"?
- man8alexd 8mo agoThere is no actual swapping in the modern kernels. Nowadays, it is paging, when the kernel pages out individual unused memory pages, not entire processes, so it keeps all non-blocked processes running, but only necessary memory pages in the memory.
- Neywiny 8mo ago"make -j" has OOMd me more than it's worth. If it's a big project I just put in how many threads I want. I do hear your point but that is a solved problem.
- Rygian 8mo agoI suffer from the same behavior, ever since I moved from Ubuntu to Debian. An interactive system that does not interact (terminal not reactive, can't ssh in, screen does not refresh) is broken. I don't understand why this is not a kernel bug. On my system, to add insult to injury, when the system does come back twenty minutes later, I get a "helpful" pop-up from the Linux Kernel saying "Memory Shortage Avoided". Which is just plain wrong. The pop-up should say "sorry, the kernel bricked your system for a solid twenty minutes for no good reason, please file a report".
- rustyhancock 8mo agoThis explains some of the issues I was having on a laptop some months back. And searching desperately for "just kill the damn thing" option.
- garaetjjte 8mo agoIf you enable Magic SysRq you can invoke OOM-killer from keyboard. https://docs.kernel.org/admin-guide/sysrq.html https://docs.kernel.org/admin-guide/sysrq.html
- AtlasBarfed 8mo agoIs it as bad with ssd?
- iberator 8mo agoAnother useless feature into Linux kernel. Who uses swap space nowadays?! Last time I used swap on Linux device was around Pentium 2 era but in reality closer to 486DX era
- Titan2189 8mo agoWe use it in production. Workloads with unpredictable memory usage (32Mb to 4Gb per process), but we also want to start enough processes to saturate the CPU. Before we configured & enabled swap we were either sitting at low CPU utilisation or OOM
- sl-1 8mo agoIt is still useful for many workloads, I use it in work and on my own machines
- ch_123 8mo agoI ran Linux without swap for some years on a laptop with a large-for-the-time amount of RAM (about 8GB). It _mostly_ worked, but sudden spikes of memory usage would render the system unresponsive. Usually it would recover, but it in some cases it required a power cycle. Similarly, on a server where you might expect most of the physical memory to get used, it ends up being very important for stability. Think of VM or container hosts in particular.
- solstice 8mo agoI had a similar experience with Kubuntu on a xps13 from 2016 with only 8GB of RAM and the system suddenly freezing so hard that a hard reboot was required. While looking for the cause, I noticed that the system had only 250 MB of swap space. After increasing that to 10 GB there have been no further instances of freezing so far.
- GCUMstlyHarmls 8mo agoI dont get why anti-swap is so prevalent in Linux discussions. Like, what does it hurt to stick 8-16-32gb extra "oh fuck" space on your drive. Either you're going to never exhaust your system ram, so it doesn't matter, minimally exhaust it and swap in some peak load but at least nothing goes down, or exhaust it all and start having things get OOM'd which feels bad to me. Am I out of touch? Surely it's the children who are wrong.
- ChocolateGod 8mo agoI'd like to see Linux gain support for actual memory compression, without the need to go through zram, similar to macOS/Windows.
- homebrewer 8mo agozram has been "obsolete" for years, I don't know why people still reach for it. Linux supports proper memory compression in the form of zswap https://wiki.archlinux.org/title/Zswap https://wiki.archlinux.org/title/Zswap
- ChocolateGod 8mo agoBecause I'd rather compress ram when running low on memory rather than swapping to my disks. zram is also default on some distros (e.g. Fedora).
- homebrewer 8mo agoDid you read the link? Additional disk swap is optional, and if for some reason you would still like to have one, it's easy to disable writeback, using just the RAM. And even if one enables zswap and configures nothing else, compressing RAM and only swapping out to disk under extreme pressure is still the default behavior.
- kasabali 8mo ago> Additional disk swap is optional Where did you read that? My experience has been that not only in reality the pool size is limited by the amount of swap space you've allocated, it's also uncompressed again when offloaded there which is ridiculous when you think about it.
- RealStickman_ 8mo agoI didn't realize zswap also uses in-memory compression. It might be a combination of poor naming and zram being continuously popular.
- dist-epoch 8mo agoBoth Canonical and Microsoft recommend enabling swap file for Ubuntu cloud images, even if you allocate plenty of RAM to the VM. Any thoughts on that?
- man8alexd 8mo agohttps://chrisdown.name/2018/01/02/in-defence-of-swap.html https://chrisdown.name/2018/01/02/in-defence-of-swap.html
- AtlasBarfed 8mo agoI get we want a magical OS that reads minds and abstracts itself from users, but these days low mem situations are technical events for people trying to optimize resources use and balance uptime. So not being able to mark apps processes as kill me first and leave others like ssh bash up is a missing feature. Shouldn't the os have a bunch of auto hooks to invoke for machines under duress? Yes you might be able to do it in userspace but .. userspace will probably get unpredictability nuked in stress situations. The WMs should have a "freeze all apps desktop except this one shell window" mode. Didn't NeXT have that? Having some swap basically required has always seemed like a smell. A legacy from the 640k oops wasn't enough days. I can see emergency memory swap system as a feature, which it currently isn't...
- man8alexd 8mo ago> So not being able to mark apps processes as kill me first and leave others like ssh bash up is a missing feature. systemd OOMScoreAdjust has existed for at least a decade. /proc/*/oom_adj since Linux kernel 2.6.11 - two decades. cgroups are also two decades old. SSH by default has OOMScoreAdjust=-1000 and is entirely protected from OOM.
- IshKebab 8mo agoYeah because Linux's memory management is quite poor and running out of RAM without swap will often mean a hard reboot. Swap definitely helps a lot, even if it doesn't fully solve the problem. To be honest I don't know why it's such an issue on Linux. Mac and Windows don't have this issue at all. Windows presumably because it doesn't over-commit memory. I'm not sure why Mac is so much better than Linux at memory management. My eventual solution was to just buy a PC with a ton of RAM (128 GB). Haven't had any hard reboots due to OOM since then!
- chmaynard 8mo agoSecond installment: https://news.ycombinator.com/item?id=46901662 https://news.ycombinator.com/item?id=46901662
- vikkymelani 8mo ago[dead]
- man8alexd 8mo agoI have updated my collection of falsehoods people believe about Linux swap and OOM https://alexeydemidov.com/2025/05/15/falsehoods-people-and-LLMs-believe-about-Linux-swap-and-OOM/ https://alexeydemidov.com/2025/05/15/falsehoods-people-and-L...