4 ms·
One pet peeve I have with virtual memory management on Linux is that, as memory usage approaches 100%, the kernel starts evicting executable pages because techn
by FooBarWidget 8mo ago
One pet peeve I have with virtual memory management on Linux is that, as memory usage approaches 100%, the kernel starts evicting executable pages because technically they're read-only and can be loaded from disk. Thus, the entire system grinds to a halt in a behavior that looks like swapping, because every program that wants to execute instructions has to load its instructions from disk again, only to have those instruction pages be evicted again when context switching to another program. This behavior is especially counter intuitive because disabling swap does not prevent this problem. There are no convenient settings for administrators for preventing this problem.
It's good that we have better swapping now, but I wish they'd address the above. I'd rather have programs getting OOMKilled or throwing errors before the system grinds to a halt, where I can't even ssh in and run 'ps'.
- robinsonb5 8mo agoIndeed. I think what's really needed is some way to mark pages as "required for interactivity" so that nothing related to the user interface gets paged out, ever. That, I think, would go at least some way towards restoring the feeling of "having a computer's full attention" that we had thirty years ago.
- akdev1l 8mo agoSeems the applications can call mlockall() to do this
- direwolf20 8mo agoAn Electron app would mark its entire 2GB as required for interactivity. If you run 4 electron apps on an 8GB system you run out of memory.
- robinsonb5 8mo agoI don't mean interactivity within apps, per se - I mean the desktop and underlying OS, so that if an electron app goes unresponsive and eats all the free RAM the window manager can still kill it. Or you can still open a new terminal window, log in and kill it. Right now it can take several minutes to get a Linux system back under control once a swapstorm starts.
- M95D 8mo agoAlt+[SysRq,f] Or Alt+[SysRq,h] for help
- dingaling 8mo agoNo effect, captain. In 30 years of using desktop Linux I've never been able to interrupt a swapstorm. The only way out is long-press the power button.
- M95D 8mo agoIt always works for me, including on SBCs over TTL serial port. Always. Never had a situation where invoking the OOM killer by sysrq didn't solve the swap storm. On your system it probably doesn't work at all, not even when idle. It can only be 3 reasons: - Your kernel doesn't have it. You probably have a generic kernel provided by the package manager without this feature enabled. I can't really help you here. I always build my own kernel from source (+ patches). - Something hijacks your keyboard input. If you have a console already opened and logged in as root, you can "echo f > /proc/sysrq-trigger". Else, you can try setting up a permanent serial console and send the command from another computer. CTRL+2 then the command letter (f). The magic sysrq key over serial console is a separate kernel option that needs to be enabled. - You're doing it wrong. On laptop keyboards, keyboards less than 104/105 keys, sysrq is one of the first keys to be removed. Getting it pressed with Fn combinations... Good luck with that!
- SAI_Peregrinus 8mo agoLinux doesn't really have any distinction between the desktop and underlying OS components in userspace and anything else in userspace. Linux is quite userland-agnostic, and distros have traditionally mixed user software with distro-managed software. You shouldn't use `sudo` to install software by default, your package manager should allow installing software for just your user. Software installed for the system could then be the only software allowed to mark itself as required for interactivity. You could do that manually to other software if you had root access, but "normal" user software installs with the package manager couldn't do so since they wouldn't get root access. That'd require some new capabilities added, and some substantial shifts in how distro maintainers & users operate, so it's extremely unlikely. It's much closer to how things like Android operate, though still not quite as secure as giving each application its own user & dedicated storage for data.
- FooBarWidget 8mo agoThere is, mlock() or mlockall(), but it requires developer support. I wish there is an administrator knob that allows me to mark whole processes without needing to modify them.
- man8alexd 8mo agoThere is cgroup memory.min
- nolist_policy 8mo agoLinux swap has been fixed on Chromebooks for years thanks to MGLRU. It's upstream since Linux 6.1 and you can try it with echo y >/sys/kernel/mm/lru_gen/enabled
- tremon 8mo agoDocumentation links: https://docs.kernel.org/next/admin-guide/mm/multigen_lru.html https://docs.kernel.org/next/admin-guide/mm/multigen_lru.htm... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/mm/multigen_lru.rst https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- M95D 8mo agoI had nothing but problems since that was introduced in 6.1. It seems that the kernel prefers to compact/defrag memory, repeatedly, each time freezing everything 1-2 seconds, rather than releasing some disk cache memory or swapping out.
- man8alexd 8mo agoActively used executable pages are explicitly excluded from reclaim. And if they are not used, why should they stay in memory when the memory is constrained? It is not the first time I have heard complaints about executable pages, but it seems to be some kind of common misunderstanding. https://news.ycombinator.com/item?id=45369516 https://news.ycombinator.com/item?id=45369516
- FooBarWidget 8mo agoWhat is "actively used"? The bash session that I was using 2 seconds before system grinded to a halt sure didn't count.
- man8alexd 8mo agoYour bash session is most likely still in memory but the system is spending 99.9% of time waiting for swap I/O, trying to free a few pages of memory. Swap random access latency is 10^3 slower than RAM.
- FooBarWidget 8mo agoThe system has swap disabled.
- man8alexd 8mo agoThen see the link in my previous comment.
- fulafel 8mo agoThe set of possibly blocking operations for interactively using bash is big. Executable pages of the bash executable are far from the only thing that could be missing. If the machine is swap trashing, all i/o goes to the same congested queue. .bash_history read or write access, memory allocation, stuff your terminal program does, stuff your wayland compositor or X11 stack does, bash accessing data in its memory that has been swapped out, etc etc. And each of those could be waiting for a while to issue their IO request since the IO system is flooded by swap IO. There should be a tool that can show the interdependent graph of pending, blocking io operations.
- worldsavior 8mo agoProgram instructions size is small thus loading is fast, so no need to worry about that too much. I'd look on different things first.
- 112233 8mo agoIs there a way to make linux kernel schedule in a "batch friendly way"? Say I do "make -j" and get 200 gcc processes diong jobserver LTO link with 2GB RSS each. In my head, optimal way through such mess is get as many processes as can fit into RAM without swapping, run them to completion, and schedule additional processes as resources become available. A depth first, "infinite latency" mode. Any combination of cgroups, /proc flags and other forbidden knobs to get such behaviour?
- direwolf20 8mo agoIt's not possible for the kernel to predict the memory needs of a process unfortunately
- man8alexd 8mo agoIt is possible to measure process memory utilitsation and set appropriate cgroup limits.
- 112233 8mo agoBut how about not scheduling swapped out processes if there currently is no free ram for their current RSS? of course kernel cannot know that a new process will balloon to eat all RAM, but once it has done so, is there a way to let it run to completion without being swapped out to "improve responsivity"?
- man8alexd 8mo agoThere is no actual swapping in the modern kernels. Nowadays, it is paging, when the kernel pages out individual unused memory pages, not entire processes, so it keeps all non-blocked processes running, but only necessary memory pages in the memory.
- Neywiny 8mo ago"make -j" has OOMd me more than it's worth. If it's a big project I just put in how many threads I want. I do hear your point but that is a solved problem.
- Rygian 8mo agoI suffer from the same behavior, ever since I moved from Ubuntu to Debian. An interactive system that does not interact (terminal not reactive, can't ssh in, screen does not refresh) is broken. I don't understand why this is not a kernel bug. On my system, to add insult to injury, when the system does come back twenty minutes later, I get a "helpful" pop-up from the Linux Kernel saying "Memory Shortage Avoided". Which is just plain wrong. The pop-up should say "sorry, the kernel bricked your system for a solid twenty minutes for no good reason, please file a report".
- rustyhancock 8mo agoThis explains some of the issues I was having on a laptop some months back. And searching desperately for "just kill the damn thing" option.
- garaetjjte 8mo agoIf you enable Magic SysRq you can invoke OOM-killer from keyboard. https://docs.kernel.org/admin-guide/sysrq.html https://docs.kernel.org/admin-guide/sysrq.html
- AtlasBarfed 8mo agoIs it as bad with ssd?