5 ms·

For reference, [1] describes the SSD near-OOM thrash problem, where the speed of refaulting file pages is so high that the kernel detects enough activity and do

For reference, [1] describes the SSD near-OOM thrash problem, where the speed of refaulting file pages is so high that the kernel detects enough activity and doesn't trigger OOM. It's a heck of a problem to deal with, and a OOM killer will go part of the way to fixing it, but I still have two complaints about the thrash problem. 1. In similar situations on other operating systems, I don't usually see the GUI freeze to an unusable state. In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. I suppose it's possible that I just haven't used other OSes enough in a long time and they have this too, but this seems like a solvable problem. I assume it's not a CPU issue, since the scheduler should handle that just fine. So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? 2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it (even if the OOM killer blames something else). Now, I understand that in theory unused memory is wasted memory. So a browser using a ton of memory can be a good thing. But this is only true when memory the browser is holding but not using can be reclaimed to be used by another program, and this seems to basically never happen. If I close and reopen the browser (with the tabs automatically restored) the memory usage drops by 80% or more; the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall. I suspect solving these two issues would largely fix thrashing for most desktop users, without the need to ever kill anything, which is obviously undesirable. Feel free to tell me if one or both of these is incorrect or impossible. These are just my observations, I'm not a kernel dev or anything close to one.

marcodiego 6y ago
> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.
raffraffraff 6y ago
> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up. If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen: * Memory rises * Some pages get swapped out * Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes * Memory rises some more * You see more pages get swapped out * (...and see more disk writes) * Memory rises some more * You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process. * Disk queue lengths rise * One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up) This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk. But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit. So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out? NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.
- bscphil 6y ago
  Thanks for the comment, that's very interesting. I hadn't considered the possibility of the desktop environment needing to do disk reads, I was thinking everything was in memory already. I suppose the solution is to figure out how to reduce or eliminate the number of disk reads after the DE initially loads, or cache everything needed in memory with a "do not evict" flag on it. Especially for the TTY, that's basically a crucial system function and keeping everything needed for it always available in memory wouldn't take much space relative to a typical desktop system. I think at least some of the problem is the program code pages getting evicted to disk, so they have to be read back to memory too, which goes to the back of the disk queue. When I'm experiencing thrashing, even basic stuff like alt-tab to switch windows doesn't work. And I assume the window manager doesn't need to do disk reads to switch windows. Even the damn X cursor freezes!
pedrocr 6y ago
For 2) last time I looked I couldn't find a way to have a memory contract with the kernel that allowed for a given allocation to be evicted at will by the kernel while also telling the application that has happened. This would be ideal for apps that can grow their cache unbounded (e.g., photo processing apps with each photo, browsers with each tab) but can also regenerate that cache at will. You end up having tunables for how much cache to hold in RAM which is never correct as depending on what else the user is doing there may be a lot or very little free RAM.
- bombcar 6y ago
  Would it be possible to run the cache in a separate process/thread and then somehow mark that process/thread as sacrificial?
- the8472 6y ago
  You can use PSI for that. And it actually makes more sense when all applications listen on the global PSI indicators so that if one application would start thrashing another can free up memory cooperatively. But if needed it can also be done per cgroup.
- nitrogen 6y ago
  Something like madvise(MADV_FREE) sounds closer than MADV_DONTNEED -- the app can check for zero where it wrote nonzero -- but doesn't guarantee the freeing will be delayed until there is memory pressure.
drran 6y ago
For browser, reduce the number of parallel processes in preferences and try to use Auto Tab Discard plugin: https://www.youtube.com/watch?v=S0rHU38OnTE https://www.youtube.com/watch?v=S0rHU38OnTE
garaetjjte 6y ago
>In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. You can recover system from this state by manually triggering OOM killer using Magic SysRq keys. (Alt+SysRq+F, provided you have SysRq enabled in /proc/sys/kernel/sysrq)
- lolc 6y ago
  This is very useful! I have to look up this combination every other month when one of my systems starts thrashing.
- bscphil 6y ago
  Is there a version of this that works on typical laptop keyboards that don't have SysRq? I could map an unused key to it with xmodmap of course, but if my X server isn't responding I don't think that would help very much!
  - garaetjjte 6y ago
    SysRq is PrintScreen, I think it should be present even on laptops?
    - bscphil 6y ago
      Oh! News to me, I definitely have it. Thanks!
TacticalCoder 6y ago
> 2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it > Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall. I went the ballistic route: most of my browsers run inside local Docker containers, on which I put CPU and RAM quota. It's my machine: I decide how "citizens" behave : )
- scaladev 6y ago
  If you're using systemd, drop docker and start your browser through something like this: #!/bin/sh systemd-run \ --user \ --scope \ --property=MemoryHigh=4G \ --slice=background \ /usr/bin/firefox "$@" Theoretically this does not prevent Firefox from using more than 4 GBs of memory (you need MemoryMax for that), but practically it does. MemoryHigh=bytes Specify the throttling limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases. This is the main mechanism to control memory usage of a unit. MemoryMax=bytes Specify the absolute limit on memory usage of the executed processes in this unit. If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit. It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense. I use a similar script to prevent my torrent client from pushing more useful data out of the page cache. You can put resource limits on basically anything (see systemd.resource-control(5)).
- arp242 6y ago
  Docker is pretty overkill for this; you can just create a new cgroup with a maximum memory, CPU, etc.
- anthk 6y ago
  OpenBSD has login.conf https://man.openbsd.org/login.conf.5 https://man.openbsd.org/login.conf.5
  - 6y ago
notyourday 6y ago
Control groups are your friends. Run your browser in a separate control group with a constrained memory. It took me discovering memory leaks from qemu running X over virgl machines to actually spend a weekend figuring it out, but it is doable. * I run separate cgroups for admin SSH with (1G min, 4G max ) - worst case scenario I can ssh into the desktop and kill offending processes. * Separate cgroup for two browsers ( 8G each max ). * Separate cgroup per qemu VM ( VM max ram + 512M each max ). After implementing that I went from periodic lock ups to a quick browser crash and a restart maybe once a week.
The_rationalist 6y ago
Regarding the file pages this is the solution https://github.com/hakavlad/le9-patch https://github.com/hakavlad/le9-patch
cipherboy 6y ago
How much memory is enough for 1? Back on Fedora 32, on a system with close to a year of uptime (it was attached to my TV), gnome-shell consumed nearly 20GB of RAM on a 32GB system! And I rarely interacted with it (outside of watching the occasional concert on it). Most of the time was spent on the lockscreen, with the TV on a different input. Perhaps this falls outside the category of "any memory condition"... :) Though, it was responsive. :D Needless to say, I think 1 is harder than expected. Fedora has made some recent changes on the OOM killer front though, perhaps those are closer to what you're interested in.
awwaiid 6y ago
Yesterday I ran into this nice article on using cgroups to contain firefox memory usage, https://dev.to/msugakov/taking-firefox-memory-usage-under-control-on-linux-4b02 https://dev.to/msugakov/taking-firefox-memory-usage-under-co...
SAI_Peregrinus 6y ago
> the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall. The "Auto Tab Discard" extension[1] for Firefox unloads tab contents, keeping only the title and favicon. Tabs can be put in an allowlist to prevent discarding. I don't know if Chrome or Safari have something similar. [1] https://addons.mozilla.org/en-US/firefox/addon/auto-tab-discard/ https://addons.mozilla.org/en-US/firefox/addon/auto-tab-disc...

marcodiego 6y ago

> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.

raffraffraff 6y ago

> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up. If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen: * Memory rises * Some pages get swapped out * Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes * Memory rises some more * You see more pages get swapped out * (...and see more disk writes) * Memory rises some more * You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process. * Disk queue lengths rise * One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up) This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk. But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit. So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out? NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.

bscphil 6y ago
Thanks for the comment, that's very interesting. I hadn't considered the possibility of the desktop environment needing to do disk reads, I was thinking everything was in memory already. I suppose the solution is to figure out how to reduce or eliminate the number of disk reads after the DE initially loads, or cache everything needed in memory with a "do not evict" flag on it. Especially for the TTY, that's basically a crucial system function and keeping everything needed for it always available in memory wouldn't take much space relative to a typical desktop system. I think at least some of the problem is the program code pages getting evicted to disk, so they have to be read back to memory too, which goes to the back of the disk queue. When I'm experiencing thrashing, even basic stuff like alt-tab to switch windows doesn't work. And I assume the window manager doesn't need to do disk reads to switch windows. Even the damn X cursor freezes!

pedrocr 6y ago

For 2) last time I looked I couldn't find a way to have a memory contract with the kernel that allowed for a given allocation to be evicted at will by the kernel while also telling the application that has happened. This would be ideal for apps that can grow their cache unbounded (e.g., photo processing apps with each photo, browsers with each tab) but can also regenerate that cache at will. You end up having tunables for how much cache to hold in RAM which is never correct as depending on what else the user is doing there may be a lot or very little free RAM.

bombcar 6y ago
Would it be possible to run the cache in a separate process/thread and then somehow mark that process/thread as sacrificial?
the8472 6y ago
You can use PSI for that. And it actually makes more sense when all applications listen on the global PSI indicators so that if one application would start thrashing another can free up memory cooperatively. But if needed it can also be done per cgroup.
nitrogen 6y ago
Something like madvise(MADV_FREE) sounds closer than MADV_DONTNEED -- the app can check for zero where it wrote nonzero -- but doesn't guarantee the freeing will be delayed until there is memory pressure.

drran 6y ago

For browser, reduce the number of parallel processes in preferences and try to use Auto Tab Discard plugin: https://www.youtube.com/watch?v=S0rHU38OnTE https://www.youtube.com/watch?v=S0rHU38OnTE

garaetjjte 6y ago

>In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. You can recover system from this state by manually triggering OOM killer using Magic SysRq keys. (Alt+SysRq+F, provided you have SysRq enabled in /proc/sys/kernel/sysrq)

lolc 6y ago
This is very useful! I have to look up this combination every other month when one of my systems starts thrashing.
bscphil 6y ago
Is there a version of this that works on typical laptop keyboards that don't have SysRq? I could map an unused key to it with xmodmap of course, but if my X server isn't responding I don't think that would help very much!
- garaetjjte 6y ago
  SysRq is PrintScreen, I think it should be present even on laptops?
  - bscphil 6y ago
    Oh! News to me, I definitely have it. Thanks!

TacticalCoder 6y ago

> 2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it > Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall. I went the ballistic route: most of my browsers run inside local Docker containers, on which I put CPU and RAM quota. It's my machine: I decide how "citizens" behave : )

scaladev 6y ago
If you're using systemd, drop docker and start your browser through something like this: #!/bin/sh systemd-run \ --user \ --scope \ --property=MemoryHigh=4G \ --slice=background \ /usr/bin/firefox "$@" Theoretically this does not prevent Firefox from using more than 4 GBs of memory (you need MemoryMax for that), but practically it does. MemoryHigh=bytes Specify the throttling limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases. This is the main mechanism to control memory usage of a unit. MemoryMax=bytes Specify the absolute limit on memory usage of the executed processes in this unit. If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit. It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense. I use a similar script to prevent my torrent client from pushing more useful data out of the page cache. You can put resource limits on basically anything (see systemd.resource-control(5)).
arp242 6y ago
Docker is pretty overkill for this; you can just create a new cgroup with a maximum memory, CPU, etc.
anthk 6y ago
OpenBSD has login.conf https://man.openbsd.org/login.conf.5 https://man.openbsd.org/login.conf.5
- 6y ago

notyourday 6y ago

Control groups are your friends. Run your browser in a separate control group with a constrained memory. It took me discovering memory leaks from qemu running X over virgl machines to actually spend a weekend figuring it out, but it is doable. * I run separate cgroups for admin SSH with (1G min, 4G max ) - worst case scenario I can ssh into the desktop and kill offending processes. * Separate cgroup for two browsers ( 8G each max ). * Separate cgroup per qemu VM ( VM max ram + 512M each max ). After implementing that I went from periodic lock ups to a quick browser crash and a restart maybe once a week.

The_rationalist 6y ago

Regarding the file pages this is the solution https://github.com/hakavlad/le9-patch https://github.com/hakavlad/le9-patch

cipherboy 6y ago

How much memory is enough for 1? Back on Fedora 32, on a system with close to a year of uptime (it was attached to my TV), gnome-shell consumed nearly 20GB of RAM on a 32GB system! And I rarely interacted with it (outside of watching the occasional concert on it). Most of the time was spent on the lockscreen, with the TV on a different input. Perhaps this falls outside the category of "any memory condition"... :) Though, it was responsive. :D Needless to say, I think 1 is harder than expected. Fedora has made some recent changes on the OOM killer front though, perhaps those are closer to what you're interested in.

awwaiid 6y ago

Yesterday I ran into this nice article on using cgroups to contain firefox memory usage, https://dev.to/msugakov/taking-firefox-memory-usage-under-control-on-linux-4b02 https://dev.to/msugakov/taking-firefox-memory-usage-under-co...

SAI_Peregrinus 6y ago

> the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall. The "Auto Tab Discard" extension[1] for Firefox unloads tab contents, keeping only the title and favicon. Tabs can be put in an allowlist to prevent discarding. I don't know if Chrome or Safari have something similar. [1] https://addons.mozilla.org/en-US/firefox/addon/auto-tab-discard/ https://addons.mozilla.org/en-US/firefox/addon/auto-tab-disc...