8 ms·

Linux oom behavior always has had complaints[0]. Hope it now finally fixes the desktop case[1]. [0] https://lwn.net/Articles/104185/ https://lwn.net/Articles/1

Linux oom behavior always has had complaints[0]. Hope it now finally fixes the desktop case[1]. [0] https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/ [1] https://lkml.org/lkml/2019/8/4/15 https://lkml.org/lkml/2019/8/4/15

warmwaffles 6y ago
[0] was a humorous read. I love these little analogies.
- bscphil 6y ago
  It's kind of a funny analogy in the context of systemd-oomd, because the latter is sort of like saying "hmm, when the plane is out of fuel, sometimes there's not enough power to actually eject anyone from the plane, so we're going to start ejecting people while there's still a little bit of fuel left!"
bscphil 6y ago
For reference, [1] describes the SSD near-OOM thrash problem, where the speed of refaulting file pages is so high that the kernel detects enough activity and doesn't trigger OOM. It's a heck of a problem to deal with, and a OOM killer will go part of the way to fixing it, but I still have two complaints about the thrash problem. 1. In similar situations on other operating systems, I don't usually see the GUI freeze to an unusable state. In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. I suppose it's possible that I just haven't used other OSes enough in a long time and they have this too, but this seems like a solvable problem. I assume it's not a CPU issue, since the scheduler should handle that just fine. So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? 2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it (even if the OOM killer blames something else). Now, I understand that in theory unused memory is wasted memory. So a browser using a ton of memory can be a good thing. But this is only true when memory the browser is holding but not using can be reclaimed to be used by another program, and this seems to basically never happen. If I close and reopen the browser (with the tabs automatically restored) the memory usage drops by 80% or more; the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall. I suspect solving these two issues would largely fix thrashing for most desktop users, without the need to ever kill anything, which is obviously undesirable. Feel free to tell me if one or both of these is incorrect or impossible. These are just my observations, I'm not a kernel dev or anything close to one.
- marcodiego 6y ago
  > So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.
- raffraffraff 6y ago
  > So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up. If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen: * Memory rises * Some pages get swapped out * Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes * Memory rises some more * You see more pages get swapped out * (...and see more disk writes) * Memory rises some more * You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process. * Disk queue lengths rise * One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up) This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk. But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit. So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out? NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.
alerighi 6y ago
In that regard Linux is much worse than Windows. In Windows you basically never go out of memory in a way to lock the system, sice the OS reserves some memory for its functions. It will block applications, but the operating system remain responsive. In Linux I was never been able to generate a similar behaviour. No matter how aggressive I set the swappiness, the system didn't began to swap until it was practically locked up with the mouse pointer lagging. Windows reserves some memory for its functions, for example the UI (maybe the fact that the GUI is in the kernel helps), and thus doesn't have these kind of problem, especially with an SSD (and I know that page file on SSDs is not ideal... but had a disk for 5 years swapping on it and never had a problem).
- minitech 6y ago
  What do you mean “didn’t begin to swap”? It should always be using swap. It starts locking up when you run out of both swap and RAM (because it’s run out of things to swap out that aren’t critical to what’s currently running). What it fails to do in time is outright kill processes.
- barrkel 6y ago
  I have some strong memories of paging death on Windows XP, on spinning rust. I recall taking some minutes to recover: Ctrl-Alt-Del and wait for secure authentication screen; hit shortcut for Task Manager and wait for it to start and come to the foreground; navigate to offending process and kill it, with blind keystrokes. The mouse cursor did move smoothly throughout, though, as I recall.

warmwaffles 6y ago

[0] was a humorous read. I love these little analogies.

bscphil 6y ago
It's kind of a funny analogy in the context of systemd-oomd, because the latter is sort of like saying "hmm, when the plane is out of fuel, sometimes there's not enough power to actually eject anyone from the plane, so we're going to start ejecting people while there's still a little bit of fuel left!"

bscphil 6y ago

For reference, [1] describes the SSD near-OOM thrash problem, where the speed of refaulting file pages is so high that the kernel detects enough activity and doesn't trigger OOM. It's a heck of a problem to deal with, and a OOM killer will go part of the way to fixing it, but I still have two complaints about the thrash problem. 1. In similar situations on other operating systems, I don't usually see the GUI freeze to an unusable state. In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. I suppose it's possible that I just haven't used other OSes enough in a long time and they have this too, but this seems like a solvable problem. I assume it's not a CPU issue, since the scheduler should handle that just fine. So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? 2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it (even if the OOM killer blames something else). Now, I understand that in theory unused memory is wasted memory. So a browser using a ton of memory can be a good thing. But this is only true when memory the browser is holding but not using can be reclaimed to be used by another program, and this seems to basically never happen. If I close and reopen the browser (with the tabs automatically restored) the memory usage drops by 80% or more; the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall. I suspect solving these two issues would largely fix thrashing for most desktop users, without the need to ever kill anything, which is obviously undesirable. Feel free to tell me if one or both of these is incorrect or impossible. These are just my observations, I'm not a kernel dev or anything close to one.

marcodiego 6y ago
> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.
raffraffraff 6y ago
> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition? Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up. If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen: * Memory rises * Some pages get swapped out * Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes * Memory rises some more * You see more pages get swapped out * (...and see more disk writes) * Memory rises some more * You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process. * Disk queue lengths rise * One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up) This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk. But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit. So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out? NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.

alerighi 6y ago

In that regard Linux is much worse than Windows. In Windows you basically never go out of memory in a way to lock the system, sice the OS reserves some memory for its functions. It will block applications, but the operating system remain responsive. In Linux I was never been able to generate a similar behaviour. No matter how aggressive I set the swappiness, the system didn't began to swap until it was practically locked up with the mouse pointer lagging. Windows reserves some memory for its functions, for example the UI (maybe the fact that the GUI is in the kernel helps), and thus doesn't have these kind of problem, especially with an SSD (and I know that page file on SSDs is not ideal... but had a disk for 5 years swapping on it and never had a problem).

minitech 6y ago
What do you mean “didn’t begin to swap”? It should always be using swap. It starts locking up when you run out of both swap and RAM (because it’s run out of things to swap out that aren’t critical to what’s currently running). What it fails to do in time is outright kill processes.
barrkel 6y ago
I have some strong memories of paging death on Windows XP, on spinning rust. I recall taking some minutes to recover: Ctrl-Alt-Del and wait for secure authentication screen; hit shortcut for Task Manager and wait for it to start and come to the foreground; navigate to offending process and kill it, with blind keystrokes. The mouse cursor did move smoothly throughout, though, as I recall.