4 ms·

Modernizing swapping: virtual swap spaces

58 points by voxadam 7mo ago

Numerlor 7mo ago
The swap/memory situation in linux has surprised me quite a bit coming from Windows. Windows remains mostly fully responsive even when memory is being pushed to the limits and swapping gigabytes per second, while on linux when I ran a stress test that ate all the memory I had trouble even terminating the script
- jauntywundrkind 7mo ago
  Like Linux / open source often, it depends on what you do with it! The kernel is very slow to kill stuff. Very very very very slow. It will try and try and try to prevent having to kill anything. It will be absolutely certain it can reclaim nothing more, and it will be at an absolute crawl trying to make every little kilobyte it can free, swapping like mad to try options to free stuff. But there are a number of daemons you can use if you want to be more proactive! Systemd now has systemd-oomd. It's pretty good! There's others, with other strategies for what to kill first, based on other indicators! The flexibility is a feature, not a bug. What distro are you on? I'm kind of surprised it didn't ship with something on?
- 01HNNWZ0MV43FF 7mo ago
  I've had that same experience. On new systems I install earlyoom. I'd rather have one app die than the whole system. You'd think after 30 years of GUIs and multi-tasking, we'd have this figured out, but then again we don't even have a good GUI framework.
  - LtWorf 7mo ago
    I used to use it but it's too aggressive. It kills stuff too quickly.
- dlcarrier 7mo ago
  There's two things that cause this. First, Windows has a variable swap file size, whereas Linux has a fixed size, so Windows can just fill up your drive, instead of running out of swap space. Second, the default behavior for the out-of-memory killer in Linux isn't very aggressive, with the default behavior being to over-commit memory instead of killing processes. As far as I know, Linux still doesn't support a variable-sized swap file, but it is possible to change how aggressively it over-commits memory or kills processes to free memory. As to why there differences are there, they're more historical than technical. My best guess is that Windows figured it out sooner, because it has always existed in an environment where multiple programs are memory hogs, whereas it wasn't common in Linux until the proliferation of web-based everything requiring hundreds of megabytes to gigabytes of memory for each process running in a Chrome tab or Electron instance, even if it's something as simple as a news article or chat client. Check out this series of blog posts. for more information on Linux memory management: https://dev.to/fritshooglandyugabyte/series/16577 https://dev.to/fritshooglandyugabyte/series/16577
  - LargoLasskhyfv 7mo ago
    > As far as I know, Linux still doesn't support a variable-sized swap file... You can add (and remove) additional swapfiles during runtime, or rather on demand. I'm just unaware of any mechanism doing that automagically, though. Could probably done in eBPF and some shell scripts, I guess?
    - dsr_ 7mo ago
      swapspace (https://github.com/Tookmund/Swapspace https://github.com/Tookmund/Swapspace) does this. Available in Debian stable.
      - LargoLasskhyfv 7mo ago
        Wow. Since 20 years. And I'm rambling about eBPF...
        7mo ago
- rwmj 7mo ago
  The annoying thing I've found with Linux under memory stress (and still haven't found a nice way to solve) is I want it to always always always kill firefox first. Instead it tends to either kill nothing (causing the system to hang) or kill some vital service.
  - pmontra 7mo ago
    I'm not sure that I'd want the OS to kill my browser while I'm working within it. Of course the browser is the largest process in my system, so when I notice that memory is running low I restart it and I gain some 15 GB. Basically I am the memory manager of my system and I've been able to run my 32 GB Linux laptop with no swap since 2014. I read that a system with no swap is suboptimal but the only tradeoff I notice is that manual OOM vs less writes on my SSD. I'm happy with it.
    - robinsonb5 7mo ago
      There are two pillars to managing RAM with virtual memory: the obvious one is is writing one program's working set to disk, so that another program can use that memory. The other one - which isn't prevented by disabling swap - is flushing parts of a program which were loaded from disk, and reloading them from disk when next needed. That second pillar is actually worse for interactivity than swapping the working set, which is why disabling swap entirely isn't considered optimal. By far the best approach is just to have an absurd amount of RAM - which of course is a much less accessible option now than it was a year ago.
  - jauntywundrkind 7mo ago
    If using systemd-oomd, you can launch Firefox into it's own cgroup / systemd.scope, that has memory pressure control settings set to not kill it. ManagedOOMPreference=avoid. https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#Memory%20Pressure%20Control https://www.freedesktop.org/software/systemd/man/latest/syst... There's a variety of oom daemons. bustd is very lightweight & new. earlyoom has been around a long time, and has an --avoid flag. https://github.com/rfjakob/earlyoom?tab=readme-ov-file#preferred-processes https://github.com/rfjakob/earlyoom?tab=readme-ov-file#prefe... Your concerns are very addressable.
- Onavo 7mo ago
  > Windows remains mostly fully responsive even when memory is being pushed to the limits and swapping gigabytes per second In my experience this is only on later versions of the NT Kernel and only on NVME (mostly the latter I think).
  - robinsonb5 7mo ago
    Yeah I think SSD / NVME makes all the difference here - I certainly remember XP / Vista / Win 7 boxes that became unusable and more-or-less unrecoverable (just like Linux) once a swap storm starts.
  - p_ing 7mo ago
    NT4 exhibits the same behavior under extreme load.
- nasretdinov 7mo ago
  In Linux the default swap behaviour is to also swap out the memory mapped to the executable file, not just memory allocated by the process. This is a relatively sane approach on servers, but not so much on desktops. I believe both Windows and macOS don't swap out code pages, so the applications remain responsive, at the of (potentially) lower swap efficiency
  - superjan 7mo ago
    Don’t know about Macs, but on Windows executable code is treated like a readonly memory mapped file that can be loaded and restored as the kernel sees fit. It could also be shared between processes, though that is not happening that much anymore due to ASLR.
  - inkyoto 7mo ago
    > In Linux the default swap behaviour is to also swap out the memory mapped to the executable file, not just memory allocated by the process […] I believe both Windows and macOS don't swap out code pages, so the applications remain responsive, at the of (potentially) lower swap efficiency Linux does not page out code pages into the swap. You might be conflating page reclamation with swapping instead. In Linux, executable «.text» pages are mapped[0] as file-backed pages, not anonymous memory, so when the kernel needs to reclaim RAM it normally drops those pages and reloads them from the executable file on the next page fault once they are accessed again (i.e. on demand) rather than writing them to swap. In this particular regard, Linux is no different from any other modern UNIX[1] kernel (*BSD, Solaris, AIX and may others). [0] Via mmap(2) in argv[0], essentially. [1] Modern UNIX is mid-1990's and onwards.
    - nasretdinov 7mo ago
      Yes, you are correct, I wasn't precise enough. It doesn't make sense to swap the existing code pages, they are just unmapped. (And that's the reason why you get "text file busy" when doing scp over the file: since the OS relies on the fact that the .text pages can be safely unmapped it needs to guarantee that they stay read-only)
  - man8alexd 7mo ago
- demosito666 7mo ago
  Oh yeah. Bug 12309 was reported now what, 20 years ago? It’s fair to say that at this point arrival of GNU Mach will happen sooner than Linux will be able to properly work under memory pressure.
- TacticalCoder 7mo ago
  > Windows remains mostly fully responsive even when memory is being pushed to the limits and swapping gigabytes per second ... The problem problem though is all the times when Windows is totally unusable even though it's doing exactly jack shit. An example would be when it's doing all its pointless updates/upgrades. I don't know what people are smoking in this world when they somehow believe that Windows 11 is an acceptable user experience and a better OS than Linux. Somehow though it's Linux that's powering billions if not tens of billions of devices worldwide and only about 12% of all new devices sold worldwide are running that piece of turd that Windows is.
robinsonb5 7mo ago
OOM killers serve a purpose but - for a desktop OS - they're missing the point. In a sane world the user would decide which application to shut down, not the OS; the user would click the appropriate application's close gadget, and the user interface would remain responsive enough for that to happen in a matter of seconds rather than minutes. I understand the many reasons why that's not possible, but it's a huge failing of Linux as a desktop OS, and OOM killers are picking around the edges of the problem, not addressing it head-on. (Which isn't to say, of course, that OOM killers aren't the right approach in a server context.)
- nasretdinov 7mo ago
  I don't even think it's not possible — surely there already exist solutions similar to BFS scheduler that would improve interactive performance
  - nasretdinov 7mo ago
    Potentially https://github.com/hakavlad/prelockd https://github.com/hakavlad/prelockd ?
- olejorgenb 7mo ago
  > I understand the many reasons why that's not possible Isn't the only reason that the UI effectively freeze under high memory pressure? If the processing involved in the core parts of the UI could be prioritized there's no reason for it not to work? I don't understand why we can't have (?) a way of saying "the pages of these few processes are holy"? (yes, it would still be slow allocating new pages, but it should be possible to write a small core part such that it doesn't need to constantly allocate memory?)
  - PhilipRoman 7mo ago
    mlockall(2) + tty/framebuffer based graphics should do the trick
- p_ing 7mo ago
anthk 7mo ago
OpenBSD and the rest have a limits file where you can set RAM limits per user and sometimes per process, so is not a big issue. On GNU/Linux and the rest not supporting dynamic swap files, you can swap into anything ensembling a file, even into virtual disk images. Also set up ZRAM as soon as possible. 1/3 of the physical RAM for ZRAM it's perfect, it will almost double your effective RAM size with ease.
- JamesTRexx 7mo ago
  I think zswap is the better option because it's not a fixed RAM storage, it merely compresses pages in RAM up to a variable limit and then writes to swap space when needed, which is more efficient. It worked very well with my preceding laptop limited to 4GB of RAM.
- gzread 7mo ago
  You can do this with cgroups but you aren't allowed to use cgroups if you use systemd, because it messes up systemd.
  - esseph 7mo ago
    systemd is the primary user of cgroups. Your comment doesn't make sense.
- man8alexd 7mo ago
  systemd allows setting cgroup memory limits.
ChocolateGod 7mo ago
Is there a reason as to why Linux has not adopted straight up page compression like Windows and macOS?
- GrayShade 7mo ago
  It did, there's two incompatible approaches, zram and zswap.
- creatonez 7mo ago
  It has? Many distros have it enabled by default, too. The article discusses it quite a bit.
hackernudes 7mo ago
I see some comments about soft lockups during memory pressure. I have struggled with this immensely over the years. I wrote a userspace memory reclaimer daemon and have not had a lockup since: https://gist.github.com/EBADBEEF/f168458028f684a91148f4d3e791ba84 https://gist.github.com/EBADBEEF/f168458028f684a91148f4d3e79... . The hangs usually happened when I was stressing VFS (the computer was a samba server) along with other workloads. To trigger a hang manually I would read in large files (bigger than available ram) in parallel while running a game. I could get it to hang even with 128GB ram. I tweaked all the vfs settings (swappiness, etc...) to no avail. I tried with and without swap. In the end it looked like memory was not getting reclaimed fast enough, like linux would wait too long to start reclaiming memory and some critical process would get stuck waiting for some memory. The system would hang for minutes or hours at a time only making the tiniest of progress between reclaims. If I caught the problem early enough (just as everything started stuttering) I could trigger a reclaim manually by writing to '/sys/fs/cgroup/memory.reclaim' and the system would recover. I wonder if it was specific to btrfs or some specific workload pattern but I was never able to figure it out.
- NooneAtAll3 7mo ago
  I want to express my gratitude for not falling for greedy marketers' lies and using correct base-2 for KB/MB/GB
- man8alexd 7mo ago
  Just tune the kernel watermarks - vm.min_free_kbytes and vm.watermark_scale_factor
  - hackernudes 7mo ago
    I do wish I had documented what I tried better! There might be a magic combo that could have helped but I tried tweaking a lot of the vm settings. One day I will probably see if I can still reproduce the original problem and be more methodical about it. More likely on list of things I might not ever get around to.
glyco 7mo ago
The new design is still wrong though - the swap system is a cache, there's no reason why a page should only exist on one level.

58 points by voxadam 7mo ago

The swap/memory situation in linux has surprised me quite a bit coming from Windows. Windows remains mostly fully responsive even when memory is being pushed to the limits and swapping gigabytes per second, while on linux when I ran a stress test that ate all the memory I had trouble even terminating the script

jauntywundrkind 7mo ago
Like Linux / open source often, it depends on what you do with it! The kernel is very slow to kill stuff. Very very very very slow. It will try and try and try to prevent having to kill anything. It will be absolutely certain it can reclaim nothing more, and it will be at an absolute crawl trying to make every little kilobyte it can free, swapping like mad to try options to free stuff. But there are a number of daemons you can use if you want to be more proactive! Systemd now has systemd-oomd. It's pretty good! There's others, with other strategies for what to kill first, based on other indicators! The flexibility is a feature, not a bug. What distro are you on? I'm kind of surprised it didn't ship with something on?
01HNNWZ0MV43FF 7mo ago
I've had that same experience. On new systems I install earlyoom. I'd rather have one app die than the whole system. You'd think after 30 years of GUIs and multi-tasking, we'd have this figured out, but then again we don't even have a good GUI framework.
- LtWorf 7mo ago
  I used to use it but it's too aggressive. It kills stuff too quickly.
dlcarrier 7mo ago
There's two things that cause this. First, Windows has a variable swap file size, whereas Linux has a fixed size, so Windows can just fill up your drive, instead of running out of swap space. Second, the default behavior for the out-of-memory killer in Linux isn't very aggressive, with the default behavior being to over-commit memory instead of killing processes. As far as I know, Linux still doesn't support a variable-sized swap file, but it is possible to change how aggressively it over-commits memory or kills processes to free memory. As to why there differences are there, they're more historical than technical. My best guess is that Windows figured it out sooner, because it has always existed in an environment where multiple programs are memory hogs, whereas it wasn't common in Linux until the proliferation of web-based everything requiring hundreds of megabytes to gigabytes of memory for each process running in a Chrome tab or Electron instance, even if it's something as simple as a news article or chat client. Check out this series of blog posts. for more information on Linux memory management: https://dev.to/fritshooglandyugabyte/series/16577 https://dev.to/fritshooglandyugabyte/series/16577
- LargoLasskhyfv 7mo ago
  > As far as I know, Linux still doesn't support a variable-sized swap file... You can add (and remove) additional swapfiles during runtime, or rather on demand. I'm just unaware of any mechanism doing that automagically, though. Could probably done in eBPF and some shell scripts, I guess?
  - dsr_ 7mo ago
    swapspace (https://github.com/Tookmund/Swapspace https://github.com/Tookmund/Swapspace) does this. Available in Debian stable.
    - LargoLasskhyfv 7mo ago
      Wow. Since 20 years. And I'm rambling about eBPF...
      - 7mo ago
rwmj 7mo ago
The annoying thing I've found with Linux under memory stress (and still haven't found a nice way to solve) is I want it to always always always kill firefox first. Instead it tends to either kill nothing (causing the system to hang) or kill some vital service.
- pmontra 7mo ago
  I'm not sure that I'd want the OS to kill my browser while I'm working within it. Of course the browser is the largest process in my system, so when I notice that memory is running low I restart it and I gain some 15 GB. Basically I am the memory manager of my system and I've been able to run my 32 GB Linux laptop with no swap since 2014. I read that a system with no swap is suboptimal but the only tradeoff I notice is that manual OOM vs less writes on my SSD. I'm happy with it.
  - robinsonb5 7mo ago
    There are two pillars to managing RAM with virtual memory: the obvious one is is writing one program's working set to disk, so that another program can use that memory. The other one - which isn't prevented by disabling swap - is flushing parts of a program which were loaded from disk, and reloading them from disk when next needed. That second pillar is actually worse for interactivity than swapping the working set, which is why disabling swap entirely isn't considered optimal. By far the best approach is just to have an absurd amount of RAM - which of course is a much less accessible option now than it was a year ago.
- jauntywundrkind 7mo ago
  If using systemd-oomd, you can launch Firefox into it's own cgroup / systemd.scope, that has memory pressure control settings set to not kill it. ManagedOOMPreference=avoid. https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#Memory%20Pressure%20Control https://www.freedesktop.org/software/systemd/man/latest/syst... There's a variety of oom daemons. bustd is very lightweight & new. earlyoom has been around a long time, and has an --avoid flag. https://github.com/rfjakob/earlyoom?tab=readme-ov-file#preferred-processes https://github.com/rfjakob/earlyoom?tab=readme-ov-file#prefe... Your concerns are very addressable.
Onavo 7mo ago
> Windows remains mostly fully responsive even when memory is being pushed to the limits and swapping gigabytes per second In my experience this is only on later versions of the NT Kernel and only on NVME (mostly the latter I think).
- robinsonb5 7mo ago
  Yeah I think SSD / NVME makes all the difference here - I certainly remember XP / Vista / Win 7 boxes that became unusable and more-or-less unrecoverable (just like Linux) once a swap storm starts.
- p_ing 7mo ago
  NT4 exhibits the same behavior under extreme load.
nasretdinov 7mo ago
In Linux the default swap behaviour is to also swap out the memory mapped to the executable file, not just memory allocated by the process. This is a relatively sane approach on servers, but not so much on desktops. I believe both Windows and macOS don't swap out code pages, so the applications remain responsive, at the of (potentially) lower swap efficiency
- superjan 7mo ago
  Don’t know about Macs, but on Windows executable code is treated like a readonly memory mapped file that can be loaded and restored as the kernel sees fit. It could also be shared between processes, though that is not happening that much anymore due to ASLR.
- inkyoto 7mo ago
  > In Linux the default swap behaviour is to also swap out the memory mapped to the executable file, not just memory allocated by the process […] I believe both Windows and macOS don't swap out code pages, so the applications remain responsive, at the of (potentially) lower swap efficiency Linux does not page out code pages into the swap. You might be conflating page reclamation with swapping instead. In Linux, executable «.text» pages are mapped[0] as file-backed pages, not anonymous memory, so when the kernel needs to reclaim RAM it normally drops those pages and reloads them from the executable file on the next page fault once they are accessed again (i.e. on demand) rather than writing them to swap. In this particular regard, Linux is no different from any other modern UNIX[1] kernel (*BSD, Solaris, AIX and may others). [0] Via mmap(2) in argv[0], essentially. [1] Modern UNIX is mid-1990's and onwards.
  - nasretdinov 7mo ago
    Yes, you are correct, I wasn't precise enough. It doesn't make sense to swap the existing code pages, they are just unmapped. (And that's the reason why you get "text file busy" when doing scp over the file: since the OS relies on the fact that the .text pages can be safely unmapped it needs to guarantee that they stay read-only)
- man8alexd 7mo ago
demosito666 7mo ago
Oh yeah. Bug 12309 was reported now what, 20 years ago? It’s fair to say that at this point arrival of GNU Mach will happen sooner than Linux will be able to properly work under memory pressure.
TacticalCoder 7mo ago
> Windows remains mostly fully responsive even when memory is being pushed to the limits and swapping gigabytes per second ... The problem problem though is all the times when Windows is totally unusable even though it's doing exactly jack shit. An example would be when it's doing all its pointless updates/upgrades. I don't know what people are smoking in this world when they somehow believe that Windows 11 is an acceptable user experience and a better OS than Linux. Somehow though it's Linux that's powering billions if not tens of billions of devices worldwide and only about 12% of all new devices sold worldwide are running that piece of turd that Windows is.

robinsonb5 7mo ago

OOM killers serve a purpose but - for a desktop OS - they're missing the point. In a sane world the user would decide which application to shut down, not the OS; the user would click the appropriate application's close gadget, and the user interface would remain responsive enough for that to happen in a matter of seconds rather than minutes. I understand the many reasons why that's not possible, but it's a huge failing of Linux as a desktop OS, and OOM killers are picking around the edges of the problem, not addressing it head-on. (Which isn't to say, of course, that OOM killers aren't the right approach in a server context.)

nasretdinov 7mo ago
I don't even think it's not possible — surely there already exist solutions similar to BFS scheduler that would improve interactive performance
- nasretdinov 7mo ago
  Potentially https://github.com/hakavlad/prelockd https://github.com/hakavlad/prelockd ?
olejorgenb 7mo ago
> I understand the many reasons why that's not possible Isn't the only reason that the UI effectively freeze under high memory pressure? If the processing involved in the core parts of the UI could be prioritized there's no reason for it not to work? I don't understand why we can't have (?) a way of saying "the pages of these few processes are holy"? (yes, it would still be slow allocating new pages, but it should be possible to write a small core part such that it doesn't need to constantly allocate memory?)
- PhilipRoman 7mo ago
  mlockall(2) + tty/framebuffer based graphics should do the trick
p_ing 7mo ago

anthk 7mo ago

OpenBSD and the rest have a limits file where you can set RAM limits per user and sometimes per process, so is not a big issue. On GNU/Linux and the rest not supporting dynamic swap files, you can swap into anything ensembling a file, even into virtual disk images. Also set up ZRAM as soon as possible. 1/3 of the physical RAM for ZRAM it's perfect, it will almost double your effective RAM size with ease.

JamesTRexx 7mo ago
I think zswap is the better option because it's not a fixed RAM storage, it merely compresses pages in RAM up to a variable limit and then writes to swap space when needed, which is more efficient. It worked very well with my preceding laptop limited to 4GB of RAM.
gzread 7mo ago
You can do this with cgroups but you aren't allowed to use cgroups if you use systemd, because it messes up systemd.
- esseph 7mo ago
  systemd is the primary user of cgroups. Your comment doesn't make sense.
man8alexd 7mo ago
systemd allows setting cgroup memory limits.

ChocolateGod 7mo ago

Is there a reason as to why Linux has not adopted straight up page compression like Windows and macOS?

GrayShade 7mo ago
It did, there's two incompatible approaches, zram and zswap.
creatonez 7mo ago
It has? Many distros have it enabled by default, too. The article discusses it quite a bit.

hackernudes 7mo ago

I see some comments about soft lockups during memory pressure. I have struggled with this immensely over the years. I wrote a userspace memory reclaimer daemon and have not had a lockup since: https://gist.github.com/EBADBEEF/f168458028f684a91148f4d3e791ba84 https://gist.github.com/EBADBEEF/f168458028f684a91148f4d3e79... . The hangs usually happened when I was stressing VFS (the computer was a samba server) along with other workloads. To trigger a hang manually I would read in large files (bigger than available ram) in parallel while running a game. I could get it to hang even with 128GB ram. I tweaked all the vfs settings (swappiness, etc...) to no avail. I tried with and without swap. In the end it looked like memory was not getting reclaimed fast enough, like linux would wait too long to start reclaiming memory and some critical process would get stuck waiting for some memory. The system would hang for minutes or hours at a time only making the tiniest of progress between reclaims. If I caught the problem early enough (just as everything started stuttering) I could trigger a reclaim manually by writing to '/sys/fs/cgroup/memory.reclaim' and the system would recover. I wonder if it was specific to btrfs or some specific workload pattern but I was never able to figure it out.

NooneAtAll3 7mo ago
I want to express my gratitude for not falling for greedy marketers' lies and using correct base-2 for KB/MB/GB
man8alexd 7mo ago
Just tune the kernel watermarks - vm.min_free_kbytes and vm.watermark_scale_factor
- hackernudes 7mo ago
  I do wish I had documented what I tried better! There might be a magic combo that could have helped but I tried tweaking a lot of the vm settings. One day I will probably see if I can still reproduce the original problem and be more methodical about it. More likely on list of things I might not ever get around to.

glyco 7mo ago

The new design is still wrong though - the swap system is a cache, there's no reason why a page should only exist on one level.