7 ms·

In defence of swap: common misconceptions (2018)

116 points by jitl 1y ago

01HNNWZ0MV43FF 1y ago
`sudo apt-get install earlyoom` Configure it to fire at like 5% and forget it. I've never seen the OOM do its dang job with or without swap.
- cmckn 1y ago
  If you've tried `systemd-oomd`, I'm curious what your thoughts are: https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html https://www.freedesktop.org/software/systemd/man/latest/syst...
  - man8alexd 1y ago
    In my environment, `systemd-oomd` does nothing with the default settings.
- internet_points 1y ago
  I enable oom_kill on sysrq, so I can hit alt+sysrq+f to invoke OOM; in /etc/sysctl.d/10-magic-sysrq.conf I have `kernel.sysrq = 240` (ie. 128+64+32+16, 128 being the one for the f key).
ggm 1y ago
Recognition that older linux swap strategies were unhelpful sometimes, which this piece of writing does, validates out past sense it wasn't working well. Regaining trust takes time. Sometimes I think if backing store and swap were more clearly delineated we might have got to decent algorithms sooner. Having a huge amount of swap pre-emptively claimed was making it look like starvation, when it was just a runtime planning strategy. It's also confusing how top and vmstat report things. Also, as a BSD mainly person, I think the differences stand out. I haven't noticed an OOM killer approach on BSD. Ancient model: twice as much swap as memory Old model: same amount of swap as memory New model: amount of swap your experience tells you this job mix demands to manage memory pressure fairly, which is a bit of a tall ask sometimes, but basically pick a number up to memory size.
- deleted 1y ago
  [deleted]
- kijin 1y ago
  For modern Linux servers with large amounts of RAM, my rule of thumb is between 1/8 and 1/32 of RAM, depending on what the machine is for. For example, one of my database servers has 128GB of RAM and 8GB of swap. It tends to stabilize around 108GB of RAM and 5GB of swap usage under normal load, so I know that a 4GB swap would have been less than optimal. A larger swap would have been a waste as well.
  - man8alexd 1y ago
    The proper rule of thumb is to make the swap large enough to keep all inactive anonymous pages after the workload has stabilized, but not too large to cause swap thrashing and a delayed OOM kill if a fast memory leak happens. Another rule of thumb is that performance degradation due to the active working set spilling into the swap is exponential - 0.1% excess causes 2x degradation, 1% - 10x degradation, 10% - 100x degradation (assuming 10^3 difference in latency between RAM and SSD).
    - kijin 1y ago
      I would approach the issue from the other direction. Start by buying enough RAM to contain the active working set for the foreseeable future. Afterward, you can start experimenting with different swap sizes (swapfiles are easier to resize, and they perform exactly as well as swap partitions!) to see how many inactive anonymous pages you can safely swap out. If you can swap out several gigabytes, that's a bonus! But don't take that for granted. Always be prepared to move everything back into RAM when needed.
  - ChocolateGod 1y ago
    I no longer use disk swap for servers, instead opting for Zram with a maximum is 50% of RAM capacity and a high swapiness value. It'd be cool if Zram could apply to the RAM itself (like macOS) rather than needing a fake swap device.
kijin 1y ago
> 6. Disabling swap doesn't prevent pathological behaviour at near-OOM, although it's true that having swap may prolong it. Whether the global OOM killer is invoked with or without swap, or was invoked sooner or later, the result is the same: you are left with a system in an unpredictable state. Having no swap doesn't avoid this. This is the most important reason I try to avoid having a large swap. The duration of pathological behavior at near-OOM is proportional to the amount of swap you have. The sooner your program is killed, the sooner your monitoring system can detect it ("Connection refused" is much more clear cut than random latency spikes) and reboot/reprovision the faulty server. We no longer live in a world where we need to keep a particular server online at all cost. When you have an army of servers, a dead server is preferable to a misbehaving server. OP tries to argue that a long period of thrashing will give you an opportunity for more visibility and controlled intervention. This does not match my experience. It takes ages even to log in to a machine that is thrashing hard, let alone run any serious commands on it. The sooner you just let it crash, the sooner you can restore the system to a working state and inspect the logs in a more comfortable environment.
- heavyset_go 1y ago
  Maybe I'm just insane, but if I'm on a machine with ample memory, and a process for some reason can't allocate resources, I want that process to fail ASAP. Same thing with high memory pressure situations, just kill greedy/hungry processes, please. Like something is going very wrong if the system is in that state, so I want everything to die immediately.
  - gfv 1y ago
    sysctl vm.overcommit_memory=2. However, programs for *nix-based systems usually expect overcommit to be on, for example, to support fork(). This is a stark contrast with Windows NT model, where an allocation will fail if it doesn't fit in the remaining memory+swap.
    - man8alexd 1y ago
      People disable memory overcommit, expecting to fix OOMs, and then they get surprised when their programs start failing mallocs while there are still tons of discardable page cache in the system. https://unix.stackexchange.com/q/797835/1027 https://unix.stackexchange.com/q/797835/1027 https://unix.stackexchange.com/q/797841/1027 https://unix.stackexchange.com/q/797841/1027
tanelpoder 1y ago
This wasn't on Linux, but on one of the old-school commercial Unixes - a customer had memory leaks in some of their daemon processes. They couldn't fix them for some reason. So they invested in additional swap space, let the processes slowly grow, swap out leaked stuff and restart them all over the weekend...
- rwmj 1y ago
  I wrote a chat server back in the 2000s which would gradually use more and more memory over a period of months. After extensive debugging, I couldn't find any memory leak and concluded the problem was likely inside glibc or caused by memory fragmentation. Solution was to have a cron job that ran every 3 months and rebooted the machine.
userbinator 1y ago
Under no/low memory contention on some workloads this may represent a non-trivial drop in performance due to stale, anonymous pages taking space away from more important use WTF?
- creshal 1y ago
  Welcome to the wonderful world of Java programs. When your tomcat abomination pulls in 500 dependencies for one method call each, and 80% of the methods aren't even called in regular use except to perform dependency injection mumbo jumbo during the 90 seconds your tomcat needs to start up, you easily end up with 70% of your application's anon pages being completely useless, but if you can't banish them to swap, they'll prevent the code on the hot path from having any memory left over for file caching. So even if you never run into OOM situations, adding a couple gigabytes of swap lets you free up that many gigabytes of RAM for file caching, and suddenly your application is on average 5x faster - but takes 3 seconds longer to service that one obscure API call that needs to dig all those pages back up. YMMV if you prefer consistently poor performance over inconsistent but usually much better performance.
  - marginalia_nu 1y ago
    Java's performance for cold code is bad period. This doesn't really have to do with code being paged out (that very rarely happens) but due to the JIT compiler not having warmed up the appropriate execution paths so that it runs in interpreted mode, often made worse as static object initialization happening when the first code that needs that particular class runs, and if you're unlucky with how the system was designed that may introduce cascading class initialization. Though any halfway competent Java developer following modern best practices will know to build systems that don't have these characteristics.
    - creshal 1y ago
      > Though any halfway competent Java developer following modern best practices will know to build systems that don't have these characteristics. I'll let you know if I ever meet any. Until then, another terabyte of RAM for tomcat.
goopypoop 1y ago
swap.avi is its own damning defence
LegionMammal978 1y ago
On my desktop system, most of my problems with swap come from dealing with the aftermath of an out-of-control process eating all my RAM. In this case, the offending program demands memory so quickly that everything from legitimate programs gets swapped out. These programs proceed to run poorly for the next several minutes to an hour depending on usage, since the OS only swaps pages back in once they are referenced, even if there is plenty of free space not even being used in the disk cache. Eventually I wrote a small script that does the equivalent of "sudo swapoff -a && sudo swapon -a" to eagerly flush everything to RAM, but I was surprised by how many people seemed to think there's no legitimate reason to ever want to do so.
- deleted 1y ago
  [deleted]
- man8alexd 1y ago
  [dead]
- hugo1789 1y ago
  That works if there is enough memory after the "bad" process has been killed. The question is, is it necessary? Many systems can live with processes performing a little bit poorly for some minutes and I wouldn't do it.
  - michaelt 1y ago
    > The question is, is it necessary? Many systems can live with processes performing a little bit poorly for some minutes and I wouldn't do it. The outage ain't resolved until things are back to operating normally. If things aren't back to 100% healthy, could be I didn't truly find the root cause of the problem - in which case I'll probably be woken up again in 30 minutes when the problem comes back.
    - whatevaa 1y ago
      Desktops are not servers. There could be no problem, just some hungry legitimate program (or vm).
vbezhenar 1y ago
I've always created swap of 1.5x - 4x RAM size on every Linux computer I've had to manage and never had any issues with it. That's my rule that I learned many years ago, follow to this day and will follow. Worst thing: I left 5% of my SSD unused which will actually be used for garbage collection and other staff. That's OK. What I don't understand is why modern Linux is so shy of touching swap. With old kernels, Linux happily pushed unused pages to a swap, so even if you don't eat memory, your swap will be filled with tens or hundreds MB of memory and that's a great thing. Modern kernel just keeps swap usage at 0, until memory is exhausted.
- kevin_thibedeau 1y ago
  I haven't used swap for 15 years. You have to be judicious about heavy app usage with only 16GiB. With 32GiB, I've never triggered OOM.
  - aidenn0 1y ago
    If you never trigger OOM, then you can basically only benefit from enabling a modest amount of swap.
- creshal 1y ago
  > I've always created swap of 1.5x - 4x RAM size on every Linux computer I've had to manage and never had any issues with it. That's a couple terabyte of swap on servers these days, and even on laptops I wouldn't want to deal with 300-ish GB swap.
- bawolff 1y ago
  > What I don't understand is why modern Linux is so shy of touching swap. With old kernels, Linux happily pushed unused pages to a swap, so even if you don't eat memory, your swap will be filled with tens or hundreds MB of memory and that's a great thing. Modern kernel just keeps swap usage at 0, until memory is exhausted. The article has the answer.
deleted 1y ago
[deleted]
creshal 1y ago
I wish people would actually read TFA instead of reflexively repeating nonsensical folk remedies.
- JdeBP 1y ago
  I've been telling people about this since the days when there were operating systems still around that actually did swapping (16-bit OS/2, old Unix, Standard Mode DOS+Windows) rather than paging (32-bit OS/2, 386 Enhanced Mode DOS+Windows, Windows NT). I wrote a Frequently Given Answer about it in 2007, I had had to repeat the point so many times since the middle 1990s; and I was far from alone even then. * http://jdebp.uk./FGA/dont-throw-those-paging-files-away.html http://jdebp.uk./FGA/dont-throw-those-paging-files-away.html The erroneous folk wisdom is widespread. It often seems to lack any mention of the concepts of a resident set and a working set, and is always mixed in with a wishful thinking idea that somehow "new" computers obviate this, when the basic principles of demand paging are the same as they were four decades ago, Parkinson's Law can still be observed operating in the world of computers, and the "new" computers all of those years ago didn't manage to obviate paging files either.
  - p_ing 1y ago
    The swapfile.sys in Windows 8+ is used for process swapping (moving the entire private working set out of memory to disk), but only for UWP applications.
- mmphosis 1y ago
  TFA means The Fine Article
Panzerschrek 1y ago
As I understand this article, swap is useful for cases where many long-lived programs (daemons) allocate a lot of memory, but almost never access it. But wouldn't it be better to avoid writing such programs? And how many memory such daemons can consume? A couple of hundred megabytes total? Is it really that much on modern systems? My experience with swap shows, that it only makes things worse. When I program, my application may sometimes allocate a lot of memory due to some silly bug. In such case the whole system practically stops working - even mouse cursor can't move. If I am happy, OOM killer will eventually kill my buggy program, but after that it's not over - almost all used memory is now in swap and the whole system works snail-slow, presumably because kernel doesn't think it should really unswap previously swapped memory and does this only on demand and only page by page. I a hypothetical case without swap this case isn't so painful. When main system memory is almost fully consumed, OOM killer kills the most memory hungry program and all other programs just continue working as before. I think that overall reliance on swap is noways just a legacy of old times when main memory was scarce and back than it maybe was useful to have swap. OS kernels should be redesigned to work without swap, this will make system behavior smoother and kernel code may be simpler (all this swapping code may be removed) and thus faster.
- Rohansi 1y ago
  > As I understand this article, swap is useful for cases where many long-lived programs (daemons) allocate a lot of memory, but almost never access it. But wouldn't it be better to avoid writing such programs? Ideally yes, but is that something you keep in mind when you write software? Do you ever consider freeing memory just because it hasn't been used in a while? How do you decide when to free it? This is all handled automatically when you have swap enabled, and at a granularity that is much higher than you can practically manually implement it.
  - Panzerschrek 1y ago
    I write mostly C++ or Rust programs. In these languages memory is freed as soon as it's no longer in use (thanks to destructors). So, usually this shouldn't be actively kept in mind. The only exception are cases like caches, but long-running programs should use caching carefully - limit cache size and free cache entries after some amount of time. Programs, which allocate large amounts of memory without strict necessity to do so, are just a consequence of swap existence. "Thanks" to swap they weren't properly tested in low-memory conditions and thus no necessary optimization were done.
tsoukase 1y ago
In my humble experience, if you run out of memory in Linux you are f... up, irrespective of swap present and/or OOM getting in. On the other side, a Raspberry Pi freezed unexpectedly (not due to low memory) until a very small swap file was enabled. It was almost never used but the freezes stopped. Fun swap stories.
- fuzzfactor 1y ago
  >There's also a lot of misunderstanding about the purpose of swap – many people just see it as a kind of "slow extra memory" for use in emergencies, but don't understand how it can contribute during normal load to the healthy operation of an operating system as a whole. That's the long-standing defect that needs to be corrected then, there should be no dependence on swap existing whatsoever as long as you have more than enough memory for the entire workload.
jitl 1y ago
I am testing a distributed database-like system at work that makes heavy use of swap. At startup, we read a table from S3 and compute a recursive materialized view over it. This needs about 4TB of “memory” per node while computing, which we provide as 512gb of RAM + 3900GB of NVMe zswap enabled swap devices. Once the computation is complete, we’re left with a much smaller working set index (about 400gb) we use to serve queries. For this use-case, swap serves as a performant and less labor intensive approach to manually spilling the computation to disk in application code (although there is some mlock going on; it’s not entirely automatic). This is like a very extreme version of the initialization-only pages idea discussed in the articule. The warm up computation does take like 1/4 the time if it can live entirely in RAM, but using NVMe as “discount RAM” reduces the United States dollar cost of the system by 97% compared to RAM-only.
- zozbot234 1y ago
  The problem with heavy swapping on NVMe (or other flash memory) is that it wears out the flash storage very quickly, even for seemingly "reasonable" workloads. In a way, the high performance of NVMe can work against you. Definitely something you want to check out via SMART or similar wearout stats.
  - man8alexd 1y ago
    [dead]
  - p_ing 1y ago
    While what you stated is overall not true, who cares with a 97% cost savings vs RAM? Just pop in another NVMe when one fails.
  - jitl 1y ago
    Let’s say we’re spending $1 million on hardware hypothetically with the swap setup. At that price point, either we use swap and let the kernel engineers move data from RAM to disk and back, or we disable swap and need user space code to move the same data to disk and back. We’d need to price out writing & maintaining the user space implementation (mmap perhaps?) for it to be fair price comparison. To avoid SSD wear and tear, we could spend $29 million a year more to put the data in RAM only. Not worth! (We rent EC2 instances from AWS, so SSD wear is baked into the pricing)
naniwaduni 1y ago
The way I learned it, swap is basically the inverse of file caching: in much the way that extra memory can be used to cache more frequently-used files, then evicted when a "better" use for that memory comes around; swap can be used to save rarely-used anonymous memory so that you can "evict" them when there are other things you'd rather have in memory, then pull them back into memory if they ever become relevant again. Or to look at it from another perspective... it lets you reclaim unprovable memory leaks.
shatsky 1y ago
Author pushes abstract idea about "page reclamation" in front of ideas of performance, reliability and controllable service degradation which people actually want; because author believes that it is the one and only solution to them; and then defends swap because it is good for it. No, this is just plain wrong. There are very specific problems which happen when there is not enough memory. 1. File-backed page reads causing more disk reads, eventually ending with "programs being executed from disk" (shared libraries are also mmaped) which feels like system lockup. This does not need any "egalitarian reclamation" abstraction and swap, and swap does not solve it. But it can be solved simply by reserving some minimal amount of memory for buf/cache, with which system is still responsive. 2. Eventually failure to allocate more memory for some process. Any solutions like "page reclamation" with pushing unused pages to some swap can only increase maximum amount of memory which can be used before it happens, from one finite value to bigger finite value. When there is no memory to free without losing data, some process must be killed. Swap does not solve this. The least bad solution would be to warn user in advance and let them choose processes to kill. See also https://github.com/hakavlad/prelockd https://github.com/hakavlad/prelockd
- man8alexd 1y ago
  Neither executables nor shared libraries are going to be evicted if they are in active use and have the "accessed" bit set in their page tables. This code has been present in the kernel mm/vmscan.c at least since 2012.
  - shatsky 1y ago
    Will look into that again. If you're right about unevictability of these pages, what is the mechanism which causes sudden extreme degradation of performance when system is almost out of memory due to some app gradually consuming it, from quite responsive system to totally unresponsive system which can stay stuck with trashing disk for ages until oom will fire?
    - man8alexd 1y ago
      Once your active working set starts spilling into swap, the performance degradation goes exponential. The difference in latency between RAM and SSD is orders of magnitude. Assuming 10^3 difference, 0.1% memory excess causes 2x degradation, 1% - 10x degradation, 10% - 100x degradation.
tmtvl 1y ago
Nothing about zram/zswap? I know that zram is more performant but I wonder how it holds up under high memory pressure compared to zswap.
shawnz 1y ago
It's crazy to me that even Fedora disables swap to disk by default now. It really speaks to how broadly misunderstood swap is
fpoling 1y ago
The article has not mentioned memory compression as an alternative to swap which many Linux distributions enable by default. On the other hand these days latest SSD are way faster than memory compression even with LUKS encryption on and even when compression uses LZ4 compression. Plus modern SSDs do not suffer from frequent writes as before so on my laptop I disabled the memory compression and then all reasoning from the article applies again. Then on a development laptop running compilations/containers/VMs/browser vm.swappines does not seems matter that much if one has enough memory. So I no longer tune it to 100 or more and leave at the default 60%.
- vlovich123 1y ago
  > these days latest SSD are way faster than memory compression That's a really provocative claim. Any benchmarks to support this?
  - fpoling 1y ago
    On my laptop with Samsung PRO 990 SSD and Intel Core Ultra 7 165U CPU with 64G RAM under Debian 13: Read test via coping to RAM memory from LUKS-encrypted BTRFS against /tmp that is a standard RAM disk: $ dd of=/tmp/input-90K.jsonl if=input-90K.jsonl conv=fdatasync bs=10M iflag=direct oflag=direct 2596+1 records in 2596+1 records out 27225334502 bytes (27 GB, 25 GiB) copied, 20.4403 s, 1.3 GB/s Write test: $ dd if=/tmp/input-90K.jsonl of=tmp.jsonl conv=fdatasync bs=10M oflag=direct 2596+1 records in 2596+1 records out 27225334502 bytes (27 GB, 25 GiB) copied, 16.8612 s, 1.6 GB/s Preparing RAM disk with zram compression to emulate zram: $ sudo zramctl --algorithm=lz4 --size=30GiB /dev/zram0 $ sudo mkfs.ext4 /dev/zram0 $ sudo mount /dev/zram0 /mnt $ df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/zram0 30G 2.1M 28G 1% /mnt Write test to lz4-compressed ZRAM: $ sudo dd if=/tmp/input-90K.jsonl of=/mnt/tmp.jsonl conv=fdatasync bs=10M oflag=direct iflag=direct 2596+1 records in 2596+1 records out 27225334502 bytes (27 GB, 25 GiB) copied, 93.2813 s, 292 MB/s Read test from lz4-compressed ZRAM: $ dd of=/tmp/input-90K.jsonl if=/mnt/tmp.jsonl conv=fdatasync bs=10M oflag=direct iflag=direct 2596+1 records in 2596+1 records out 27225334502 bytes (27 GB, 25 GiB) copied, 34.8479 s, 781 MB/s So SSD with LUKS is 1.5 times faster than zram for read and 5 times faster than zram for write. Note without LUKS but native SSD encryption the speed of SSD will be at least 2 times faster. Also using recent kernel is important so LUKS uses CPU instructions for AES encryptions. Without that SSD under LUKS will be several times slower.
ciupicri 1y ago
The problem with swap for me is that in time for some reason Chrome tends to be swapped out even if free shows me some available memory when I notice its sluggishness. I've tried setting vm.swapping to <20, even 5, but it doesn't help too much.
- blueflow 1y ago
  Because Chrome's program text itself is reclaimable and counts as "available".
  - man8alexd 1y ago
    active pages are not reclaimable.
    - blueflow 1y ago
      Pages will not stay in the active LRU if the accessor is stalled. Edit: You keep repeating that all over the thread - i started my week an hour ago and i had two of these memory stall events during the weekend. Again its the machines with big binaries running and no swap. Maybe you could provide a better explanation than "this doesn't happen"?
      - man8alexd 1y ago
        You keep repeating that the program text pages are being evicted as if that were the only thing that happens under memory pressure. It is the last thing that happens after the kernel drops all regular page cache, causing I/O starvation, and without swap, the kernel can't do anything about inactive anon pages. This means that at this moment, you are almost out of memory and the kernel can't do anything other than shuffle the remaining pages, thrashing the I/O. The kernel can't predict the future and doesn't know if it is a temporary situation due to some load spike or if it is a memory leak and resources won't be available again. The kernel doesn't know SLOs and operator priorities - whether the workload should survive at all costs, or the stalls are not acceptable and the workload should be OOM-killed. It is the job of the operator to give the kernel appropriate information by setting cgroup limits and monitoring memory pressure. If you want the workload to be OOM-killed before it consumes all the memory and starts causing I/O thrashing - set `memory.max` and `memory.swap.max` cgroup limits. If you want the workload memory not to be reclaimed by the kernel - set `memory.min`.
hedora 1y ago
Is linux considered stable with swap completely disabled these days? In the past, they recommended against that for deadlock reasons. On the workloads I care about (desktop, and servers that avoid mmap), the anonymous dirty page part of the kernel heap is < 10% of RAM, so swap is mostly just there to waste space and slow down the oomkiller.
egberts1 1y ago
It is still problematic to get your swap space encrypted-by-default (usual way is via LVM); still doable, workable, but jumpy. The Linux Kernel still have this global lock for swapiness. Downside is no smooth playing video.

116 points by jitl 1y ago

`sudo apt-get install earlyoom` Configure it to fire at like 5% and forget it. I've never seen the OOM do its dang job with or without swap.

cmckn 1y ago
If you've tried `systemd-oomd`, I'm curious what your thoughts are: https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html https://www.freedesktop.org/software/systemd/man/latest/syst...
- man8alexd 1y ago
  In my environment, `systemd-oomd` does nothing with the default settings.
internet_points 1y ago
I enable oom_kill on sysrq, so I can hit alt+sysrq+f to invoke OOM; in /etc/sysctl.d/10-magic-sysrq.conf I have `kernel.sysrq = 240` (ie. 128+64+32+16, 128 being the one for the f key).

ggm 1y ago

Recognition that older linux swap strategies were unhelpful sometimes, which this piece of writing does, validates out past sense it wasn't working well. Regaining trust takes time. Sometimes I think if backing store and swap were more clearly delineated we might have got to decent algorithms sooner. Having a huge amount of swap pre-emptively claimed was making it look like starvation, when it was just a runtime planning strategy. It's also confusing how top and vmstat report things. Also, as a BSD mainly person, I think the differences stand out. I haven't noticed an OOM killer approach on BSD. Ancient model: twice as much swap as memory Old model: same amount of swap as memory New model: amount of swap your experience tells you this job mix demands to manage memory pressure fairly, which is a bit of a tall ask sometimes, but basically pick a number up to memory size.

deleted 1y ago
[deleted]
kijin 1y ago
For modern Linux servers with large amounts of RAM, my rule of thumb is between 1/8 and 1/32 of RAM, depending on what the machine is for. For example, one of my database servers has 128GB of RAM and 8GB of swap. It tends to stabilize around 108GB of RAM and 5GB of swap usage under normal load, so I know that a 4GB swap would have been less than optimal. A larger swap would have been a waste as well.
- man8alexd 1y ago
  The proper rule of thumb is to make the swap large enough to keep all inactive anonymous pages after the workload has stabilized, but not too large to cause swap thrashing and a delayed OOM kill if a fast memory leak happens. Another rule of thumb is that performance degradation due to the active working set spilling into the swap is exponential - 0.1% excess causes 2x degradation, 1% - 10x degradation, 10% - 100x degradation (assuming 10^3 difference in latency between RAM and SSD).
  - kijin 1y ago
    I would approach the issue from the other direction. Start by buying enough RAM to contain the active working set for the foreseeable future. Afterward, you can start experimenting with different swap sizes (swapfiles are easier to resize, and they perform exactly as well as swap partitions!) to see how many inactive anonymous pages you can safely swap out. If you can swap out several gigabytes, that's a bonus! But don't take that for granted. Always be prepared to move everything back into RAM when needed.
- ChocolateGod 1y ago
  I no longer use disk swap for servers, instead opting for Zram with a maximum is 50% of RAM capacity and a high swapiness value. It'd be cool if Zram could apply to the RAM itself (like macOS) rather than needing a fake swap device.

kijin 1y ago

> 6. Disabling swap doesn't prevent pathological behaviour at near-OOM, although it's true that having swap may prolong it. Whether the global OOM killer is invoked with or without swap, or was invoked sooner or later, the result is the same: you are left with a system in an unpredictable state. Having no swap doesn't avoid this. This is the most important reason I try to avoid having a large swap. The duration of pathological behavior at near-OOM is proportional to the amount of swap you have. The sooner your program is killed, the sooner your monitoring system can detect it ("Connection refused" is much more clear cut than random latency spikes) and reboot/reprovision the faulty server. We no longer live in a world where we need to keep a particular server online at all cost. When you have an army of servers, a dead server is preferable to a misbehaving server. OP tries to argue that a long period of thrashing will give you an opportunity for more visibility and controlled intervention. This does not match my experience. It takes ages even to log in to a machine that is thrashing hard, let alone run any serious commands on it. The sooner you just let it crash, the sooner you can restore the system to a working state and inspect the logs in a more comfortable environment.

heavyset_go 1y ago
Maybe I'm just insane, but if I'm on a machine with ample memory, and a process for some reason can't allocate resources, I want that process to fail ASAP. Same thing with high memory pressure situations, just kill greedy/hungry processes, please. Like something is going very wrong if the system is in that state, so I want everything to die immediately.
- gfv 1y ago
  sysctl vm.overcommit_memory=2. However, programs for *nix-based systems usually expect overcommit to be on, for example, to support fork(). This is a stark contrast with Windows NT model, where an allocation will fail if it doesn't fit in the remaining memory+swap.
  - man8alexd 1y ago
    People disable memory overcommit, expecting to fix OOMs, and then they get surprised when their programs start failing mallocs while there are still tons of discardable page cache in the system. https://unix.stackexchange.com/q/797835/1027 https://unix.stackexchange.com/q/797835/1027 https://unix.stackexchange.com/q/797841/1027 https://unix.stackexchange.com/q/797841/1027

tanelpoder 1y ago

This wasn't on Linux, but on one of the old-school commercial Unixes - a customer had memory leaks in some of their daemon processes. They couldn't fix them for some reason. So they invested in additional swap space, let the processes slowly grow, swap out leaked stuff and restart them all over the weekend...

rwmj 1y ago
I wrote a chat server back in the 2000s which would gradually use more and more memory over a period of months. After extensive debugging, I couldn't find any memory leak and concluded the problem was likely inside glibc or caused by memory fragmentation. Solution was to have a cron job that ran every 3 months and rebooted the machine.

userbinator 1y ago

Under no/low memory contention on some workloads this may represent a non-trivial drop in performance due to stale, anonymous pages taking space away from more important use WTF?

creshal 1y ago
Welcome to the wonderful world of Java programs. When your tomcat abomination pulls in 500 dependencies for one method call each, and 80% of the methods aren't even called in regular use except to perform dependency injection mumbo jumbo during the 90 seconds your tomcat needs to start up, you easily end up with 70% of your application's anon pages being completely useless, but if you can't banish them to swap, they'll prevent the code on the hot path from having any memory left over for file caching. So even if you never run into OOM situations, adding a couple gigabytes of swap lets you free up that many gigabytes of RAM for file caching, and suddenly your application is on average 5x faster - but takes 3 seconds longer to service that one obscure API call that needs to dig all those pages back up. YMMV if you prefer consistently poor performance over inconsistent but usually much better performance.
- marginalia_nu 1y ago
  Java's performance for cold code is bad period. This doesn't really have to do with code being paged out (that very rarely happens) but due to the JIT compiler not having warmed up the appropriate execution paths so that it runs in interpreted mode, often made worse as static object initialization happening when the first code that needs that particular class runs, and if you're unlucky with how the system was designed that may introduce cascading class initialization. Though any halfway competent Java developer following modern best practices will know to build systems that don't have these characteristics.
  - creshal 1y ago
    > Though any halfway competent Java developer following modern best practices will know to build systems that don't have these characteristics. I'll let you know if I ever meet any. Until then, another terabyte of RAM for tomcat.

goopypoop 1y ago

swap.avi is its own damning defence

LegionMammal978 1y ago

On my desktop system, most of my problems with swap come from dealing with the aftermath of an out-of-control process eating all my RAM. In this case, the offending program demands memory so quickly that everything from legitimate programs gets swapped out. These programs proceed to run poorly for the next several minutes to an hour depending on usage, since the OS only swaps pages back in once they are referenced, even if there is plenty of free space not even being used in the disk cache. Eventually I wrote a small script that does the equivalent of "sudo swapoff -a && sudo swapon -a" to eagerly flush everything to RAM, but I was surprised by how many people seemed to think there's no legitimate reason to ever want to do so.

deleted 1y ago
[deleted]
man8alexd 1y ago
[dead]
hugo1789 1y ago
That works if there is enough memory after the "bad" process has been killed. The question is, is it necessary? Many systems can live with processes performing a little bit poorly for some minutes and I wouldn't do it.
- michaelt 1y ago
  > The question is, is it necessary? Many systems can live with processes performing a little bit poorly for some minutes and I wouldn't do it. The outage ain't resolved until things are back to operating normally. If things aren't back to 100% healthy, could be I didn't truly find the root cause of the problem - in which case I'll probably be woken up again in 30 minutes when the problem comes back.
  - whatevaa 1y ago
    Desktops are not servers. There could be no problem, just some hungry legitimate program (or vm).

vbezhenar 1y ago

I've always created swap of 1.5x - 4x RAM size on every Linux computer I've had to manage and never had any issues with it. That's my rule that I learned many years ago, follow to this day and will follow. Worst thing: I left 5% of my SSD unused which will actually be used for garbage collection and other staff. That's OK. What I don't understand is why modern Linux is so shy of touching swap. With old kernels, Linux happily pushed unused pages to a swap, so even if you don't eat memory, your swap will be filled with tens or hundreds MB of memory and that's a great thing. Modern kernel just keeps swap usage at 0, until memory is exhausted.

kevin_thibedeau 1y ago
I haven't used swap for 15 years. You have to be judicious about heavy app usage with only 16GiB. With 32GiB, I've never triggered OOM.
- aidenn0 1y ago
  If you never trigger OOM, then you can basically only benefit from enabling a modest amount of swap.
creshal 1y ago
> I've always created swap of 1.5x - 4x RAM size on every Linux computer I've had to manage and never had any issues with it. That's a couple terabyte of swap on servers these days, and even on laptops I wouldn't want to deal with 300-ish GB swap.
bawolff 1y ago
> What I don't understand is why modern Linux is so shy of touching swap. With old kernels, Linux happily pushed unused pages to a swap, so even if you don't eat memory, your swap will be filled with tens or hundreds MB of memory and that's a great thing. Modern kernel just keeps swap usage at 0, until memory is exhausted. The article has the answer.

deleted 1y ago

[deleted]

creshal 1y ago

I wish people would actually read TFA instead of reflexively repeating nonsensical folk remedies.

JdeBP 1y ago
I've been telling people about this since the days when there were operating systems still around that actually did swapping (16-bit OS/2, old Unix, Standard Mode DOS+Windows) rather than paging (32-bit OS/2, 386 Enhanced Mode DOS+Windows, Windows NT). I wrote a Frequently Given Answer about it in 2007, I had had to repeat the point so many times since the middle 1990s; and I was far from alone even then. * http://jdebp.uk./FGA/dont-throw-those-paging-files-away.html http://jdebp.uk./FGA/dont-throw-those-paging-files-away.html The erroneous folk wisdom is widespread. It often seems to lack any mention of the concepts of a resident set and a working set, and is always mixed in with a wishful thinking idea that somehow "new" computers obviate this, when the basic principles of demand paging are the same as they were four decades ago, Parkinson's Law can still be observed operating in the world of computers, and the "new" computers all of those years ago didn't manage to obviate paging files either.
- p_ing 1y ago
  The swapfile.sys in Windows 8+ is used for process swapping (moving the entire private working set out of memory to disk), but only for UWP applications.
mmphosis 1y ago
TFA means The Fine Article

Panzerschrek 1y ago

As I understand this article, swap is useful for cases where many long-lived programs (daemons) allocate a lot of memory, but almost never access it. But wouldn't it be better to avoid writing such programs? And how many memory such daemons can consume? A couple of hundred megabytes total? Is it really that much on modern systems? My experience with swap shows, that it only makes things worse. When I program, my application may sometimes allocate a lot of memory due to some silly bug. In such case the whole system practically stops working - even mouse cursor can't move. If I am happy, OOM killer will eventually kill my buggy program, but after that it's not over - almost all used memory is now in swap and the whole system works snail-slow, presumably because kernel doesn't think it should really unswap previously swapped memory and does this only on demand and only page by page. I a hypothetical case without swap this case isn't so painful. When main system memory is almost fully consumed, OOM killer kills the most memory hungry program and all other programs just continue working as before. I think that overall reliance on swap is noways just a legacy of old times when main memory was scarce and back than it maybe was useful to have swap. OS kernels should be redesigned to work without swap, this will make system behavior smoother and kernel code may be simpler (all this swapping code may be removed) and thus faster.

Rohansi 1y ago
> As I understand this article, swap is useful for cases where many long-lived programs (daemons) allocate a lot of memory, but almost never access it. But wouldn't it be better to avoid writing such programs? Ideally yes, but is that something you keep in mind when you write software? Do you ever consider freeing memory just because it hasn't been used in a while? How do you decide when to free it? This is all handled automatically when you have swap enabled, and at a granularity that is much higher than you can practically manually implement it.
- Panzerschrek 1y ago
  I write mostly C++ or Rust programs. In these languages memory is freed as soon as it's no longer in use (thanks to destructors). So, usually this shouldn't be actively kept in mind. The only exception are cases like caches, but long-running programs should use caching carefully - limit cache size and free cache entries after some amount of time. Programs, which allocate large amounts of memory without strict necessity to do so, are just a consequence of swap existence. "Thanks" to swap they weren't properly tested in low-memory conditions and thus no necessary optimization were done.

tsoukase 1y ago

In my humble experience, if you run out of memory in Linux you are f... up, irrespective of swap present and/or OOM getting in. On the other side, a Raspberry Pi freezed unexpectedly (not due to low memory) until a very small swap file was enabled. It was almost never used but the freezes stopped. Fun swap stories.

fuzzfactor 1y ago
>There's also a lot of misunderstanding about the purpose of swap – many people just see it as a kind of "slow extra memory" for use in emergencies, but don't understand how it can contribute during normal load to the healthy operation of an operating system as a whole. That's the long-standing defect that needs to be corrected then, there should be no dependence on swap existing whatsoever as long as you have more than enough memory for the entire workload.

jitl 1y ago

I am testing a distributed database-like system at work that makes heavy use of swap. At startup, we read a table from S3 and compute a recursive materialized view over it. This needs about 4TB of “memory” per node while computing, which we provide as 512gb of RAM + 3900GB of NVMe zswap enabled swap devices. Once the computation is complete, we’re left with a much smaller working set index (about 400gb) we use to serve queries. For this use-case, swap serves as a performant and less labor intensive approach to manually spilling the computation to disk in application code (although there is some mlock going on; it’s not entirely automatic). This is like a very extreme version of the initialization-only pages idea discussed in the articule. The warm up computation does take like 1/4 the time if it can live entirely in RAM, but using NVMe as “discount RAM” reduces the United States dollar cost of the system by 97% compared to RAM-only.

zozbot234 1y ago
The problem with heavy swapping on NVMe (or other flash memory) is that it wears out the flash storage very quickly, even for seemingly "reasonable" workloads. In a way, the high performance of NVMe can work against you. Definitely something you want to check out via SMART or similar wearout stats.
- man8alexd 1y ago
  [dead]
- p_ing 1y ago
  While what you stated is overall not true, who cares with a 97% cost savings vs RAM? Just pop in another NVMe when one fails.
- jitl 1y ago
  Let’s say we’re spending $1 million on hardware hypothetically with the swap setup. At that price point, either we use swap and let the kernel engineers move data from RAM to disk and back, or we disable swap and need user space code to move the same data to disk and back. We’d need to price out writing & maintaining the user space implementation (mmap perhaps?) for it to be fair price comparison. To avoid SSD wear and tear, we could spend $29 million a year more to put the data in RAM only. Not worth! (We rent EC2 instances from AWS, so SSD wear is baked into the pricing)

naniwaduni 1y ago

The way I learned it, swap is basically the inverse of file caching: in much the way that extra memory can be used to cache more frequently-used files, then evicted when a "better" use for that memory comes around; swap can be used to save rarely-used anonymous memory so that you can "evict" them when there are other things you'd rather have in memory, then pull them back into memory if they ever become relevant again. Or to look at it from another perspective... it lets you reclaim unprovable memory leaks.

shatsky 1y ago

Author pushes abstract idea about "page reclamation" in front of ideas of performance, reliability and controllable service degradation which people actually want; because author believes that it is the one and only solution to them; and then defends swap because it is good for it. No, this is just plain wrong. There are very specific problems which happen when there is not enough memory. 1. File-backed page reads causing more disk reads, eventually ending with "programs being executed from disk" (shared libraries are also mmaped) which feels like system lockup. This does not need any "egalitarian reclamation" abstraction and swap, and swap does not solve it. But it can be solved simply by reserving some minimal amount of memory for buf/cache, with which system is still responsive. 2. Eventually failure to allocate more memory for some process. Any solutions like "page reclamation" with pushing unused pages to some swap can only increase maximum amount of memory which can be used before it happens, from one finite value to bigger finite value. When there is no memory to free without losing data, some process must be killed. Swap does not solve this. The least bad solution would be to warn user in advance and let them choose processes to kill. See also https://github.com/hakavlad/prelockd https://github.com/hakavlad/prelockd

man8alexd 1y ago
Neither executables nor shared libraries are going to be evicted if they are in active use and have the "accessed" bit set in their page tables. This code has been present in the kernel mm/vmscan.c at least since 2012.
- shatsky 1y ago
  Will look into that again. If you're right about unevictability of these pages, what is the mechanism which causes sudden extreme degradation of performance when system is almost out of memory due to some app gradually consuming it, from quite responsive system to totally unresponsive system which can stay stuck with trashing disk for ages until oom will fire?
  - man8alexd 1y ago
    Once your active working set starts spilling into swap, the performance degradation goes exponential. The difference in latency between RAM and SSD is orders of magnitude. Assuming 10^3 difference, 0.1% memory excess causes 2x degradation, 1% - 10x degradation, 10% - 100x degradation.

tmtvl 1y ago

Nothing about zram/zswap? I know that zram is more performant but I wonder how it holds up under high memory pressure compared to zswap.

shawnz 1y ago

It's crazy to me that even Fedora disables swap to disk by default now. It really speaks to how broadly misunderstood swap is

fpoling 1y ago

The article has not mentioned memory compression as an alternative to swap which many Linux distributions enable by default. On the other hand these days latest SSD are way faster than memory compression even with LUKS encryption on and even when compression uses LZ4 compression. Plus modern SSDs do not suffer from frequent writes as before so on my laptop I disabled the memory compression and then all reasoning from the article applies again. Then on a development laptop running compilations/containers/VMs/browser vm.swappines does not seems matter that much if one has enough memory. So I no longer tune it to 100 or more and leave at the default 60%.

vlovich123 1y ago
> these days latest SSD are way faster than memory compression That's a really provocative claim. Any benchmarks to support this?
- fpoling 1y ago
  On my laptop with Samsung PRO 990 SSD and Intel Core Ultra 7 165U CPU with 64G RAM under Debian 13: Read test via coping to RAM memory from LUKS-encrypted BTRFS against /tmp that is a standard RAM disk: $ dd of=/tmp/input-90K.jsonl if=input-90K.jsonl conv=fdatasync bs=10M iflag=direct oflag=direct 2596+1 records in 2596+1 records out 27225334502 bytes (27 GB, 25 GiB) copied, 20.4403 s, 1.3 GB/s Write test: $ dd if=/tmp/input-90K.jsonl of=tmp.jsonl conv=fdatasync bs=10M oflag=direct 2596+1 records in 2596+1 records out 27225334502 bytes (27 GB, 25 GiB) copied, 16.8612 s, 1.6 GB/s Preparing RAM disk with zram compression to emulate zram: $ sudo zramctl --algorithm=lz4 --size=30GiB /dev/zram0 $ sudo mkfs.ext4 /dev/zram0 $ sudo mount /dev/zram0 /mnt $ df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/zram0 30G 2.1M 28G 1% /mnt Write test to lz4-compressed ZRAM: $ sudo dd if=/tmp/input-90K.jsonl of=/mnt/tmp.jsonl conv=fdatasync bs=10M oflag=direct iflag=direct 2596+1 records in 2596+1 records out 27225334502 bytes (27 GB, 25 GiB) copied, 93.2813 s, 292 MB/s Read test from lz4-compressed ZRAM: $ dd of=/tmp/input-90K.jsonl if=/mnt/tmp.jsonl conv=fdatasync bs=10M oflag=direct iflag=direct 2596+1 records in 2596+1 records out 27225334502 bytes (27 GB, 25 GiB) copied, 34.8479 s, 781 MB/s So SSD with LUKS is 1.5 times faster than zram for read and 5 times faster than zram for write. Note without LUKS but native SSD encryption the speed of SSD will be at least 2 times faster. Also using recent kernel is important so LUKS uses CPU instructions for AES encryptions. Without that SSD under LUKS will be several times slower.

ciupicri 1y ago

The problem with swap for me is that in time for some reason Chrome tends to be swapped out even if free shows me some available memory when I notice its sluggishness. I've tried setting vm.swapping to <20, even 5, but it doesn't help too much.

blueflow 1y ago
Because Chrome's program text itself is reclaimable and counts as "available".
- man8alexd 1y ago
  active pages are not reclaimable.
  - blueflow 1y ago
    Pages will not stay in the active LRU if the accessor is stalled. Edit: You keep repeating that all over the thread - i started my week an hour ago and i had two of these memory stall events during the weekend. Again its the machines with big binaries running and no swap. Maybe you could provide a better explanation than "this doesn't happen"?
    - man8alexd 1y ago
      You keep repeating that the program text pages are being evicted as if that were the only thing that happens under memory pressure. It is the last thing that happens after the kernel drops all regular page cache, causing I/O starvation, and without swap, the kernel can't do anything about inactive anon pages. This means that at this moment, you are almost out of memory and the kernel can't do anything other than shuffle the remaining pages, thrashing the I/O. The kernel can't predict the future and doesn't know if it is a temporary situation due to some load spike or if it is a memory leak and resources won't be available again. The kernel doesn't know SLOs and operator priorities - whether the workload should survive at all costs, or the stalls are not acceptable and the workload should be OOM-killed. It is the job of the operator to give the kernel appropriate information by setting cgroup limits and monitoring memory pressure. If you want the workload to be OOM-killed before it consumes all the memory and starts causing I/O thrashing - set `memory.max` and `memory.swap.max` cgroup limits. If you want the workload memory not to be reclaimed by the kernel - set `memory.min`.

hedora 1y ago

Is linux considered stable with swap completely disabled these days? In the past, they recommended against that for deadlock reasons. On the workloads I care about (desktop, and servers that avoid mmap), the anonymous dirty page part of the kernel heap is < 10% of RAM, so swap is mostly just there to waste space and slow down the oomkiller.

egberts1 1y ago

It is still problematic to get your swap space encrypted-by-default (usual way is via LVM); still doable, workable, but jumpy. The Linux Kernel still have this global lock for swapiness. Downside is no smooth playing video.