9 ms·
We're moving away from swap partitions on our Linux servers
- mcculley 4y agoThere should be a set of VFS hooks so that a filesystem can offer up pages of swap to the kernel without an explicit swap file. I remember tinkering with this around 2000, but I didn’t get very far. I would love to see this explored.
- kevincox 4y agoI'd love to see this. There appear to be some programs to monitor usage and dynamically add more swap but they all appear buggy and unmaintained. If course if you need more than a bit of swap there is something suspicious about your system but I have often found this extra RAM is useful. For example I often have many open files in my editor, great candidates to swap out the ones that I am not currently editing. Or /tmp in RAM, I don't need it persisted but if the RAM could be better used for other things feel free to swap the unused files out.
- transfire 4y agoSwap on zram seems very interesting. My current home computer is a NUC with 16GiB, but I sometimes build from source and a tmpfs drive is helpful for that so I’m a little concerned I might not have quite enough RAM. Thoughts? I’ll definitely double my RAM for my next system (in a year or two) and will definitely have to consider swap on zram then.
- nottorp 4y agoI wonder if swap on zram is similar to what Mac OS calls "compressed memory".
- dtgriscom 4y agoThat's my understanding.
- 95014_refugee 4y agoNo. ZRAM is a compressed ramdisk. Darwin “compressed memory” is an intermediate state for pages as they age out of the working set, but before they are written out to a swapfile. Incidentally, Darwin creates, defragments and reclaims swapfiles dynamically. All this complaining about swap partition / file sizes is kind of … silly. None of that code is rocket science.
- WirelessGigabit 4y agoI wouldn’t do less than 64GB. I’m running 32 on my current NUC as my main machine with Windows 11 and WSL and a RAM drive. With current source sizes and Electron apps consuming RAM, just 32 GB is no longer a nice to have.
- mixmastamyk 4y agoI do fine on 16gb, with loaded browser and 4gb vm of Windows running.
- WirelessGigabit 4y agoGB*. I don’t. Docker containers and IDEs all over. I also don’t close stuff.
- mixmastamyk 4y agoContainers use much less ram than vms. One can always use and write wasteful programs but it is a choice, rather than necessity as folks often portray.
- jakogut 4y ago> I sometimes build from source and a tmpfs drive is helpful for that so I’m a little concerned I might not have quite enough RAM. I used to do this as well, but consider that the VFS cache basically does the same thing. The main difference is that cached data can be dropped or flushed if necessary to free memory, whereas this can't be done automatically with tmpfs. Benchmark both cases and I'll bet you'd be surprised at how little difference there is.
- kevincox 4y agoThe nice thing about swap here is that the kernel is under no obligation to sync the data to disk "soon". I have lots of tmp files that are written and read quickly and never hit the disk. With tmpfs+swap the kernel is free to swap out unused files if that memory is better used elsewhere, but otherwise won't bother touching the disk. I do a lot of builds on my system and most are small enough to entirely live in RAM. But when you build Firefox it is best to start flushing some data to disk.
- jakogut 4y agoThat's another great point. With a large build in tmpfs, you can either run into the size limit of the tmpfs, breaking the build and wasting time, or run out of memory and end up with an unresponsive machine. Entrusting this to the VFS cache has better predictability, and nearly the same performance in many cases.
- opencl 4y agoNot having quite enough RAM is the exact scenario when swap on ZRAM is most useful. Assuming the data you have in RAM is compressible it lets you use a few extra gigabytes before hitting disk swap or OOM (depending on if you have a disk swap device configured in addition to ZRAM). Zswap is somewhat similar and might also work well for you, it uses disk swap but with a compressed cache in RAM.
- Proven 4y agoUh, I moved to swap files 10 years ago and more recently to no swap. I didn't find it necessary to blog about it.
- hinkley 4y agoThe bigger question is why now? I thought swap partitions were down to almost nothing about ten years ago, because the overhead was too high? I guess NVMe gave it a second wind?
- mixmastamyk 4y agoBecause recent distributions have enabled it by default. I hadn’t used swap for a decade or so since acquiring 16gb ram, but one day noticed a new swapfile in my rootfs.
- indigodaddy 4y agoWhich distros can you think of offhand? I wasn’t aware of this.
- mixmastamyk 4y agoUbuntus. Fedora is now using zswap.
- jmbwell 4y agoIt seems reasonable that a university would take a moment to share their methods and findings so that others who might consider doing the same thing can benefit from the experience of others.
- Proven 4y ago
- mannyv 4y agoRemoving swap is something most people do in production because performance matters. It's been this way for 20 years. Was that not normal?
- bityard 4y agoSwap actually increases system performance for many common workloads: https://chrisdown.name/2018/01/02/in-defence-of-swap.html https://chrisdown.name/2018/01/02/in-defence-of-swap.html
- morelisp 4y agoThe basic argument is that swap can be preferrable to hard eviction for managing disk cache, which can be true, but disk-heavy workloads are less common than ever especially in general-use clusters. (The other argument is the usual mix of "but it can swap out unused areas" in which case you should fix the program to not allocate a ton of memory it doesn't need; and "but the OOM killer will come" and yes that's the entire point please move my process elsewhere).
- shawnz 4y ago> "but it can swap out unused areas" in which case you should fix the program to not allocate a ton of memory it doesn't need Isn't this basically asking every application to independently reimplement their own swap-style functionality? Surely it makes more sense to have swap as a system level feature that every application can take advantage of? That way the system has more information available to make swapping decisions since it can take into account the memory usage of all applications and drivers/etc together, and applications can take advantage of the most advanced swapping algorithms in the latest versions of the system without having to be individually updated.
- morelisp 4y ago> Isn't this basically asking every application to independently reimplement their own swap-style functionality? No, it's asking for applications not to allocate a bunch of space they don't need in the first place.
- nixcraft 4y agoThe main problem is my little VM powered by CentOS or Amazon Linux 2 (which is based upon Fedora Linux) runs out of memory when I run "yum update". I have 1 core CPU with 512MB ram. The BaseOS repo from RHEL and co is just too big. It requires a minimum of 3 GB RAM. So work around is to add swap to AWS EC2/Lightsail instance. I believe there is a dnf bug open in RHN, but no progress was made so far https://bugzilla.redhat.com/show_bug.cgi?id=2040170 https://bugzilla.redhat.com/show_bug.cgi?id=2040170 Adding a 4/8GB swap space file solved my problem with the `yum update`. That is just one example. There are many users with few resources.
- 2Gkashmiri 4y agothere are vps that are being sold for peants. i have a 128mb ram NAT vps... bought it out of curiosity and i concur. i am unable to update that thing because of the out of memory thing
- indigodaddy 4y agoChange it to Debian or Alpine? Any VM under 512M I’d never deploy as RH-based.
- numpad0 4y ago“128MB NAT VPS” thing is OpenVZ or otherwise container based. $5 per YEAR for a container, 128MB RAM, 1GB persistent storage, SolusVM user panel, and 123.45.67.89:16300 - 16320 NATd to your enp0s3. swapon/swapoff don’t work. I tried installing QEMU, didn’t work either because storage is I/O rate limited. IIRC, OpenVZ has some sort of unified cache control for RAM and storage I/O, and trying to circumvent quota through “disk” don’t work because of it. I’m not sure what the use case might be, legal or not, but there seems to be dozen or so operators taking our credit cards in the exact same manner. Maybe the operation itself is a money laundering, or maybe it’s to run elaborate offsite contraptions for illegal activities, or to run SEO fake sites? I don’t know.
- indigodaddy 4y ago
- ForOldHack 4y agoAny swap issues I had, I just added a scratch monkey. I just tried this out. Amazing. The swap file compresses nicely. two machines, 4GB and a 16Gb.
- dschuetz 4y agoSwap or no swap: https://haydenjames.io/linux-performance-almost-always-add-swap-space/ https://haydenjames.io/linux-performance-almost-always-add-s... It's the use cases that make a difference.
- bloaf 4y agoThat is an article which 1. Makes claims about performance 2. Provides no benchmarks The only concrete answer to "why have swap if I have enough RAM" is "well, if you run out of RAM..."
- lazide 4y agoIf you use swap because you don’t have enough RAM for things you need in RAM, you’re going to have a very bad time. The only time swap isn’t just delaying (and growing) the inevitable shitshow is when, for some reason: 1) some program is consuming some significant amount of Ram, but doesn’t need to run for awhile, and it’s somehow faster/easier to let it swap to disk than shutting it down and restarting it while you do something else that consumes Ram. 2) something you’re running has some amount of bloat or leak that you know will never get exercised, but also never will grow larger than your swap file and take your system out entirely. Both of those situations are not only rare, but easy to misjudge and end up wedging your system.
- leni536 4y agoI found that gcc works mostly alright on swap for compiling C++. I assume that data structures for template instantiations are never really deallocated, and it often falls into category 2 of your deacription. And then compilation eventually finishes and the allocated memory is released on process termination.
- guipsp 4y agoThere is a large class of programs that may allocate large amounts of memory, but only work on a small set at once
- VLM 4y agoI always felt nervous about resizing a filesystem that has a swap file on it. Never had a kernel panic but would swapoff the file just in case. If I created a swap partition in LVM I can force it and move it around to the best location WRT multiple hard drives and possible mix of HDD and SSD. With a file its a little more abstract and I could emulate it with some pointless abstraction, but why bother. A typical "this century" example would be running my boot and swap on local disk and storing all real world data on the NAS over iSCSI. Obviously now that "Everything" is virtualized this kind of system administration is void unless you're admin on the cluster itself LOL. The problem with swap is memory seems to have gotten cheaper than storage and swap of any form takes storage space. Its not the capex that's the problem, its the opex of now you've got an extra 32GB that "has to" be backed up and "has to" be virus scanned or whatever security theatre, and that swap file "has to" be treated as the highest security PII HIPPA PCI category because who knows whats been swapped out of memory onto it, it could be chock full of CC numbers how do you know for sure unless its empty or not there? Paying for more memory doesn't increase your backup storage / security risk / virus scan times and the latter costs more than the former. I don't think non-cloudy admins realize there are cloudy admins out there with forests of little systems with like 8 GB of ram and 16 GB of storage, so the "old" rules from the 1980s about provision twice your ram as swap would increase the disk required from 16 GB to 32 GB doubling my costs for, essentially, nothing. Another problem with swap is in the old days if you needed more memory than you have physically you had to buy more memory, then for awhile you could emulate memory very slowly with disk but much faster than calling an IBM CE and buying more memory, now if you need more memory you spin up more instances or rebuild the instance on a larger flavor about as fast as how you'd add swap in the olden days, probably faster and it can be done dynamically in some architectures.
- tatref 4y agoDo you also feel nervous when resizing a FS that has files in use? I've never thought about the AV scan of such a file!
- CoolCold 4y agoFor growing FS, with regular files, I'm not feeling nervous. I was at the beginning like 15 years ago, but both XFS/Ext4/BTRFS works fine. As side note, not nervous when resizing NTFS in Windows, even for shrinking (still recommend to have backup for shrinking though).
- danans 4y agoMany of the commenters here seem to think this is about moving away from swap altogether, but according to the article this is switching from a swap partition to a swap file, so still using swap.
- deleted 4y ago[deleted]
- password4321 4y agoUh-oh, somebody read the article! Thanks for the heads-up.
- rubyist5eva 4y agoBeen using swap files for years and most recently started turning on zram and it’s been smooth sailing.
- bxparks 4y agoI read somewhere that Linux hibernation requires swap (1-1.5X RAM?). So I have been configuring my laptops with a 1.5X swap partition. Unfortunately Linux hibernation has not worked for me since the early 2000's, across 10-15 different laptops. Apparently hibernation can use swapfiles, but I've never been able to verify that.
- OJFord 4y agoI use a swapfile, and hibernate frequently (after iirc 2h of suspend, so sometimes multiple times a day). Does take a bit of setup (by far the most supportive of people arguing against Linux in favour of macOS/Windows thing I've done) but works well now it is.
- Jnr 4y ago10+ years ago I also had problems getting hibernation to work. Then I read the Archlinux wiki on that topic and haven't had problems since. It is working fine on Linux, you just have to set it up correctly.
- somehnguy 4y agoIf it’s seemingly as easy as reading a wiki and configuring a few things I’m curious why it doesn’t just work by default. Does it just not get enough dev love?
- billyzs 4y agoMy personal experience is that the situation has improved somewhat but the hardware still plays a large part. A Kaby Lake laptop I had setup hibernation to encrypted swap (not file but partition iirc) worked well enough besides a few times when some peripherals e.g. wifi failed to power on after resuming from hibernation; A Zen3+ laptop from the same manufacturer that I got last week, running Arch (with the latest kernel and packages) _always_ had problem resuming, wifi (and who knows what else) never gets turned back on, the system sometimes enter into a weird state showing just a blank screen, not accepting any keypresses, or otherwise get stuck in some stage in the boot process with no output so it's hard to debug. So far it's been a real crapshoot.
- unpopularopp 4y agoI'm more surprised that they are using Ubuntu. Just feels there seems to be so many compromise nowadays compared to for example Debian or Arch
- stjohnswarts 4y agoThere is absolutely no reason to use those over Ubuntu and many reasons not to use Arch in your critical server infrastructure, but Debian v. Ubuntu would be essentially half a dozen of these or 6 of those.
- skywhopper 4y agoUbuntu’s primary benefit over Other distributions is that the strict release cycle allows for efficient planning of upgrade cycles. Pretty much anything specific to the local workloads I wouldn’t use the distro packages for anyway, so the distro doesn’t matter so much.
- OJFord 4y agoI stopped using earlyoom because I got frustrated with it always seeming to be exactly the wrong thing for a given moment that would be killed. Now though OOM events seem to turn into situations that require me to force a reboot, because kswapd goes crazy, using all available CPU, and I can barely (if at all) register any keyboard/mouse input. Seems like a bad reason to disable swap, but I can't find a way to do what I really want, which is simply to reserve some CPU for input. And then I suppose in order to do something useful with it, require that any process that wants it be allowed at least some small allocation. Maybe that's the hard part and why I haven't found a way? It doesn't happen often, it's just really annoying when it does.
- __turbobrew__ 4y agoIn theory this should be possible with cgroups. The mechanism is there but I don’t know of a way to easily set up a policy that does what you want. It should be possible to allocate say 95% of the system resources to the default cgroup and then you could create a secondary cgroup — recovery — which has access to the last 5% of the system resources which you could use to run commands such as “kill” or “top” to recover the system state. Additionally you could run a second ssh server in the recovery cgroup which you could ssh into in the case of system lock up. In reality it is probably easier to just reboot in most cases, or if you are dealing with servers use ipmi.
- jaggirs 4y agoI don't understand why this issue still persists on linux. As far as I can tell all earlyoom needs to do is kill the process that has eaten the most memory in the last minute or so. On windows this issue is non-existent.
- viraptor 4y agoIt's not that simple. Imagine something leaking memory running on parallel with something bursty. For example your browser leaks, but you run a big grep|sort in the background. Or have some GC runtime which allocates in batches and just decided it needs another chunk to manage. On Windows you don't have oom at all because it trades that solution for just swapping forever until either you can't do anything or manage to kill the right app yourself.
- bumblebritches5 4y ago
- wongarsu 4y agoWindows has been fine with swap files since the age of spinning disks, so I guess using a swap file instead of a swap partition is fine. Speaking of Windows: How is it that Linux seems to grind to a halt when it is forced to swap, while Windows works just fine in the same scenario? Sure, on Windows a lot of swap file "use" is just bookkeeping of empty pages because Windows doesn't overcommit memory. But even if it actually swaps it stays responsive, while I've had to reboot linux boxes more than once because they stayed unresponsive after memory pressure. Does Windows have a smarter swap algorithm? Is it more proactive about swapping stuff back in once memory pressure is gone? A better scheduler that gives more precedence to "interactive" things? And most importantly: can linux implement those things too?
- hulitu 4y ago> How is it that Linux seems to grind to a halt when it is forced to swap /proc/sys/vm/swapiness or something like this. AFAIK shall be bigger than 50% to not see the behaviour that you describe.
- Manu40 4y agoAh yes, swapiness. I think Windows is set to something like ... 65% or so. Or was it 85%. I forget, but I looked into this a long time ago when I was having to suffer through having about half as much ram as was ideal for my use case scenarios on both windows and linux. I remember coming across some method to get it changed on windows as well, but I forget that method too. Or rather, I think it's some reg edit that needs doing, but I forget what/where.
- MrStonedOne 4y ago
- simoncion 4y agoBased on the explanation in the documentation <https://docs.kernel.org/admin-guide/sysctl/vm.html?highlight=swappiness#swappiness https://docs.kernel.org/admin-guide/sysctl/vm.html?highlight...>, I set 'swappiness' to 100. I figure that SSDs are fine with random IO, so swap isn't likely to be much -if at all- slower than pulling pages back in from disk.
- netik 4y agoEvery commercial job I’ve had has had swap set to zero. We’d rather run out of memory than swap.
- hulitu 4y agoI bet it was never boring.
- nomel 4y agoIn every commercial setting I’ve been involved with, something has gone very wrong if you’re swapping, since you spec the system for the use case, rather than a fraction of it. You’ll still have the logs to debug whatever went wrong. What problems do you see?
- Arnavion 4y agoYes, exactly. If your machine is starting to use swap, then it's already running away on memory that you hadn't planned for. The swap just delayed the inevitable OOM. You might as well let it OOM immediately.
- VectorLock 4y agoFail fast is in most situations better than swap hell.
- whartung 4y agoIndeed. As a Java shop, the last thing you want to do is swap. Garbage Collection and Swap tend to be very bad cohabitants.
- kevincox 4y agoYeah, no swap probtnakes sense for GCed languages. If you are going to touch all of the memory frequently it doesn't make sense to try to swap it out. You are better off forcing Linux to swap out filesystem cache which at least has a chance of not being used in the next minute. The only real thing that you may be missing is SSH being swapped out and some other system processes but those may be 2MiB altogether. Maybe an option would be disabling swap just for the JVM. IDK if that is something you can do with cgroups. But if 99% of your anonymous RAM is pinned anyways there is little to gain.
- iforgotpassword 4y agoI tried swap on zram briefly, but then switched back to a swap partition and enabling zswap. It's basically like zram but when the reserved ram section runs full it will hit the disk with the lru pages. A tiered system. That means as long a you're only slightly using too much ram, it will swap in compressed RAM entirely. So the only use case I see for swap in zram is if you never want to swap to disk, but still want to use swap for some reason.
- jmount 4y agoJust about the only think I use swap for on Linux, is not bombing out during the fork/exec (even for something like a system call) from large processes. Back when I last looked, it appeared they need enough backable virtual address space, even when they don't use it. The filesystem swap seems good enough for that.
- kevincox 4y agoI don't think this is the default configuration but it is an option to avoid overcommit.
- xani_ 4y agoEh, just use LVM. I have no idea why it isn't a standard on some distros. Then you can mix, match and resize at will, and as a bonus you can migrate system live from one device to another without rebooting. > (If you're willing to use swap files anyway you can always add more swap space with a swap file even if you started with a swap partition. However, if your swap partition is too big, shrinking it or limiting how much of it gets used is more annoying.) You can't shrink swap file without swapoff & swapon anyway so it's illusion of an improvement
- oneplane 4y agoRegarding swap partitions: on virtualised systems (like clouds) this is rather easy since you can just edit EBS/LUN/whatever on the provider side, grow, shrink, add completely new ones without doing any of the classical disk management tasks. On cloud machines, LVM doesn't make a lot of sense, and neither does lots of partitions. Partitions are from a time where you have one disk and want to setup filesystem boundaries. But in a virtualised setup you can slice block storage any way you want, and present it to the VM as completely separate disks. It's a bit like having LVM "on the outside" of the VM. This way, the VM itself becomes simpler and has to care less about the specifics of the disk. If you are on physical hardware, I'd argue that if you're not using LVM or ZFS, you're probably doing it wrong. Only really small scale (SD card, eMMC, USB SSD, raspberry PI type of stuff) works better without it. That said, even LVM wouldn't be too bad in such cases. (You'd have to get down to JFFS2 and the likes to really not use LVM)
- koala_man 4y agoI was surprised to learn that a swap file on hdd is not nearly as fast as a dedicated swap partition at the start of the drive. https://www.vidarholen.net/contents/blog/?p=1110 https://www.vidarholen.net/contents/blog/?p=1110
- fuzzfactor 4y agoFrom the 1990's there was a perceptible difference in Linux or Windows if you could dedicate one entire secondary HDD to a swap partition. At the beginning of the HDD was where the sectors are most rapidly accessed physically. For SSD there should not be much difference between a separate partition or a file, or whether the reserved sectors the swap occupies are at the beginning of the drive. Should probably be aligned with the block size of the SSD though. Still may be helpful to have a separate SSD for swap besides the drive the OS is on. It would be worth trying but I like a swap file in Linux so I can confine Linux to a single type EXT4 partition right next to my NTFS WIndows partition. With Windows it does work good without a swap file as long as you have plenty of memory left over after Windows and your desired apps are loaded. Handling further data loads within reason.
- ilovecaching 4y agoMy coworker at Meta and kernel memory management expert Chris Down already wrote about why you need swap: https://chrisdown.name/2018/01/02/in-defence-of-swap.html https://chrisdown.name/2018/01/02/in-defence-of-swap.html
- hk1337 4y agoThis seems like it's becoming the norm as we move to SSDs
- phendrenad2 4y ago> Swap files appear to work fine, including on a mirrored root filesystem, and I've read that they're basically just as efficient as swap partitions these days I've been saying this for awhile, and I always get berated by angry linux nerds who didn't do any research. The times they are a changin'?
- lamontcg 4y agoWe were using swapfiles on Linux at Amazon in 2001 just fine. No idea if someone perf tested it, but we did have pretty decent perf testing (and we did adjust swappiness based on perf testing).
- tinus_hn 4y agoWhatever, just make sure you don’t move to swap files on ZFS because it doesn’t work properly.