10 ms·

Linux Performance: Why You Should Almost Always Add Swap Space

46 points by ashitlerferad 9y ago

Kenji 9y ago
I'm sorry, but not a single reason was presented in favour of swap when you're 100% confident that you will not max out your RAM. Swap = completely useless slowdown if you have total control over your system. Better use ulimit to prevent processes with e.g. memory leaks from causing havoc.
- __david__ 9y ago
  They presented one reason: the system will swap out more or less unused memory and give that much more space for disk cache, etc. Now, I'm not sure I find this a particularly compelling reason. I suppose it depends on how much RAM you have available for cache vs how much unused junk gets swapped out. If you've got 5G of unused RAM and 300M extra of stale allocations that get swapped out, then it's probably not going to impact your system in any measurable way. I personally like a quick OOM, myself, rather than a machine slowing down to a crawl for multiple minutes while swap gets exhausted.
- subway 9y ago
  RAM freed by swapping some infrequently accessed memory out to disk becomes available for cache use. If you have total control of your system, you know when you can safely allow data to be paged, and when you need to lock memory to physical ram.
- ChrisSD 9y ago
  Can you ever be 100% confident?
  - notyourday 9y ago
    Yes. ulimit and cgroups do wonders.
  - warrenm 9y ago
    You can 100% be confident (except in a few very unusual use cases) that swap is important, yes. https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
- efaref 9y ago
  The most compelling reason to enable at least some swap space is that the Linux kernel's memory handling algorithms can't cope with there being no swap. If you have no swap, then you will encounter stalls and OOMs even if you don't actually run out of memory. Adding even a tiny amount of swap prevents this from happening.
  - albertzeyer 9y ago
    This sounds as if it is a serious issue and should be fixed. Is there a bug report about it?
    - lolc 9y ago
      This is due to memory overcommit, it's a performance feature. You can turn it off.
      - kevin_nisbet 9y ago
        Interesting, I sort of remember an interaction with NUMA as well. I don't believe this is a bug, just a miss-understanding of the way the underlying system works. From what I remember when swapiness is 0, there are lots of reports of processes that use more than a single NUMA node worth of memory getting killed by the OOM killer, even with plenty of free RAM available. I unfortunately don't remember the details, but this prevented the memory from being able to be allocated to the additional node. I've heard of this mostly with mysql, where it's common to have a big server with lots of RAM, but a single large process that uses most of the system RAM. The way we got around this, was by setting the process to allocate interleaved among the NUMA nodes. I'll have to dig into the overcommit.
    - warrenm 9y ago
      It's not a serious issue at all. It's intentional.
  - bechampion 9y ago
    i agree with this , lack of swap makes oom too aggressive . , question what's worst no app (and react to OOM) or app swapping
- warrenm 9y ago
  Want reasoning? Here's a whole huge list: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447 When it is the case in the overwhelming majority of cases that you should use swap, you should just use it.
martin_andrino 9y ago
Swap isn’t needed at all in this age. It’s just there for people who can’t afford more RAM and therefore have to resort to hacky solutions like this.
- geofft 9y ago
  The article specifically lists reasons why you should enable swap when there is plenty of RAM. Can you elaborate on what part of the article's reasoning you disagree with?
- tedeh 9y ago
  Disk is still way cheaper than RAM, and I always add some swap space to my servers just to lessen the chance of getting ENOMEM errors if/when an unexpected or unusual workload would come along.
- majewsky 9y ago
  > It’s just there for people who can’t afford more RAM So I should throw away my 5-year-old notebook that's still working perfectly fine just because the RAM is soldered in? Doesn't sound like efficient resource usage (both money and rare earths).
- bogomipz 9y ago
  What an absurd statement. Do you use a laptop? I do and I'm using swap space right now. I would like to buy a laptop with 32 or 64 Gigs of RAM but I can't as most(all?) laptop makers don't ship with a memory controller that permits more than 16 Gigs of RAM. And I can afford to buy more RAM.
  - cztomsik 9y ago
    What is the rule for setting swap size, or can I use just swap file just like on win/mac? (I heard it will disable hibernation then)
    - warrenm 9y ago
      The basic rule is `RAM + 2GB` I usually peak at 34GB, unless I have reason (like needing/wanting to enable full hibernation) to use 66GB, 130GB, etc.
djb_hackernews 9y ago
What do the people that use cloud instances that don't have local disks do? You definitely don't want swap to be on EBS...
- sharms 9y ago
  Its a good question, but the swap advice is suspicious. If I would normally purchase a 4GB of ram instance with 4GB swap, then what happens when I purchase a 8GB instance? Do I still need 4GB swap? Hopefully this helps illustrate that it is really the applications you intend to run and the swap doesn't mean the system gets more efficient
- LinuxBender 9y ago
  Allocate 20+% more memory that you could ever need. Use cgroups, lxc, systemd to constrain applications and containers to specific amounts of memory, cpu, etc. Properly engineered systems will have enough memory for everything else under the hood beyond the application. VM and container abstraction does not negate the requirement to calculate this.
bryanlarsen 9y ago
Kubernetes forces you to start it with flag to tell it "I know what I'm doing, I know this machine has swap, but start Kubernetes anyways even though it's normally a bad idea to run Kubernetes on a machine with swap".
hannob 9y ago
I wondered this for a while, unfortunately a major question I have remains unanswered in the text: What about SSDs and SSD-only systems? (Like... any average laptop.) Should I be worried that putting a swap on an SSD will cause it to wear out fast due to many write cycles?
- ktpsns 9y ago
  Same here. How does file swap (cf. https://wiki.archlinux.org/index.php/swap#Swap_file_creation https://wiki.archlinux.org/index.php/swap#Swap_file_creation) could help out to let an underlying file system distribute the usage of the ssd? Combined with recreating a swap file on every boot (say on a frequently booting desktop machine), this could make a difference.
- INTPenis 9y ago
  I set vm.swappiness to 0 on my Fedora laptop. Meaning the kernel will only swap to save the system. I've been doing this for years and had no issues. Edit: Actually in recent kernels, which I am using, 0 means it's disabled. I was thinking of 1.
  - AstralStorm 9y ago
    It does not help, just makes the problem appear later in a more dire way. Try overcommit settings instead.
    - INTPenis 9y ago
      I googled that and was pleasantly surprised to learn something new.[1] But if I'm reading that right I would probably want to do both. Because overcommit settings seem to be a memory saving technique. To avoid needing swap by decreasing the allowed overcommit memory. Anyways, in the many years that I've been changing vm.swappiness I've never had any issues and I've always had a minimum of 8G RAM so I don't think this will be a big issue. 1. http://engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/ http://engineering.pivotal.io/post/virtual_memory_settings_i...
- 9y ago
dennisjac 9y ago
I always see people caring way to much about the amount of swap space used and not enough about the swap activity going on. It isn't the amount of swap space that slows a system down but actual swapping in/out of memory pages. In vmstat you want to pay attention to the "si" and "so" columns and in your alerting/graphing you want to keep track of the values "pswpin" and "pswpout" in /proc/vmstat. If these are almost always exactly or near zero then that means the fact that some memory pages are swapped out has virtually no impact on the performance of your system. There are two other important issues to take into account though. 1) Even if the memory pages swapped out are not accessed generally they might be forces back into memory because of some specific action. One example is a database that has all the hot records in memory but other records that are accessed only very rarely swapped out to disk. In general everything will perform fine in this situation but the moment you do e.g. a table scan and a lot of these records need to be moved back into memory you might see a disk I/O peak that might be quite a kick on the neck for overall performance if the database is really busy. 2) If you don't have any swap space configured you might still run into problems with swap which seem to be caused by a bug in the memory handling in the kernel. I've seen this on some KVM hypervisors which were CentOS 7 systems. These systems were equipped with 128G of RAM and had two virtual machines running which both were configured with 32G of virtual RAM. They ran fine until one day the kswapd kernel process ran with 100% cpu usage even though no swap swap was configured whatsoever (to avoid the situation mentioned above). The "fix" was to dump the systems caches with "echo 3 > /proc/sys/vm/drop_caches" which seemed to calm down kswapd again. As best as I can tell what happened is that the system used all the free ram for the page cache and buffers and when the system needed some memory it apparently prefered to swap pages out to disk (even though no swap was configured) rather than reclaiming page cache of which there was plenty to reclaim. Unfortunately that means there seems to be no bulletproof way to say "only use physical ram and never try to swap anything out to disk". Even /proc/sys/vm/swappiness can be dangerous as a value of "0" doesn't actually tell the system to only swap if absolutely necessary but can lead to OOM situations even if swap space is still available (see https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=fe35004fbf9eaf67482b074a2e032abb9c89b1dd https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux... for details). TL;DR: 1) Don't just pay attention to the amount of swap space but to actual swap activity over time 2) Be aware of corner cases and bugs relating to swap
mcculley 9y ago
When running an instance where the biggest process is a garbage collected runtime, one does not want any swap at all. If the garbage collector is forced to walk through all of the pages in swap hunting for live references, performance is terrible. It is better then to have the process see only as much virtual memory as there is RAM in the system. It is a failure of Unix/Linux that there is no sensible API for managing real RAM from the point of view of the process.
LinuxBender 9y ago
If you must use swap (you probably don't need to), then at least set zswap.enabled=1 in the kernel boot options, typically in grub. This will enable lzo compression of swap in memory so there is less writing to disk. Some newer kernels use lz4. Either way, what most folks are actually missing is the correct kernel settings for the amount of memory they have. Sadly, the kernel does not dynamically adjust these based on your total amount of memory. Below is based loosely on Redhat suggestions. Please note, that if you fall below the min free, the kernel will decide what to do next, based on your oom and panic settings. The settings below will have the kernel free cache and other memory earlier so that you do not hit those stalling conditions and can even prevent some OOM race conditions. Some might suggest tuned, but use tuned with caution or at least read up on everything it does. MEM=`grep ^MemTotal /proc/meminfo | awk {'print $2'}` if [ ${MEM} -gt 1129241478 ] ; then sysctl -q -w vm.min_free_kbytes=16384000 elif [ ${MEM} -gt 564620739 ] ; then sysctl -q -w vm.min_free_kbytes=8192000 elif [ ${MEM} -gt 352887962 ] ; then sysctl -q -w vm.min_free_kbytes=4096000 elif [ ${MEM} -gt 176443981 ] ; then sysctl -q -w vm.min_free_kbytes=1024000 elif [ ${MEM} -gt 88221990 ] ; then sysctl -q -w vm.min_free_kbytes=524288 else sysctl -q -w vm.min_free_kbytes=262144 fi If you have small VM's, then perhaps set the default above to something a little smaller. You can of course free up about 128MB on default installations by removing "crashkernel" from your grub config and rebooting. Then do this regardless, because overcommit set to 0 does not mean off, so we set the ration to 0 as well. Overcommit is good for developers testing code and finding the correct ways to manage memory in their applications during development. sysctl -q -w vm.overcommit_ratio=0 And of course, cache pressure plays into early evacuation of the right cache based on your usage: ## default is 100 (optimal for file servers). 4000+ for in memory databases. ## 10000 means always prefer page cache. sysctl -q -w vm.vfs_cache_pressure=1000 And if you have people oversubscribing a lot: (adjust based on your memory capacity) sysctl -q -w vm.admin_reserve_kbytes=131072 sysctl -q -w vm.user_reserve_kbytes=262144 Please do read up on all of these before testing on your test machines. [1] [1] https://www.kernel.org/doc/Documentation/sysctl/ https://www.kernel.org/doc/Documentation/sysctl/ Then finally, make sure you have Transparent Huge Pages disabled unless you know for sure you need it. THP can leak a lot of memory and it is nearly impossible to see without extensive kernel debugging. In grub, set this and reboot: transparent_hugepage=madvise Or to manually disable THP during run-time, echo -n "madvise" > /sys/kernel/mm/transparent_hugepage/enabled echo -n "never" > /sys/kernel/mm/transparent_hugepage/defrag Then restart your applications. THP defrag can also cause stalling and lag spikes, especially in large memory java deployments (it will look like FGC's) and in MongoDB, Cassandra, others. If you did this manually, stop your apps, flush cache, compact memory, then start your apps. sync;sync;sync echo 3 > /proc/sys/vm/drop_caches echo 1 > /proc/sys/vm/compact_memory Some will say 3 sync's is not required. This is mostly true, but some old raid controllers treat this differently.
kevin_nisbet 9y ago
In my experience, I think engineering swap usage is more complex than this article leads on. While adding more swap may make sense for many systems, I've personally worked on several systems where disabling swap made a huge difference. It's just one of those things where there isn't a hard rule. One of the issues I've encountered more than once with Java software, is what appears to be longer than expected pauses in the GC, that I've linked to swap usage. The theory is, if you have a reference that lives long enough to get promoted to an old generation, but then is released, the memory in question won't be used for quite some time. This can lead the kernel to detect the memory pages not being used, swap them to disk, but then when it's time to do GC, the process is just frozen while the GC is running and get's backed up. Depending on the workload this might be fine, however, on both the platforms I encountered this issue on, were real time applications which were engineered to have available memory for the application and did (and also had no benefit from using the free'd memory for IO cache). So in this case, disabling swap had a real and measurable performance boost for us, getting rid of some of the variability in our GC runtime and pauses. Under low memory conditions I think is also subjective. So if I'm a distributed / highly available system that's doing message routing in a telco environment, I would rather have the OOM killer kill the process, and failover to a backup, then for the system to be up but unresponsive because it's spending a majority of it's time swapping too/from disk. Again, this totally depends on the platform.
nisa 9y ago
This is dangerous advice. At least on Ubuntu 16.04 with stock kernel running due to memory pressure into swap kills the systems. Basically everything times out and you can only hard-reboot. Without swap you have an OOM at least something you can recover from. If you need swap just add a little like 1-4gb - because if you have spinning disks 64gb swap won't help you at all because every request to swap is magnitudes more slower. Might be different with NVMe but it's still noticable.
- ausjke 9y ago
  indeed on ubuntu 16.04 64bit, when my chrome has 20+ tabs, some new tab could trigger a memory thrashing, that uses up all the memory and eats up a few GB disk, cpu load jumps up to 10x, the computer basically becomes unusable. i would rather a OOM kill the chrome process in this case, it wastes less time on me, a re-open of chrome will keep all the histories anyways. also on embedded system where the RAM is fairly limited, and no hard drive attached, and its storage is measured by MB instead of GB, you simply can not have the swap partition. I want to run the system without swap actually.
  - warrenm 9y ago
    >when my chrome has 20+ tabs That's not a swap problem, that's a Chrome problem.
    - AstralStorm 9y ago
      It is a swap problem. Swap is supposed to make things work and not stop. I suppose this is actually a problem of Linux optimistic malloc policy.
- tfha 9y ago
  Not just ubuntu. We tell our engineers to remove their swap partitions because sometimes the test suite will consume all remaining memory and push into swap. If that happens, pretty much the only thing you can do is a hard poweroff. If you're fast, you can Ctrl+C and only lose a minute or two, but often you're just completely stuck. Much better to OOM and have the test suite killed early.
mrweasel 9y ago
I'm not going to disagree with the article, because I haven't done any testing myself that would suggest that the point made are wrong. I do however find it amusing to login to a system with 32 or 64GB of memory with a 2GB swap. It seems humorous that those 2GB should somehow do anything if you already used 64GB of RAM.
- warrenm 9y ago
  I can save you a bunch of testing time. Just enable swap (I typically peak at 34GB, but follow the `RAM + 2GB` rule-of-thumb before that Here's a bunch of data that helps understand why swap is important: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
jacquesm 9y ago
Cargo cult system administration. No, you should not 'almost always add swap space'. What you should do instead is tune your system for its intended use. If you need swap as an 'early warning system' that you're about to run out of memory you're already doing it wrong and the OOM killer is a piece of code that has default settings that can be tuned, ditto for the virtual memory manager in the kernel. http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html http://www.oracle.com/technetwork/articles/servers-storage-d... https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-captun https://access.redhat.com/documentation/en-us/red_hat_enterp... Select appropriate equivalent for your kernel where applicable, and don't forget about the per-process limits in ulimit. This sort of 'almost useless' advice is pretty annoying, it shows that the writer has an interest in the matter but didn't bother to dig in deep enough to make the advice truly useful. Also, the comment on that article that implies that swap will allow you to update a running process because it is backing the image is wrong, the backing is simply done through the filesystem and swap space has nothing to do with it. The practical upshot of which is that you're not going to be able to reclaim the space held by the binary if you should remove it because the 'file is busy' until the last instance of that process exits. In the meantime, if you really want to run the newer version under the old name or path then you're free to unlink it or 'mv' it to a different location and to start the new binary from the default location. On some Unixes you can even overwrite the binary using an mv command making it look like the old one has disappeared but again, under the hood it will still be held open, you can check this using lsof or whatever is your local equivalent to see that the file is indeed still present. In a pinch (and you really really have messed up if you ever need this trick) you can re-link the running image by hard linking to the inode.
- warrenm 9y ago
  You should just about always add at least some swap. Anyone who thinks otherwise hasn't read much or seen very many systems. start here: - https://www.kernel.org/doc/gorman/html/understand/understand013.html https://www.kernel.org/doc/gorman/html/understand/understand... - https://www.kernel.org/doc/gorman/html/understand/understand014.html https://www.kernel.org/doc/gorman/html/understand/understand... - https://www.linuxquestions.org/questions/linux-kernel-70/why-is-my-linux-server-using-swap-when-inactive-memory-is-available-4175580818 https://www.linuxquestions.org/questions/linux-kernel-70/why... - https://kerneltalks.com/disk-management/swap-addition-in-linux https://kerneltalks.com/disk-management/swap-addition-in-lin... - https://linuxaria.com/howto/linux-memory-management https://linuxaria.com/howto/linux-memory-management - https://kernelnewbies.org/Linux_4.8#head-a0b674ff33af7e3e7b80fcfa266a9d366f094e93 https://kernelnewbies.org/Linux_4.8#head-a0b674ff33af7e3e7b8... - https://kernelnewbies.org/Linux_4.11#head-680ed3eee137456d6a0665b3191be34d9a17bab2 https://kernelnewbies.org/Linux_4.11#head-680ed3eee137456d6a... - https://kernelnewbies.org/Linux_4.14#head-5327850512ba28763de4c8838246b4145df59033 https://kernelnewbies.org/Linux_4.14#head-5327850512ba28763d... - https://help.ubuntu.com/community/SwapFaq#How_much_swap_do_I_need.3F https://help.ubuntu.com/community/SwapFaq#How_much_swap_do_I... - https://access.redhat.com/solutions/15244 https://access.redhat.com/solutions/15244 - https://serverfault.com/a/684800/2321 https://serverfault.com/a/684800/2321 - https://serverfault.com/q/329928/2321 https://serverfault.com/q/329928/2321 - https://serverfault.com/q/25653/2321 https://serverfault.com/q/25653/2321 - https://serverfault.com/a/825915/2321 https://serverfault.com/a/825915/2321 - https://askubuntu.com/q/184217/3544 https://askubuntu.com/q/184217/3544 - https://antipaucity.com/2011/08/08/why-technical-intricacies-matter/#.WjfiqlQ-fgo https://antipaucity.com/2011/08/08/why-technical-intricacies... I've been in environments ranging from a few dozen individual servers to 100s of 1000s. In companies ranging from mom-and-pop shops to international investment banks. Every last one is configured with swap. Because unless you have done all the profiling to truly know you don't need it, you should be running it.
grahn 9y ago
> As a last resort, the Kernel will deploy OOM killer to nuke high-memory process(es) Yes! That is exactly what I want to happen! When the system runs out of RAM, things will generally stop functioning, swap enabled or not. The only question is how you want it to stop functioning when that happens. In almost every situation, I'll easily take the kernel killing whatever single process it thinks is most appropriate to get rid of, and keep everything else up and running smoothly, over grinding the the entire system to a halt by upping the effective memory access time by orders of magnitude. Simply put: If everything doesn't fit in memory, then don't try to run everything! Properly designed software nowadays is designed to be able to crash without corrupting data. As far as I'm concerned, it is almost always preferable to kill and restart instead of giving CPR to processes that don't fit in the working memory.
- kevin_nisbet 9y ago
  Funny story though, we had a database server get nuked because a client process used too much memory. We hadn't done the tuning correctly and someone decided to run their queries locally. So it was entirely our fault. But mysqld - 220GB, mysql (client) - 30GB, let's nuke the server process. Lesson learned, make sure you adjust the OOM killer on things like services that use lots of RAM.
- parenthephobia 9y ago
  I don't want this to happen. On servers, in almost every situation, I'd rather have the server reboot than have some random process crash and leave the system in a potentially broken state. On desktops, in every situation, I'd rather have X, my window manager and an emergency terminal pinned to RAM so I can always decide what to kill for myself. (https://github.com/stiletto/angrymlocker https://github.com/stiletto/angrymlocker helps with setting this up.)
  - Dylan16807 9y ago
    > I'd rather have the server reboot than have some random process crash and leave the system in a potentially broken state. Set the OOM killer to trigger a reboot? Crawling to a swapping halt is the worst of both worlds, it's like a full system crash but the server never comes back.
bandrami 9y ago
So, wouldn't the best answer clearly be to buy even more RAM, and use a RAMdisk for swap?
- dingaling 9y ago
  That would be the ideal response, but unfortunately there are cases when it's not possible. 1. System is already maxed-out at its board limit 2. Specific RAM simply isn't available any more from preferred vendors
- warrenm 9y ago
  You still want swap. See here for more: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
mnw21cam 9y ago
Swap engineering is definitely way more complicated than the article lets on. What people often don't realise is that Linux distinguishes between dirty and clean memory, with dirty memory not reflected in swap, and clean memory already stored somewhere on disc, and program code is counted as clean memory that just happens to live somewhere other than swap. Therefore, under memory pressure (especially if you set swappiness to zero), you will be preferentially swapping out your program code (because it is always clean) in preference to your program data. If you have no swap, then this is what causes the system to grind to a halt when the RAM is full- all your program code gets discarded from RAM, and nothing can run without reading it from disc again. The recommendation to have a little bit of swap is absolutely fine. However, as the amount of RAM in your system increases, the penalty for running out of RAM increases as well. On larger systems (for example 256GB RAM), I recommend using something like EarlyOOM[1] to kill off tasks before the pathological swapping case occurs. Otherwise, you could end up with an unresponsive system. If you have lots of RAM, the kernel OOM killer waits far too late, and the system is already unresponsive. [1] https://github.com/rfjakob/earlyoom https://github.com/rfjakob/earlyoom
- borman 9y ago
  > Therefore, under memory pressure (especially if you set swappiness to zero), you will be preferentially swapping out your program code (because it is always clean) in preference to your program data. Regarding systems with near 100% utilization, wouldn't it be a better advice to pin all executable code to memory via tmpfs (at system level) or mlockall() (application level)? Encountered this case on some heavily-loaded batch data processing workers: at some point, client programs would slow down and generate random disk I/O (which in turn is detected by monitoring and throttled). Turned out, processes would go through major page faults at random moments in time, when their executable code pages are discarded from memory and subsequently re-read from disk.
  - mnw21cam 9y ago
    I'd love to have a configuration option on Linux that elevates the penalty for evicting executable pages, or just makes them non-discardable. I think it would help a lot.
zzzcpan 9y ago
Nowadays linux has this thing called zram disk. It compresses data in memory on the fly. I think general swapping advice would serve most people better if they always add swap on zram first, before swap on ssd or hdd. The difference is that getting data in and out of zram is so fast that you can actually use it as a tier of a slower ram, transparently compressed.
- warrenm 9y ago
  Not exactly helpful. Swap is vital in modern systems: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
  - zzzcpan 9y ago
    I don't think 2.6 kernel reasons are relevant anymore, in fact they apply to ram compression techniques better, than to old raw swap on disk.
    - warrenm 9y ago
      Did you even read the list of resources in that link?
- therealmarv 9y ago
  is this activated per default today? Last information I've got from Ubuntu 16.04 times is that it is not activated by default. I know that macOS and Windows do similar to this by default.
  - zzzcpan 9y ago
    Some systems have it activated by default, on ubuntu it was a matter of installing a package zram-config with default configuration.
  - detaro 9y ago
    Distro-dependent. I think at least some distros for lightweight desktop systems and ChromeOS use it by default, RedHat trusts it enough to officially support it, but I don't think any of the major distros has it on by default.
voidmain0001 9y ago
Interesting; Jeff Atwood of Coding Horror/Stackoverflow/Discourse forces swap creation as part of the installation of Discourse as a Docker container. He prefers slow performance to OOM. https://meta.discourse.org/t/create-a-swapfile-for-your-linux-server/13880 https://meta.discourse.org/t/create-a-swapfile-for-your-linu...
- warrenm 9y ago
  Pretty much all smart sysadmins want swap. The kernel expects it, and performance is almost always worse without it. https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
  - AstralStorm 9y ago
    Only slightly worse. Technically, kernel will sometimes swap out live executable code unless you set swappiness to 0. This makes performance completely unpredictable under load. Kernel does not "expect" things, the defaults are just heuristics not failing to malloc past amount of ram you have. This causes all of the mentioned behaviour. Apps allocating too big RSS for instance, deferring GC and heap compaction too late causing kernel to unload executable code. And once you really run out of RAM, even on fast SSD swap will kill any performance. Unless you're using only hugepages.
makz 9y ago
System tuning is largely dependent on workload and use case. You can't just give a generic advice like this.
- warrenm 9y ago
  >System tuning is largely dependent on workload and use case Yes, you can - and should - give general advice like this Heres a whole host of reasons why you almost surely need/want swap: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
troisdetroie 9y ago
I've often disabled swap on systems running high-throughput databases like ElasticSearch and Cassandra because paging to disk will cause one or several nodes to slow down, which affects the performance of the whole cluster. The better thing to do in those cases is let the node fail right away by disabling swapping. I fact, elasticsearch prefers to be run with `swapoff` : https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...
- hyperpape 9y ago
  There's even a name for a slow node stalling an entire cluster: https://danluu.com/limplock https://danluu.com/limplock.

46 points by ashitlerferad 9y ago

I'm sorry, but not a single reason was presented in favour of swap when you're 100% confident that you will not max out your RAM. Swap = completely useless slowdown if you have total control over your system. Better use ulimit to prevent processes with e.g. memory leaks from causing havoc.

__david__ 9y ago
They presented one reason: the system will swap out more or less unused memory and give that much more space for disk cache, etc. Now, I'm not sure I find this a particularly compelling reason. I suppose it depends on how much RAM you have available for cache vs how much unused junk gets swapped out. If you've got 5G of unused RAM and 300M extra of stale allocations that get swapped out, then it's probably not going to impact your system in any measurable way. I personally like a quick OOM, myself, rather than a machine slowing down to a crawl for multiple minutes while swap gets exhausted.
subway 9y ago
RAM freed by swapping some infrequently accessed memory out to disk becomes available for cache use. If you have total control of your system, you know when you can safely allow data to be paged, and when you need to lock memory to physical ram.
ChrisSD 9y ago
Can you ever be 100% confident?
- notyourday 9y ago
  Yes. ulimit and cgroups do wonders.
- warrenm 9y ago
  You can 100% be confident (except in a few very unusual use cases) that swap is important, yes. https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
efaref 9y ago
The most compelling reason to enable at least some swap space is that the Linux kernel's memory handling algorithms can't cope with there being no swap. If you have no swap, then you will encounter stalls and OOMs even if you don't actually run out of memory. Adding even a tiny amount of swap prevents this from happening.
- albertzeyer 9y ago
  This sounds as if it is a serious issue and should be fixed. Is there a bug report about it?
  - lolc 9y ago
    This is due to memory overcommit, it's a performance feature. You can turn it off.
    - kevin_nisbet 9y ago
      Interesting, I sort of remember an interaction with NUMA as well. I don't believe this is a bug, just a miss-understanding of the way the underlying system works. From what I remember when swapiness is 0, there are lots of reports of processes that use more than a single NUMA node worth of memory getting killed by the OOM killer, even with plenty of free RAM available. I unfortunately don't remember the details, but this prevented the memory from being able to be allocated to the additional node. I've heard of this mostly with mysql, where it's common to have a big server with lots of RAM, but a single large process that uses most of the system RAM. The way we got around this, was by setting the process to allocate interleaved among the NUMA nodes. I'll have to dig into the overcommit.
  - warrenm 9y ago
    It's not a serious issue at all. It's intentional.
- bechampion 9y ago
  i agree with this , lack of swap makes oom too aggressive . , question what's worst no app (and react to OOM) or app swapping
warrenm 9y ago
Want reasoning? Here's a whole huge list: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447 When it is the case in the overwhelming majority of cases that you should use swap, you should just use it.

martin_andrino 9y ago

Swap isn’t needed at all in this age. It’s just there for people who can’t afford more RAM and therefore have to resort to hacky solutions like this.

geofft 9y ago
The article specifically lists reasons why you should enable swap when there is plenty of RAM. Can you elaborate on what part of the article's reasoning you disagree with?
tedeh 9y ago
Disk is still way cheaper than RAM, and I always add some swap space to my servers just to lessen the chance of getting ENOMEM errors if/when an unexpected or unusual workload would come along.
majewsky 9y ago
> It’s just there for people who can’t afford more RAM So I should throw away my 5-year-old notebook that's still working perfectly fine just because the RAM is soldered in? Doesn't sound like efficient resource usage (both money and rare earths).
bogomipz 9y ago
What an absurd statement. Do you use a laptop? I do and I'm using swap space right now. I would like to buy a laptop with 32 or 64 Gigs of RAM but I can't as most(all?) laptop makers don't ship with a memory controller that permits more than 16 Gigs of RAM. And I can afford to buy more RAM.
- cztomsik 9y ago
  What is the rule for setting swap size, or can I use just swap file just like on win/mac? (I heard it will disable hibernation then)
  - warrenm 9y ago
    The basic rule is `RAM + 2GB` I usually peak at 34GB, unless I have reason (like needing/wanting to enable full hibernation) to use 66GB, 130GB, etc.

djb_hackernews 9y ago

What do the people that use cloud instances that don't have local disks do? You definitely don't want swap to be on EBS...

sharms 9y ago
Its a good question, but the swap advice is suspicious. If I would normally purchase a 4GB of ram instance with 4GB swap, then what happens when I purchase a 8GB instance? Do I still need 4GB swap? Hopefully this helps illustrate that it is really the applications you intend to run and the swap doesn't mean the system gets more efficient
LinuxBender 9y ago
Allocate 20+% more memory that you could ever need. Use cgroups, lxc, systemd to constrain applications and containers to specific amounts of memory, cpu, etc. Properly engineered systems will have enough memory for everything else under the hood beyond the application. VM and container abstraction does not negate the requirement to calculate this.

bryanlarsen 9y ago

Kubernetes forces you to start it with flag to tell it "I know what I'm doing, I know this machine has swap, but start Kubernetes anyways even though it's normally a bad idea to run Kubernetes on a machine with swap".

hannob 9y ago

I wondered this for a while, unfortunately a major question I have remains unanswered in the text: What about SSDs and SSD-only systems? (Like... any average laptop.) Should I be worried that putting a swap on an SSD will cause it to wear out fast due to many write cycles?

ktpsns 9y ago
Same here. How does file swap (cf. https://wiki.archlinux.org/index.php/swap#Swap_file_creation https://wiki.archlinux.org/index.php/swap#Swap_file_creation) could help out to let an underlying file system distribute the usage of the ssd? Combined with recreating a swap file on every boot (say on a frequently booting desktop machine), this could make a difference.
INTPenis 9y ago
I set vm.swappiness to 0 on my Fedora laptop. Meaning the kernel will only swap to save the system. I've been doing this for years and had no issues. Edit: Actually in recent kernels, which I am using, 0 means it's disabled. I was thinking of 1.
- AstralStorm 9y ago
  It does not help, just makes the problem appear later in a more dire way. Try overcommit settings instead.
  - INTPenis 9y ago
    I googled that and was pleasantly surprised to learn something new.[1] But if I'm reading that right I would probably want to do both. Because overcommit settings seem to be a memory saving technique. To avoid needing swap by decreasing the allowed overcommit memory. Anyways, in the many years that I've been changing vm.swappiness I've never had any issues and I've always had a minimum of 8G RAM so I don't think this will be a big issue. 1. http://engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/ http://engineering.pivotal.io/post/virtual_memory_settings_i...
9y ago

dennisjac 9y ago

I always see people caring way to much about the amount of swap space used and not enough about the swap activity going on. It isn't the amount of swap space that slows a system down but actual swapping in/out of memory pages. In vmstat you want to pay attention to the "si" and "so" columns and in your alerting/graphing you want to keep track of the values "pswpin" and "pswpout" in /proc/vmstat. If these are almost always exactly or near zero then that means the fact that some memory pages are swapped out has virtually no impact on the performance of your system. There are two other important issues to take into account though. 1) Even if the memory pages swapped out are not accessed generally they might be forces back into memory because of some specific action. One example is a database that has all the hot records in memory but other records that are accessed only very rarely swapped out to disk. In general everything will perform fine in this situation but the moment you do e.g. a table scan and a lot of these records need to be moved back into memory you might see a disk I/O peak that might be quite a kick on the neck for overall performance if the database is really busy. 2) If you don't have any swap space configured you might still run into problems with swap which seem to be caused by a bug in the memory handling in the kernel. I've seen this on some KVM hypervisors which were CentOS 7 systems. These systems were equipped with 128G of RAM and had two virtual machines running which both were configured with 32G of virtual RAM. They ran fine until one day the kswapd kernel process ran with 100% cpu usage even though no swap swap was configured whatsoever (to avoid the situation mentioned above). The "fix" was to dump the systems caches with "echo 3 > /proc/sys/vm/drop_caches" which seemed to calm down kswapd again. As best as I can tell what happened is that the system used all the free ram for the page cache and buffers and when the system needed some memory it apparently prefered to swap pages out to disk (even though no swap was configured) rather than reclaiming page cache of which there was plenty to reclaim. Unfortunately that means there seems to be no bulletproof way to say "only use physical ram and never try to swap anything out to disk". Even /proc/sys/vm/swappiness can be dangerous as a value of "0" doesn't actually tell the system to only swap if absolutely necessary but can lead to OOM situations even if swap space is still available (see https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=fe35004fbf9eaf67482b074a2e032abb9c89b1dd https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux... for details). TL;DR: 1) Don't just pay attention to the amount of swap space but to actual swap activity over time 2) Be aware of corner cases and bugs relating to swap

mcculley 9y ago

When running an instance where the biggest process is a garbage collected runtime, one does not want any swap at all. If the garbage collector is forced to walk through all of the pages in swap hunting for live references, performance is terrible. It is better then to have the process see only as much virtual memory as there is RAM in the system. It is a failure of Unix/Linux that there is no sensible API for managing real RAM from the point of view of the process.

LinuxBender 9y ago

If you must use swap (you probably don't need to), then at least set zswap.enabled=1 in the kernel boot options, typically in grub. This will enable lzo compression of swap in memory so there is less writing to disk. Some newer kernels use lz4. Either way, what most folks are actually missing is the correct kernel settings for the amount of memory they have. Sadly, the kernel does not dynamically adjust these based on your total amount of memory. Below is based loosely on Redhat suggestions. Please note, that if you fall below the min free, the kernel will decide what to do next, based on your oom and panic settings. The settings below will have the kernel free cache and other memory earlier so that you do not hit those stalling conditions and can even prevent some OOM race conditions. Some might suggest tuned, but use tuned with caution or at least read up on everything it does. MEM=`grep ^MemTotal /proc/meminfo | awk {'print $2'}` if [ ${MEM} -gt 1129241478 ] ; then sysctl -q -w vm.min_free_kbytes=16384000 elif [ ${MEM} -gt 564620739 ] ; then sysctl -q -w vm.min_free_kbytes=8192000 elif [ ${MEM} -gt 352887962 ] ; then sysctl -q -w vm.min_free_kbytes=4096000 elif [ ${MEM} -gt 176443981 ] ; then sysctl -q -w vm.min_free_kbytes=1024000 elif [ ${MEM} -gt 88221990 ] ; then sysctl -q -w vm.min_free_kbytes=524288 else sysctl -q -w vm.min_free_kbytes=262144 fi If you have small VM's, then perhaps set the default above to something a little smaller. You can of course free up about 128MB on default installations by removing "crashkernel" from your grub config and rebooting. Then do this regardless, because overcommit set to 0 does not mean off, so we set the ration to 0 as well. Overcommit is good for developers testing code and finding the correct ways to manage memory in their applications during development. sysctl -q -w vm.overcommit_ratio=0 And of course, cache pressure plays into early evacuation of the right cache based on your usage: ## default is 100 (optimal for file servers). 4000+ for in memory databases. ## 10000 means always prefer page cache. sysctl -q -w vm.vfs_cache_pressure=1000 And if you have people oversubscribing a lot: (adjust based on your memory capacity) sysctl -q -w vm.admin_reserve_kbytes=131072 sysctl -q -w vm.user_reserve_kbytes=262144 Please do read up on all of these before testing on your test machines. [1] [1] https://www.kernel.org/doc/Documentation/sysctl/ https://www.kernel.org/doc/Documentation/sysctl/ Then finally, make sure you have Transparent Huge Pages disabled unless you know for sure you need it. THP can leak a lot of memory and it is nearly impossible to see without extensive kernel debugging. In grub, set this and reboot: transparent_hugepage=madvise Or to manually disable THP during run-time, echo -n "madvise" > /sys/kernel/mm/transparent_hugepage/enabled echo -n "never" > /sys/kernel/mm/transparent_hugepage/defrag Then restart your applications. THP defrag can also cause stalling and lag spikes, especially in large memory java deployments (it will look like FGC's) and in MongoDB, Cassandra, others. If you did this manually, stop your apps, flush cache, compact memory, then start your apps. sync;sync;sync echo 3 > /proc/sys/vm/drop_caches echo 1 > /proc/sys/vm/compact_memory Some will say 3 sync's is not required. This is mostly true, but some old raid controllers treat this differently.

kevin_nisbet 9y ago

In my experience, I think engineering swap usage is more complex than this article leads on. While adding more swap may make sense for many systems, I've personally worked on several systems where disabling swap made a huge difference. It's just one of those things where there isn't a hard rule. One of the issues I've encountered more than once with Java software, is what appears to be longer than expected pauses in the GC, that I've linked to swap usage. The theory is, if you have a reference that lives long enough to get promoted to an old generation, but then is released, the memory in question won't be used for quite some time. This can lead the kernel to detect the memory pages not being used, swap them to disk, but then when it's time to do GC, the process is just frozen while the GC is running and get's backed up. Depending on the workload this might be fine, however, on both the platforms I encountered this issue on, were real time applications which were engineered to have available memory for the application and did (and also had no benefit from using the free'd memory for IO cache). So in this case, disabling swap had a real and measurable performance boost for us, getting rid of some of the variability in our GC runtime and pauses. Under low memory conditions I think is also subjective. So if I'm a distributed / highly available system that's doing message routing in a telco environment, I would rather have the OOM killer kill the process, and failover to a backup, then for the system to be up but unresponsive because it's spending a majority of it's time swapping too/from disk. Again, this totally depends on the platform.

nisa 9y ago

This is dangerous advice. At least on Ubuntu 16.04 with stock kernel running due to memory pressure into swap kills the systems. Basically everything times out and you can only hard-reboot. Without swap you have an OOM at least something you can recover from. If you need swap just add a little like 1-4gb - because if you have spinning disks 64gb swap won't help you at all because every request to swap is magnitudes more slower. Might be different with NVMe but it's still noticable.

ausjke 9y ago
indeed on ubuntu 16.04 64bit, when my chrome has 20+ tabs, some new tab could trigger a memory thrashing, that uses up all the memory and eats up a few GB disk, cpu load jumps up to 10x, the computer basically becomes unusable. i would rather a OOM kill the chrome process in this case, it wastes less time on me, a re-open of chrome will keep all the histories anyways. also on embedded system where the RAM is fairly limited, and no hard drive attached, and its storage is measured by MB instead of GB, you simply can not have the swap partition. I want to run the system without swap actually.
- warrenm 9y ago
  >when my chrome has 20+ tabs That's not a swap problem, that's a Chrome problem.
  - AstralStorm 9y ago
    It is a swap problem. Swap is supposed to make things work and not stop. I suppose this is actually a problem of Linux optimistic malloc policy.
tfha 9y ago
Not just ubuntu. We tell our engineers to remove their swap partitions because sometimes the test suite will consume all remaining memory and push into swap. If that happens, pretty much the only thing you can do is a hard poweroff. If you're fast, you can Ctrl+C and only lose a minute or two, but often you're just completely stuck. Much better to OOM and have the test suite killed early.

mrweasel 9y ago

I'm not going to disagree with the article, because I haven't done any testing myself that would suggest that the point made are wrong. I do however find it amusing to login to a system with 32 or 64GB of memory with a 2GB swap. It seems humorous that those 2GB should somehow do anything if you already used 64GB of RAM.

warrenm 9y ago
I can save you a bunch of testing time. Just enable swap (I typically peak at 34GB, but follow the `RAM + 2GB` rule-of-thumb before that Here's a bunch of data that helps understand why swap is important: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447

jacquesm 9y ago

Cargo cult system administration. No, you should not 'almost always add swap space'. What you should do instead is tune your system for its intended use. If you need swap as an 'early warning system' that you're about to run out of memory you're already doing it wrong and the OOM killer is a piece of code that has default settings that can be tuned, ditto for the virtual memory manager in the kernel. http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html http://www.oracle.com/technetwork/articles/servers-storage-d... https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-captun https://access.redhat.com/documentation/en-us/red_hat_enterp... Select appropriate equivalent for your kernel where applicable, and don't forget about the per-process limits in ulimit. This sort of 'almost useless' advice is pretty annoying, it shows that the writer has an interest in the matter but didn't bother to dig in deep enough to make the advice truly useful. Also, the comment on that article that implies that swap will allow you to update a running process because it is backing the image is wrong, the backing is simply done through the filesystem and swap space has nothing to do with it. The practical upshot of which is that you're not going to be able to reclaim the space held by the binary if you should remove it because the 'file is busy' until the last instance of that process exits. In the meantime, if you really want to run the newer version under the old name or path then you're free to unlink it or 'mv' it to a different location and to start the new binary from the default location. On some Unixes you can even overwrite the binary using an mv command making it look like the old one has disappeared but again, under the hood it will still be held open, you can check this using lsof or whatever is your local equivalent to see that the file is indeed still present. In a pinch (and you really really have messed up if you ever need this trick) you can re-link the running image by hard linking to the inode.

warrenm 9y ago
You should just about always add at least some swap. Anyone who thinks otherwise hasn't read much or seen very many systems. start here: - https://www.kernel.org/doc/gorman/html/understand/understand013.html https://www.kernel.org/doc/gorman/html/understand/understand... - https://www.kernel.org/doc/gorman/html/understand/understand014.html https://www.kernel.org/doc/gorman/html/understand/understand... - https://www.linuxquestions.org/questions/linux-kernel-70/why-is-my-linux-server-using-swap-when-inactive-memory-is-available-4175580818 https://www.linuxquestions.org/questions/linux-kernel-70/why... - https://kerneltalks.com/disk-management/swap-addition-in-linux https://kerneltalks.com/disk-management/swap-addition-in-lin... - https://linuxaria.com/howto/linux-memory-management https://linuxaria.com/howto/linux-memory-management - https://kernelnewbies.org/Linux_4.8#head-a0b674ff33af7e3e7b80fcfa266a9d366f094e93 https://kernelnewbies.org/Linux_4.8#head-a0b674ff33af7e3e7b8... - https://kernelnewbies.org/Linux_4.11#head-680ed3eee137456d6a0665b3191be34d9a17bab2 https://kernelnewbies.org/Linux_4.11#head-680ed3eee137456d6a... - https://kernelnewbies.org/Linux_4.14#head-5327850512ba28763de4c8838246b4145df59033 https://kernelnewbies.org/Linux_4.14#head-5327850512ba28763d... - https://help.ubuntu.com/community/SwapFaq#How_much_swap_do_I_need.3F https://help.ubuntu.com/community/SwapFaq#How_much_swap_do_I... - https://access.redhat.com/solutions/15244 https://access.redhat.com/solutions/15244 - https://serverfault.com/a/684800/2321 https://serverfault.com/a/684800/2321 - https://serverfault.com/q/329928/2321 https://serverfault.com/q/329928/2321 - https://serverfault.com/q/25653/2321 https://serverfault.com/q/25653/2321 - https://serverfault.com/a/825915/2321 https://serverfault.com/a/825915/2321 - https://askubuntu.com/q/184217/3544 https://askubuntu.com/q/184217/3544 - https://antipaucity.com/2011/08/08/why-technical-intricacies-matter/#.WjfiqlQ-fgo https://antipaucity.com/2011/08/08/why-technical-intricacies... I've been in environments ranging from a few dozen individual servers to 100s of 1000s. In companies ranging from mom-and-pop shops to international investment banks. Every last one is configured with swap. Because unless you have done all the profiling to truly know you don't need it, you should be running it.

grahn 9y ago

> As a last resort, the Kernel will deploy OOM killer to nuke high-memory process(es) Yes! That is exactly what I want to happen! When the system runs out of RAM, things will generally stop functioning, swap enabled or not. The only question is how you want it to stop functioning when that happens. In almost every situation, I'll easily take the kernel killing whatever single process it thinks is most appropriate to get rid of, and keep everything else up and running smoothly, over grinding the the entire system to a halt by upping the effective memory access time by orders of magnitude. Simply put: If everything doesn't fit in memory, then don't try to run everything! Properly designed software nowadays is designed to be able to crash without corrupting data. As far as I'm concerned, it is almost always preferable to kill and restart instead of giving CPR to processes that don't fit in the working memory.

kevin_nisbet 9y ago
Funny story though, we had a database server get nuked because a client process used too much memory. We hadn't done the tuning correctly and someone decided to run their queries locally. So it was entirely our fault. But mysqld - 220GB, mysql (client) - 30GB, let's nuke the server process. Lesson learned, make sure you adjust the OOM killer on things like services that use lots of RAM.
parenthephobia 9y ago
I don't want this to happen. On servers, in almost every situation, I'd rather have the server reboot than have some random process crash and leave the system in a potentially broken state. On desktops, in every situation, I'd rather have X, my window manager and an emergency terminal pinned to RAM so I can always decide what to kill for myself. (https://github.com/stiletto/angrymlocker https://github.com/stiletto/angrymlocker helps with setting this up.)
- Dylan16807 9y ago
  > I'd rather have the server reboot than have some random process crash and leave the system in a potentially broken state. Set the OOM killer to trigger a reboot? Crawling to a swapping halt is the worst of both worlds, it's like a full system crash but the server never comes back.

bandrami 9y ago

So, wouldn't the best answer clearly be to buy even more RAM, and use a RAMdisk for swap?

dingaling 9y ago
That would be the ideal response, but unfortunately there are cases when it's not possible. 1. System is already maxed-out at its board limit 2. Specific RAM simply isn't available any more from preferred vendors
warrenm 9y ago
You still want swap. See here for more: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447

mnw21cam 9y ago

Swap engineering is definitely way more complicated than the article lets on. What people often don't realise is that Linux distinguishes between dirty and clean memory, with dirty memory not reflected in swap, and clean memory already stored somewhere on disc, and program code is counted as clean memory that just happens to live somewhere other than swap. Therefore, under memory pressure (especially if you set swappiness to zero), you will be preferentially swapping out your program code (because it is always clean) in preference to your program data. If you have no swap, then this is what causes the system to grind to a halt when the RAM is full- all your program code gets discarded from RAM, and nothing can run without reading it from disc again. The recommendation to have a little bit of swap is absolutely fine. However, as the amount of RAM in your system increases, the penalty for running out of RAM increases as well. On larger systems (for example 256GB RAM), I recommend using something like EarlyOOM[1] to kill off tasks before the pathological swapping case occurs. Otherwise, you could end up with an unresponsive system. If you have lots of RAM, the kernel OOM killer waits far too late, and the system is already unresponsive. [1] https://github.com/rfjakob/earlyoom https://github.com/rfjakob/earlyoom

borman 9y ago
> Therefore, under memory pressure (especially if you set swappiness to zero), you will be preferentially swapping out your program code (because it is always clean) in preference to your program data. Regarding systems with near 100% utilization, wouldn't it be a better advice to pin all executable code to memory via tmpfs (at system level) or mlockall() (application level)? Encountered this case on some heavily-loaded batch data processing workers: at some point, client programs would slow down and generate random disk I/O (which in turn is detected by monitoring and throttled). Turned out, processes would go through major page faults at random moments in time, when their executable code pages are discarded from memory and subsequently re-read from disk.
- mnw21cam 9y ago
  I'd love to have a configuration option on Linux that elevates the penalty for evicting executable pages, or just makes them non-discardable. I think it would help a lot.

zzzcpan 9y ago

Nowadays linux has this thing called zram disk. It compresses data in memory on the fly. I think general swapping advice would serve most people better if they always add swap on zram first, before swap on ssd or hdd. The difference is that getting data in and out of zram is so fast that you can actually use it as a tier of a slower ram, transparently compressed.

warrenm 9y ago
Not exactly helpful. Swap is vital in modern systems: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
- zzzcpan 9y ago
  I don't think 2.6 kernel reasons are relevant anymore, in fact they apply to ram compression techniques better, than to old raw swap on disk.
  - warrenm 9y ago
    Did you even read the list of resources in that link?
therealmarv 9y ago
is this activated per default today? Last information I've got from Ubuntu 16.04 times is that it is not activated by default. I know that macOS and Windows do similar to this by default.
- zzzcpan 9y ago
  Some systems have it activated by default, on ubuntu it was a matter of installing a package zram-config with default configuration.
- detaro 9y ago
  Distro-dependent. I think at least some distros for lightweight desktop systems and ChromeOS use it by default, RedHat trusts it enough to officially support it, but I don't think any of the major distros has it on by default.

voidmain0001 9y ago

Interesting; Jeff Atwood of Coding Horror/Stackoverflow/Discourse forces swap creation as part of the installation of Discourse as a Docker container. He prefers slow performance to OOM. https://meta.discourse.org/t/create-a-swapfile-for-your-linux-server/13880 https://meta.discourse.org/t/create-a-swapfile-for-your-linu...

warrenm 9y ago
Pretty much all smart sysadmins want swap. The kernel expects it, and performance is almost always worse without it. https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447
- AstralStorm 9y ago
  Only slightly worse. Technically, kernel will sometimes swap out live executable code unless you set swappiness to 0. This makes performance completely unpredictable under load. Kernel does not "expect" things, the defaults are just heuristics not failing to malloc past amount of ram you have. This causes all of the mentioned behaviour. Apps allocating too big RSS for instance, deferring GC and heap compaction too late causing kernel to unload executable code. And once you really run out of RAM, even on fast SSD swap will kill any performance. Unless you're using only hugepages.

makz 9y ago

System tuning is largely dependent on workload and use case. You can't just give a generic advice like this.

warrenm 9y ago
>System tuning is largely dependent on workload and use case Yes, you can - and should - give general advice like this Heres a whole host of reasons why you almost surely need/want swap: https://news.ycombinator.com/item?id=15952447 https://news.ycombinator.com/item?id=15952447

troisdetroie 9y ago

I've often disabled swap on systems running high-throughput databases like ElasticSearch and Cassandra because paging to disk will cause one or several nodes to slow down, which affects the performance of the whole cluster. The better thing to do in those cases is let the node fail right away by disabling swapping. I fact, elasticsearch prefers to be run with `swapoff` : https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...

hyperpape 9y ago
There's even a name for a slow node stalling an entire cluster: https://danluu.com/limplock https://danluu.com/limplock.