8 ms·
Almost Always Add Swap Space
- sgt 7y agoLearned about atop. Wish it existed on macOS too.
- jihadjihad 7y agoI've been using htop for a while now, it is pretty great: https://hisham.hm/htop/ https://hisham.hm/htop/
- IHLayman 7y agoAlso check out glances: https://nicolargo.github.io/glances/ https://nicolargo.github.io/glances/ A good terminal display overview of processes, network, disk, and battery.
- tutfbhuf 7y ago> The Linux Kernel will move memory pages which are hardly ever used into swap space to ensure that even more cachable space is made available in-memory for more frequently used memory pages (a page is a piece of memory). Why should I care if I always have enough RAM? Gedankenexperiment: Let's say u'd have infinite RAM, would swap make any sense?
- vectorEQ 7y agowhat if an application contains a memory leak. it could hypothetically use up your infinite ram, and linux will then start randomly killing stuff to try and recover.... adding swap allows you to trigger on memory issues / overload without random shit getting killed. you could then kill the offending process yourself specifically being triggered by some memory usage monitor or w/e instead of running the risk of other critical processes being killed.
- saalweachter 7y agoWell to start, slap default resource limits on your processes so they can't use all the RAM in the world. Then your leaky processes will crash when their mallocs fail and everyone will be as happy as they deserve.
- scottlamb 7y agoI agree with the idea, but a nit: on 64-bit Linux with the default vm.overcommit_memory=0, their mallocs will likely continue to succeed. Those just expand the address space, which is probably not what your resource limits are controlling. [1] The actual limit [2] is hit when they try to use the virtual address space, causing a minor page fault (when it's backed by a zero-filled page of physical RAM). [1] You can add in such a limit via RLIMIT_AS, but it'd have a lot of collateral damage like preventing LMDB from mmap()ing a big database. I don't recommend this. [2] I'd recommend setting limits on a cgroup; with a systemd unit file, see MemoryMax= and friends in the systemd.resource-control(5) manpage.
- xorcist 7y agomallocs never fails. By design. That's the whole point of all this. On Linux you can make them fail, but don't do that unless you know what you're doing.
- zamadatix 7y agoWhy do you need to wait for it to swap to trigger a memory usage alert?
- mason55 7y agoI was hoping this post was going to say something like turning on some small amount of swap will allow the OS to cache some things that normally it wouldn't cache at all, and so you get some improvement there. What's actually in this post doesn't make any sense to me. If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping. First, my production servers are not doing many other things besides being production servers. It's not like they are running a bunch of unnecessary services, and if I kill some then my application can recover. If a process is out of control then it's almost assuredly something important that I'm going to have to kill anyway. Second, I find it's much harder to detect degraded performance than it is to detect a dead process. It's very, very easy to have a health check that will detect a host who has stopped listening and drop that host from the LB. And alerting on that scenario is very easy as well. The alternative is a host that's operating in a degraded state, which I need to detect with more sophisticated health check + alerting, and in the end my resolution is just going to be to kill everything anyway. In a properly designed HA environment the loss of a host should be no big deal. Architecture should be focused on making sure a host goes down ASAP if it's having problems, not letting it survive in some kind of zombie state.
- RoutinePlayer 7y agoAwesome comment! Especially as it relates to VMs in modern cloud environments. There really is no need for swap space, and letting them fail quickly is usually best. Using swap space may make sense for a traditional physical, stand-alone server, where having it fail quickly doesn't really help. Or maybe for a workstation.
- deathanatos 7y ago> If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. The memory freed from a page that got pushed to swap might be better used for serving disk cache, for example. If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when they do become busy, you won't have to wait. At least, that's the theory as I understand it. Whether or not it works in practice, IDK. > And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping. I have successfully recovered hosts that began swapping, and even hosts that filled their swap. But it's beyond painful, and in most of the cases where I have a host doing such swapping, an OOM kill would have been a welcome reprieve. > Second, I find it's much harder to detect degraded performance than it is to detect a dead process. Agreed, and restarting something that's been OOM'd is simple enough too. Our hosts tend to stop reporting metrics when they start swapping, simply b/c so little is actually getting done, so they get labelled as "down". Linux also exposes a metric called "major page faults" (IIRC, it's "pgmajfault" in /proc/vmstat) that records the number of major (required service from disk) page faults; if the rate of that value is too steep for too long, that ≅ swap thrashing.
- brylie 7y agoI have manually added swap on SSD VPS nodes that come without it by default. I was under the impression that the hosting providers were trying to preserve the hardware. Is swap space still bad for SSD drives?
- IHLayman 7y agoFor another perspective, the Kubernetes community has had a long discussion about using swap space over the course of the last 2 1/2 years, as you can see in this feature thread: https://github.com/kubernetes/kubernetes/issues/53533 https://github.com/kubernetes/kubernetes/issues/53533 TL;dr: As of version 1.8 any system running k8s has to disable swap space or formally mark in the kubelet that you are turning the check for it off. There are good arguments on both sides of this issue, and the discussion is still continuing.
- Waterluvian 7y agoI have 32GB Ram on Ubuntu 18.04. Can I just turn swap off? If I do, what's the worst that will happen?
- devoply 7y agoOut of memory processes will be terminated automatically with only a message in the kernel log, i.e. dmesg.
- Beldin 7y agoWhat happens in case you have swap space and run out of memory?
- devoply 7y agosome stuff in your memory gets written to the swap page. https://en.wikipedia.org/wiki/Paging https://en.wikipedia.org/wiki/Paging
- pjc50 7y agoYour system thrashes to a halt, until eventually you run out of swap as well and the OOM killer is activated anyway. It just takes longer and is preceded by a period of dismal performance.
- Dylan16807 7y agoEven without swap, you get a period of thrashing and dismal performance as your disk cache is squeezed to nothing. If you have swap, this happens later and more gradually, and lasts longer before the OOM killer kicks in.
- serf 7y agoin a sane OS? Nothing until you run out of ram. In a Microsoft OS? Depends on the app -- some are dependent (many games) on having swap available (and may err silently or under false pretenses without one). I've found that a lot of MMO style games include cheat detection systems (see : rootkit) that rely on swap, and fail without it. As to why they do that, i'm unsure.
- scottlamb 7y agoThis is bad advice when the swap is backed by spinning disk rather than SSD, in large part because paging in once the problem is solved is not aggressive enough. Let me make it concrete: say 1 GiB of your application's RAM gets swapped out then is later paged in only as needed (4 KiB pages) with no readahead. Now the application's VM space is a minefield: there are up to 262,144 times it can stall for a ~10 ms disk seek (for a total of ~40 minutes). Sequentially reading in 1 GiB of RAM from disk would take only ~5 seconds. Hopefully OSs use some readahead, but I'm sure Linux and macOS don't use enough. I find a system that has ever swapped to be totally unusable until I do "sudo swapoff -a" (on Linux) to force everything to be paged in, or just reboot (on macOS, I haven't found any other way). Some of this can happen even without swap: the OS will still drop clean file-backed pages, so unless you mlock() your executables after startup (my production binary at work does this), you can still have major page fault-induced latency spikes. Swap backed by SSD (or compressed RAM) should be more reasonable, but I've had enough bad experiences with swap on spinning disk that authoritative-sounding articles that encourage swap without mentioning this problem piss me off. In contrast, if you don't have swap and run out of RAM, something will die and get restarted. In many cases this is a much better failure mode than continuing to run slowly.
- uaygsfdbzf 7y agoAuthor said almost always. Nothing is absolute in cases like this. I agree with the article. We enable swap on our production servers precisely because the engineers don't rectify their memory leaks or whatever problems are internal to their software. Some of those engineers are our own and we can't convince them to fix their software. Others are for things we have no control over, such as Kafka, for example. When Kafka starts going out of control and you've got no swap, oom killer will kill it and you'll end up with corrupted index files, which is a bigger problem than the performance problem you'll see if it has to swap for a while until it gets itself sorted out.
- magicalhippo 7y agoI've had the opposite experience on Windows. I always put swap on HDD, because I've been bitten several times by memory leaking applications rendering the entire OS unusable until the SSD with the swap file ran out of space. Windows would happily grow the pagefile beyond the limits I had set, and if the allocation loop was tight enough the OS chocked on handling all page faults. As in using literally minutes to respond to keystrokes. By putting swap on a slow HDD, Windows has plenty of time to service mouse and keyboard input while waiting for the disk to respond, allowing me to kill the offending application before saturating my disk. I assume this behavior was OK back when computers had very limited memory, and disks were slow anyway. But these days, if a single application wants more than the available memory (physical + pagefile), I'd rather the OS just said no to the application.
- paulcarroty 7y agoSwap doesn't make difference, just a parachute for your apps: give some work guaranties, but make things slower & eats resources of SSDs. I'm prefer add RAM as much my tasks needs it.
- mnw21cam 7y agoWithout swap, your server could potentially become suddenly unresponsive, as memory pressure forces all your running code out. The system will still be thrashing, and even worse than it you did have swap. If you do have swap, then you get a slightly more graceful degradation, and hopefully a chance to fix the problem first. What you really want is to never have a severe out of memory situation, and you can do that either by getting enough RAM that you never use it all, or by killing stuff nice and early when memory starts getting low.
- paulcarroty 7y ago> Without swap, your server could potentially become suddenly unresponsive The server never be unresponsive 'cause lack of swap. Don't like the logic "we just add a swap and maybe it will be slower a bit, but still working".
- sys_64738 7y agoSSD V Spinner? Metal V VM?
- deleted 7y ago[deleted]
- lone_haxx0r 7y agoIs it true that swap reduces the life expectancy of SSDs by writing too frequently?
- JustSomeNobody 7y agoI think this used to be the case with older SSDs, but newer ones are able to survive much longer.
- deleted 7y ago[deleted]
- LargoLasskhyfv 7y agoSwap is good. If it is in RAM. Just use zram. Like so: https://github.com/armbian/build/blob/master/packages/bsp/common/usr/lib/armbian/armbian-zram-config https://github.com/armbian/build/blob/master/packages/bsp/co... zramctl output from a tiny NanoPiNeo2 with 1GB Ram, running from a 64GB Sandisk SDXC card: NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram1 lzo 496.7M 4K 76B 12K 4 [SWAP] /dev/zram0 zstd 50M 2.5M 449K 832K 4 var/log Edit: Formatting (useless)
- zzzcpan 7y agoWhat I agree is that these days you should still use swap, but swap should always be either on zram or on zswap if you can't use zram for some reason, have too little ram, can't load zram module or something. Raw swap on disk is just not acceptable anymore for performance, security, wearout reasons.
- gnode 7y ago> performance, security, wearout reasons If memory pressure is low, and the swap is used to page out rarely / never used pages in favour of more disk cache, then there is a performance gain to be had, and wear should be minimal. You can encrypt swap with a randomly generated key to aid security. Ideally memory compression and disk swap would be used together; there's a gain to be had by shrinking lesser used pages, but no sense in taking up any space in RAM for pages showing no sign of life.
- zzzcpan 7y agoMore vfs cache for swap space is not a trade off here, but rather a mostly incorrect oversimplification. The value of more cache is very non linear, it depends on the workload and usually only matters when nearly all of the disk I/O is already exhausted, the exact situation you want to prevent by using more of the same disk I/O for swap. Essentially trading more cache for abrupt performance degradation in terms of latency and responsiveness. I mean, sure, if you can't use zram, say you are on a system that's a bit behind on performance features, like freebsd, very light swappiness might be ok, might free up some of very unused memory and just not be aggressive enough to swap something that causes latency, responsiveness problems, although this is not certain. With memory compression you can swap and compress much more of it more aggressively and be certain that no abrupt performance degradation can happen. It feels like having more ram that's gradually affecting tail latency the more you use it.
- Dylan16807 7y agoZram is nice when you're barely using it, but every time I've seen it get low on memory it degrades quite abruptly, much worse than ssd swap.
- parliament32 7y agoIt's not quite as black and white as the author makes it out to be. I've worked on (specialized) systems with 512gb of RAM with a 180gb SSD installed... No, you're not going to see any performance improvement at all by adding swap to the already insane amount of RAM, and if you run into an OOM issue where swap would be your parachute, you're already fucked.
- everybodyknows 7y agoOP serves up animated ads directly from the origin URL -- unblockable by umatrix. Very sophisticated indeed.
- LinuxBender 7y agoAlmost never add swap. [1] [1] - https://github.com/ohdns/sysctl_and_thp_test https://github.com/ohdns/sysctl_and_thp_test
- yellowapple 7y agoOne thing the article doesn't mention for those of us who run Linux on laptops (There are dozens of us! Dozens!) is that having a swap partition at least as big as one's physical RAM (if not bigger) is mandatory for hibernation. Linux (like any OS kernel that supports hibernation) needs to save the current contents of RAM somewhere, and (last I checked) it'll only work with swap partitions (swap files don't work for some reason).