8 ms·

Yeah, no way. As soon as you hit swap, _most_ apps are going to have a bad, bad time. This is well known, so much so that all EC2 instances in AWS disable it by

• gchamonlive 1y ago
  How programs use ram also changed from the 90s. Back then they were written targeting machines that they knew would have a hard time fitting all their data in memory, so hitting swap wouldn't hurt perceived performance too drastically since many operations were already optimized to balance data load between memory and disk. Nowadays when a program hits swap it's not going to fallback to a different memory usage profile that prioritises disk access. It's going to use swap as if it were actual ram, so you get to see the program choking the entire system.
  • winrid 1y ago
    Exactly. Nowadays, most web services are run in a GC'ed runtime. That VM will walk pointers all over the place and reach into swap all the time.
    • cogman10 1y ago
      Depends entirely on the runtime. If your GC is a moving collector, then absolutely this is something to watch out for. There are, however, a number of runtimes that will leave memory in place. They are effectively just calling `malloc` for the objects and `free` when the GC algorithm detects an object is dead. Go, the CLR, Ruby, Python, Swift, and I think node(?) all fit in this category. The JVM has a moving collector.
      • zozbot234 1y ago
        Every garbage collector has to constantly sift through the entire reference graph of the running program to figure out what objects have become garbage. Generational GC's can trace through the oldest generations less often, but that's about it. Tracing garbage collectors solve a single problem really really well - managing a complex, possibly cyclical reference graph, which is in fact inherent to some problems where GC is thus irreplaceable - and are just about terrible wrt. any other system-level or performance-related factor of evaluation.
        • cogman10 1y ago
          > Every garbage collector has to constantly sift through the entire reference graph of the running program to figure out what objects have become garbage. There's a lot of "it depends" here. For example, an RC garbage collector (Like swift and python?) doesn't ever trace through the graph. The reason I brought up moving collectors is by their nature, they take up a lot more heap space, at least 2x what they need. The advantage of the non-moving collectors is they are much more prompt at returning memory to the OS. The JVM in particular has issues here because it has pretty chunky objects.
          • Dylan16807 1y ago
            > The reason I brought up moving collectors is by their nature, they take up a lot more heap space, at least 2x what they need. If the implementer cares about memory use it won't. There are ways to compact objects that are a lot less memory-intensive than copying the whole graph from A to B and then deleting A.
        • eru 1y ago
          Modern garbage collectors have come a long way. Even not so modern ones: have you heard of generational garbage collection? But even in eg Python they introduced 'immortal objects' which the GC knows not to bother with.
          • winrid 1y ago
            It doesn't matter. The GC does not know what heap allocations are in memory vs swap, and since you don't write applications thinking about that, running a VM with a moving GC on swap is a bad idea.
      • eru 1y ago
        A moving GC should be better at this, because it can compact your memory.
        • cogman10 1y ago
          A moving collector has to move to somewhere and, generally by it's nature, it's constantly moving data all across the heap. That's what makes it end up touching a lot more memory while also requiring more memory. On minor collections I'll move memory between 2 different locations and on major collections it'll end up moving the entire old gen. It's that "touching" of all the pages controlled by the GC that ultimately wrecks swap performance. But also the fact that moving collector like to hold onto memory as downsizing is pretty hard to do efficiently. Non-moving collectors are generally ultimately using C allocators which are fairly good at avoiding fragmentation. Not perfect and not as fast as a moving collector, but also fast enough for most use cases. Java's G1 collector would be the worst example of this. It's constantly moving blocks of memory all over the place.
          • eru 1y ago
            > It's that "touching" of all the pages controlled by the GC that ultimately wrecks swap performance. But also the fact that moving collector like to hold onto memory as downsizing is pretty hard to do efficiently. The memory that's now not in use, but still held onto, can be swapped out.
      • manwe150 1y ago
        MemBalancer is a relatively new analysis paper that argues having swap allows maximum performance by allowing small excesses, that avoids needing to over-provision ram instead. The kind of gc does not matter since data spends very little time in that state and on the flip side, most of the time the application has twice has access to twice as much memory to use
      • masklinn 1y ago
        Python’s not a mover but the cycle breaker will walk through every object in the VM. Also since the refcounts are inline, adding a reference to a cold object will update that object. IIRC Swift has the latter issue as well (unless the heap object’s RC was moved to the side table).
  • zoeysmithe 1y ago
    This is really interesting and I've never really heard about this. What is going on with the kernel team then? Are they just going to keep swap as-is for backwards compatibility then everyone else just disables it? Or if this advice just for high performance clusters?
    • kccqzy 1y ago
      No. I use swap for my home machines. Most people should leave swap enabled. In fact I recommend the setup outlined in the kernel docs for tmpfs: https://docs.kernel.org/filesystems/tmpfs.html https://docs.kernel.org/filesystems/tmpfs.html which is to have a big swap and use tmpfs for /tmp and /var/tmp.
    • gchamonlive 1y ago
      As someone else said, swap is important not only in the case the system exhaust main memory, but it's used to efficiently use system memory before that (caching, offload page blocks to swap that aren't frequently used etc...)
• henryfjordan 1y ago
  Does HDD vs SSD matter at all these days? I can think of certain caching use-cases where swapping to an SSD might make sense, if the access patterns were "bursty" to certain keys in the cache
  • winrid 1y ago
    It's still extremely slow and can cause very unpredictable performance. I have swap setup with swappiness=1 on some boxes, but I wouldn't generally recommend it.
    • eru 1y ago
      HDDs are much, much slower than SSD. If swapping to SSD is 'extremely slow', what's your term for swapping to HDD?
      • baq 1y ago
        ‘Hard reboot’ (not OP)
• LaurensBER 1y ago
  The beauty of ZRAM is that on any modern-ish CPU it's surprisingly fast. We're talking 2-3 ms instead of 2-3 seconds ;) I regularly use it on my Snapdragon 870 tablet (not exactly a top of the line CPU) to prevent OOM crashes (it's running an ancient kernel and the Android OOM killer basically crashes the whole thing) when running a load of tabs in Brave and a Linux environment (through Tmux) at the same time. ZRAM won't save you if you do actually need to store and actively use more than the physical memory but if 60% of your physical memory is not actively used (think background tabs or servers that are running but not taking requests) it absolutely does wonders! On most (web) app servers I happily leave it enabled to handle temporary spikes, memory leaks or applications that load a whole bunch of resources that they never ever use. I'm also running it on my Kubernetes cluster. It allows me to set reasonable strict memory limits while still having the certainty that Pods can handle (short) spikes above my limit.
  • geokon 1y ago
    My understanding was that if you're doing random access - ZRAM has near-zero overhead. While data is being fetched from RAM, you have enough cycles to decompress blocks. Would love to be corrected if I'm wrong
• bayindirh 1y ago
  This is a wrong belief because a) SSDs make swap almost invisible, so you can have that escape ramp if something goes wrong b) SWAP space is not solely an escape ramp which RAM overflows into anymore. In the age of microservices and cattle servers, reboot/reinstall might be cheap, but in the long run it is not. A long running server, albeit being cattle, is always a better solution because esp. with some excess RAM, the server "warms up" with all hot data cached and will be a low latency unit in your fleet, given you pay the required attention to your software development and service configuration. Secondly, Kernel swaps out unused pages to SWAP, relieving pressure from RAM. So, SWAP is often used even if you fill 1% of your RAM. This allows for more hot data to be cached, allowing better resource utilization and performance in the long run. So, eff it, we ball is never a good system administration strategy. Even if everything is ephemeral and can be rebooted in three seconds. Sure, some things like Kubernetes forces "no SWAP, period" policies because it kills pods when pressure exceeds some value, but for more traditional setups, it's still valuable.
  • gchamonlive 1y ago
    > SSDs make swap almost invisible It doesn't. SSDs came a long way but so did memory dies and buses, and with that the way programs work also changed as more and more they are able to fit their stacks and heaps on memory more often than not. I have had a problem with shellcheck that for some reason eats up all my ram when I open I believe .zshrc and trust me, it's not invisible. The system crawls to a halt.
    • bayindirh 1y ago
      It depends on the SSD, I may say. If we're talking about SATA SSDs which top at 600MBps, then yes, an aggressive application can make itself known. However, if you have a modern NVMe, esp. a 4x4 one like Samsung 9x0 series or if you're using a Mac, I bet you'll notice the problem much later, if ever. Remember the SSD trashing problem on M1 Macs? People never noticed that system used SWAP that heavily and trashed the SSD on board. Then, if you're using a server with a couple of SAS or NVMe SSDs, you'll not notice the problem again, esp. if these are backed by RAID (even md counts).
• 01HNNWZ0MV43FF 1y ago
  It's not just 3 seconds for a button click, every time I've run out of RAM on a Linux system, everything locks up and it thrashes. It feels like 100x slowdown. I've had better experiences when my CPU was underclocked to 20% speed. I enable swap and install earlyoom. Let processes die, as long as I can move the mouse and operate a terminal.
  • C7E69B041F 1y ago
    This, I'm used to restarting my Plasma 2 times a day cause PHPStorm just leaks memory and it eventually crashes and requires hard reboot.
  • zozbot234 1y ago
    > It feels like 100x slowdown. Yup, this is a thing. It happens because file-backed program text and read-only data eventually get evicted from RAM (to make room for process memory) so every access to code and/or data beyond the current 4K page can potentially involve a swap-in from disk. It would be nice if we had ways of setting up the system so that pages of code or data that are truly critical for real-time responsiveness (including parts of the UI) could not get evicted from RAM at all (except perhaps to make room for the OOM reaper itself to do its job) - but this is quite hard to do in practice.
• KaiserPro 1y ago
  Yeahna, thats just memory exhaustion. Swap helps you use ram more efficiently, as you put the hot stuff in swap and let the rest fester on disk. Sure if you overwhelm it, then you're gonna have a bad day, but thats the same without swap. Seriously, swap is good, don't believe the noise.
  • gchamonlive 1y ago
    It's good, and Aws shouldn't disable it by default, but it won't save the system from OOM.
    • matt-p 1y ago
      I bet there's a big "burns through our SSDs faster" spreadsheet column or similar that caused it to be disabled.
      • gchamonlive 1y ago
        Maybe. Or maybe it's an arbitrary decision. Many won't enable swap. For some swap wouldn't help anyways, but others it could help soak up spikes. The latter in some cases will upgrade to a larger instance without even evaluating if swap could help, generating AWS more money. Either way it's far-fetched to derive intention from the fact.
  • adastra22 1y ago
    I don’t understand. If you provision the system with enough RAM, then you can for every page in RAM, hot or not.
    • akvadrako 1y ago
      Only if you have more RAM than disk space, which is wasteful for many applications.
      • adastra22 1y ago
• zymhan 1y ago
  Where on earth did you get this misconception?
  • commandersaki 1y ago
    Lived experience? With swap system stays up but is unresponsive, without it is either responsive due to oom kill or completely down.
    • GuinansEyebrows 1y ago
      in either case, what do you do? if you can't reach a box and it's otherwise safe to do so, you just reboot it. so is it just a matter of which situation occurs more often?
      • commandersaki 1y ago
        The thing is you can survive memory exhaustion if the oom killer can do its job, which it can't many times when there's swap. I guess the topmost response to this thread talks about an earlyoom tool that might alleivate this, but I've never used it, and I don't find swap helpful anyway so there's no need for me to go down this route.
• akerl_ 1y ago
  Is it possible you misread the comment you're replying to? They aren't recommending adding swap, they're recommending adjusting the memory tunables to make the OOM killer a bit more aggressive so that it starts killing things before the whole server goes to hell.
• the8472 1y ago
  YMMV. Garbage-collected/pointer-chasing languages suffer more from swapping because they touch more of the heap all the time. AWS suffers more from swap because EBS is ridiculously slow and even their instance-attached NVMe is capped compared physical NVMe sticks.
• elwebmaster 1y ago
  what an ignorant and clueless comment. Guess what? Todays disks are NVMe drives which are orders of magnitude faster than the 5400rpm HDDs of the 90s. Today's swap is 90s RAM.
• slyall 1y ago
  My 2cents is that in a lot of cases swap is being used for unimportant stuff leave more RAM for your app. Do a "ps aux" and look at all the RAM used by weird stuff. Good news is those things will be swapped out. Example on my personal VPS $ free -m total used free shared buff/cache available Mem: 3923 1225 328 217 2369 2185 Swap: 1535 1335 200
• deleted 1y ago
  [deleted]
• Dylan16807 1y ago
  "as soon as you hit swap" is a bad way of looking at things. Looking around at some servers I run, most of them have .5-2GB of swap used despite a bunch of gigabytes of free memory. That data is never or almost never going to be touched, and keeping it in memory would be a waste. On a smaller server that can be a significant waste. Swap is good to have. The value is limited but real. Also not having swap doesn't prevent thrashing, it just means that as memory gets completely full you start dropping and re-reading executable code over and over. The solution is the same in both cases, kill programs before performance falls off a cliff. But swap gives you more room before you reach the cliff.
• goodpoint 1y ago
  No, swap is absolutely fine if used correctly.
• Hendrikto 1y ago
  > This is well known But also false. Swap is there so anonymous pages can be evicted. Not as a “slow overflow for RAM”, as a lot of people still believe. By disabling swap you can actually *increase* thrashing, because the kernel is more limited in what it can do with the virtual memory.
• AlexandrB 1y ago
  > Maybe back in the 90s, it was okay to wait 2-3 seconds for a button click, but today we just assume the thing is dead and reboot. My experience is the exact opposite. If anything 2-3 second button clicks are more common than ever today since everything has to make a roundtrip to a server somewhere whereas in the 90s 2-3s button click meant your computer was about to BSOD. Edit: Apple recently brought "2-3s to open tab" technology to Safari[1]. [1] https://old.reddit.com/r/MacOS/comments/1nm534e/sluggish_safari_26_when_opening_new_tabs_or/ https://old.reddit.com/r/MacOS/comments/1nm534e/sluggish_saf...