5 ms·

This is bad advice when the swap is backed by spinning disk rather than SSD, in large part because paging in once the problem is solved is not aggressive enough

This is bad advice when the swap is backed by spinning disk rather than SSD, in large part because paging in once the problem is solved is not aggressive enough. Let me make it concrete: say 1 GiB of your application's RAM gets swapped out then is later paged in only as needed (4 KiB pages) with no readahead. Now the application's VM space is a minefield: there are up to 262,144 times it can stall for a ~10 ms disk seek (for a total of ~40 minutes). Sequentially reading in 1 GiB of RAM from disk would take only ~5 seconds. Hopefully OSs use some readahead, but I'm sure Linux and macOS don't use enough. I find a system that has ever swapped to be totally unusable until I do "sudo swapoff -a" (on Linux) to force everything to be paged in, or just reboot (on macOS, I haven't found any other way). Some of this can happen even without swap: the OS will still drop clean file-backed pages, so unless you mlock() your executables after startup (my production binary at work does this), you can still have major page fault-induced latency spikes. Swap backed by SSD (or compressed RAM) should be more reasonable, but I've had enough bad experiences with swap on spinning disk that authoritative-sounding articles that encourage swap without mentioning this problem piss me off. In contrast, if you don't have swap and run out of RAM, something will die and get restarted. In many cases this is a much better failure mode than continuing to run slowly.

uaygsfdbzf 7y ago
Author said almost always. Nothing is absolute in cases like this. I agree with the article. We enable swap on our production servers precisely because the engineers don't rectify their memory leaks or whatever problems are internal to their software. Some of those engineers are our own and we can't convince them to fix their software. Others are for things we have no control over, such as Kafka, for example. When Kafka starts going out of control and you've got no swap, oom killer will kill it and you'll end up with corrupted index files, which is a bigger problem than the performance problem you'll see if it has to swap for a while until it gets itself sorted out.
magicalhippo 7y ago
I've had the opposite experience on Windows. I always put swap on HDD, because I've been bitten several times by memory leaking applications rendering the entire OS unusable until the SSD with the swap file ran out of space. Windows would happily grow the pagefile beyond the limits I had set, and if the allocation loop was tight enough the OS chocked on handling all page faults. As in using literally minutes to respond to keystrokes. By putting swap on a slow HDD, Windows has plenty of time to service mouse and keyboard input while waiting for the disk to respond, allowing me to kill the offending application before saturating my disk. I assume this behavior was OK back when computers had very limited memory, and disks were slow anyway. But these days, if a single application wants more than the available memory (physical + pagefile), I'd rather the OS just said no to the application.
- scottlamb 7y ago
  I don't use Windows, but it sounds like you can't you set limits on how large the swapfile can get? That's crazy! > But these days, if a single application wants more than the available memory (physical + pagefile), I'd rather the OS just said no to the application. I agree.
  - magicalhippo 7y ago
    Oh you can set limits. And then Windows will promptly ignore them, at least under certain circumstances. Why I have no idea.
raverbashing 7y ago
Swap sucks, but OOM killer sucks more. It certainly isn't a great solution, but it's good for temporary spikes. You can adjust "swappiness" if it's moving stuff to disk too aggressively.
- scottlamb 7y ago
  > Swap sucks, but OOM killer sucks more. It certainly isn't a great solution, but it's good for temporary spikes. I disagree. The OOM killer usually kills the right thing, and if it doesn't a quick restart is usually better than continuing to run slowly. You can also just put things in containers and set per-container limits. > You can adjust "swappiness" if it's moving stuff to disk too aggressively. What I want is to adjust un-swappiness: aggressiveness of paging stuff back in. Once the memory spike is over, it should page everything back in automatically.
  - mchristen 7y ago
    The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen. Containers don't really solve your problem, it just adds a layer of indirection between the processes hogging RAM and the OOM killer, which in my experience adds even more uncertainty into what the OOM killer is going to do.
    - VectorLock 7y ago
      Tuning oomkiller's hitlist makes it much more managable and the new BPF tools to track it make it a ton nicer than a system grinding into swap.
    - scottlamb 7y ago
      > The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen. I won't say there's no circumstance in which the OOM killer is worse, but in general I disagree with you. A stateful server application: maybe a standard ACID DBMS like PostgreSQL. I'd much rather it get killed (failing whatever transactions were in progress) and restart in a healthy state than continue to run slowly. Here I'm relying on a correct implementation of Durability, but if you don't have that, you have other problems. Even more so for a quorum-based database server in which the leader dying (and another taking over) is much better than continuing to run slowly. And yes, even more so in stateless servers, particularly when there are several of them. Going down and restarting quickly (a brief loss of capacity, a brief spike of errors if the client doesn't retry) is a lot better than continuing to be slow (a sustained loss of capacity, possibly leading to cascading overload; and user-visible latency problems if there's no hedging). For something like a desktop application, it's debatable, but personally I'd still rather endure the up-front nuisance of having to restart it than have the OS try to paper over it until I finally realize why it's so painful and do something about it. (Insert anecdote about frog boiling here; I hear frogs actually do jump out before they boil, but yet somehow the story still rings true, if you know what I mean.)
Johnny555 7y ago
Also bad when it's backed by an attached drive from the cloud provider, like an AWS EBS volume -- it's particularly nefarious if you swap to the root drive, swapping can exhaust the EBS burst balance, and instead of 3000 IOPS to the root drive, after swapping sets in you're down to 300 IOPS (for a 100GB drive), and performance really slows to a crawl. I know I'm giving up some RAM that could be swapped out without impact but it's worth the stability of never swapping to an attached drive.

uaygsfdbzf 7y ago

Author said almost always. Nothing is absolute in cases like this. I agree with the article. We enable swap on our production servers precisely because the engineers don't rectify their memory leaks or whatever problems are internal to their software. Some of those engineers are our own and we can't convince them to fix their software. Others are for things we have no control over, such as Kafka, for example. When Kafka starts going out of control and you've got no swap, oom killer will kill it and you'll end up with corrupted index files, which is a bigger problem than the performance problem you'll see if it has to swap for a while until it gets itself sorted out.

magicalhippo 7y ago

I've had the opposite experience on Windows. I always put swap on HDD, because I've been bitten several times by memory leaking applications rendering the entire OS unusable until the SSD with the swap file ran out of space. Windows would happily grow the pagefile beyond the limits I had set, and if the allocation loop was tight enough the OS chocked on handling all page faults. As in using literally minutes to respond to keystrokes. By putting swap on a slow HDD, Windows has plenty of time to service mouse and keyboard input while waiting for the disk to respond, allowing me to kill the offending application before saturating my disk. I assume this behavior was OK back when computers had very limited memory, and disks were slow anyway. But these days, if a single application wants more than the available memory (physical + pagefile), I'd rather the OS just said no to the application.

scottlamb 7y ago
I don't use Windows, but it sounds like you can't you set limits on how large the swapfile can get? That's crazy! > But these days, if a single application wants more than the available memory (physical + pagefile), I'd rather the OS just said no to the application. I agree.
- magicalhippo 7y ago
  Oh you can set limits. And then Windows will promptly ignore them, at least under certain circumstances. Why I have no idea.

raverbashing 7y ago

Swap sucks, but OOM killer sucks more. It certainly isn't a great solution, but it's good for temporary spikes. You can adjust "swappiness" if it's moving stuff to disk too aggressively.

scottlamb 7y ago
> Swap sucks, but OOM killer sucks more. It certainly isn't a great solution, but it's good for temporary spikes. I disagree. The OOM killer usually kills the right thing, and if it doesn't a quick restart is usually better than continuing to run slowly. You can also just put things in containers and set per-container limits. > You can adjust "swappiness" if it's moving stuff to disk too aggressively. What I want is to adjust un-swappiness: aggressiveness of paging stuff back in. Once the memory spike is over, it should page everything back in automatically.
- mchristen 7y ago
  The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen. Containers don't really solve your problem, it just adds a layer of indirection between the processes hogging RAM and the OOM killer, which in my experience adds even more uncertainty into what the OOM killer is going to do.
  - VectorLock 7y ago
    Tuning oomkiller's hitlist makes it much more managable and the new BPF tools to track it make it a ton nicer than a system grinding into swap.
  - scottlamb 7y ago
    > The OOM killer is just fine for stateless server applications, but pretty much anything else and it's a disaster waiting to happen. I won't say there's no circumstance in which the OOM killer is worse, but in general I disagree with you. A stateful server application: maybe a standard ACID DBMS like PostgreSQL. I'd much rather it get killed (failing whatever transactions were in progress) and restart in a healthy state than continue to run slowly. Here I'm relying on a correct implementation of Durability, but if you don't have that, you have other problems. Even more so for a quorum-based database server in which the leader dying (and another taking over) is much better than continuing to run slowly. And yes, even more so in stateless servers, particularly when there are several of them. Going down and restarting quickly (a brief loss of capacity, a brief spike of errors if the client doesn't retry) is a lot better than continuing to be slow (a sustained loss of capacity, possibly leading to cascading overload; and user-visible latency problems if there's no hedging). For something like a desktop application, it's debatable, but personally I'd still rather endure the up-front nuisance of having to restart it than have the OS try to paper over it until I finally realize why it's so painful and do something about it. (Insert anecdote about frog boiling here; I hear frogs actually do jump out before they boil, but yet somehow the story still rings true, if you know what I mean.)

Johnny555 7y ago

Also bad when it's backed by an attached drive from the cloud provider, like an AWS EBS volume -- it's particularly nefarious if you swap to the root drive, swapping can exhaust the EBS burst balance, and instead of 3000 IOPS to the root drive, after swapping sets in you're down to 300 IOPS (for a 100GB drive), and performance really slows to a crawl. I know I'm giving up some RAM that could be swapped out without impact but it's worth the stability of never swapping to an attached drive.