4 ms·

I was hoping this post was going to say something like turning on some small amount of swap will allow the OS to cache some things that normally it wouldn't cac

I was hoping this post was going to say something like turning on some small amount of swap will allow the OS to cache some things that normally it wouldn't cache at all, and so you get some improvement there. What's actually in this post doesn't make any sense to me. If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping. First, my production servers are not doing many other things besides being production servers. It's not like they are running a bunch of unnecessary services, and if I kill some then my application can recover. If a process is out of control then it's almost assuredly something important that I'm going to have to kill anyway. Second, I find it's much harder to detect degraded performance than it is to detect a dead process. It's very, very easy to have a health check that will detect a host who has stopped listening and drop that host from the LB. And alerting on that scenario is very easy as well. The alternative is a host that's operating in a degraded state, which I need to detect with more sophisticated health check + alerting, and in the end my resolution is just going to be to kill everything anyway. In a properly designed HA environment the loss of a host should be no big deal. Architecture should be focused on making sure a host goes down ASAP if it's having problems, not letting it survive in some kind of zombie state.

RoutinePlayer 7y ago
Awesome comment! Especially as it relates to VMs in modern cloud environments. There really is no need for swap space, and letting them fail quickly is usually best. Using swap space may make sense for a traditional physical, stand-alone server, where having it fail quickly doesn't really help. Or maybe for a workstation.
deathanatos 7y ago
> If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. The memory freed from a page that got pushed to swap might be better used for serving disk cache, for example. If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when they do become busy, you won't have to wait. At least, that's the theory as I understand it. Whether or not it works in practice, IDK. > And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping. I have successfully recovered hosts that began swapping, and even hosts that filled their swap. But it's beyond painful, and in most of the cases where I have a host doing such swapping, an OOM kill would have been a welcome reprieve. > Second, I find it's much harder to detect degraded performance than it is to detect a dead process. Agreed, and restarting something that's been OOM'd is simple enough too. Our hosts tend to stop reporting metrics when they start swapping, simply b/c so little is actually getting done, so they get labelled as "down". Linux also exposes a metric called "major page faults" (IIRC, it's "pgmajfault" in /proc/vmstat) that records the number of major (required service from disk) page faults; if the rate of that value is too steep for too long, that ≅ swap thrashing.
- mnw21cam 7y ago
  The key point is that if you have no swap, and you run low on memory, then the OS has no choice but to throw away stuff that is clean. This includes stuff that is being actively used right now, like the code that is in your processes. So your system will still thrash, even though it has no swap, because it is trying to load pages of running code in on-demand, while constantly throwing them away again because it doesn't have enough space. This will incapacitate a server even more thoroughly than if there is swap, and the working set is larger than physical memory. The more RAM a server has, the more severe this incapacitation will be. For this reason, you want an OOM killer to kill stuff way earlier than when you actually run out of RAM, because if you ever reach that point it is too late. I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects.
  - scottlamb 7y ago
    > I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects. Along those lines, take a look at this: [1] > Pressure stall information were added to Linux 4.20 as a way to quantify resource pressure in the system in a better way than the traditional load average. PSI aggregates and reports the overall wallclock time in which the tasks in a system (or cgroup) wait for cpu, io or memory. > This release [Linux 5.2] lets users to configure sensitive thresholds and use poll() and friends to be notified when a certain pressure thresold is breached within a user-defined time window. With this mechanism, Android can monitor for, and ward off, mounting memory shortages before they cause problems for the user. For example, using memory stall, monitors in userspace like the low memory killer daemon (lmkd) can detect mounting pressure and kill less important processes before device becomes visibly sluggish. In memory stress testing psi memory monitors produce roughly 10x less false positives compared to vmpressure. I haven't tried it yet, but it sounds promising. [1] https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_Information_for_better_resource_monitoring https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_I...
- 7y ago
wrycoder 7y ago
From TFA: "The Linux Kernel will move memory pages which are hardly ever used into swap space to ensure that even more cachable space is made available in-memory for more frequently used memory pages (a page is a piece of memory). Swap usage becomes a performance problem when the Kernel is pressured to continuously move memory pages in and out of memory and swap space."
- mason55 7y ago
  I still don't get how that answers the question. If I have enough RAM for both the "hardly ever used" pages and the "more frequently used" pages then what good is the swap doing me? The only way that makes sense is if there is some cache the kernel has which is faster than RAM. BUT the kernel cannot move things between this cache and RAM, it can only treat the cache + RAM as a single blob, so to free up space in the cache the kernel has to move pages to disk. So by having some swap space the kernel can move hardly ever used pages to disk which is the only way to free up space in this cache. It's possible that what I just described really exists, I don't know that much about the internals of how the kernel manages memory. But if that is what's happening then THAT is the detail that would have been useful in this post.
  - wrycoder 7y ago
    It does work that way. I had a small, very lightly loaded VPS that had no swap. Occasionally, it would not start all its services on boot, in particular the db, due to OOM. Adding a small amount of swap allowed it to boot cleanly. The swap was not subsequently used. The alternative was to upgrade the server RAM at double the monthly cost.
erik_seaberg 7y ago
Redundancy helps with uncorrelated failures, but if you start getting requests that cause memory pressure and OOM kills, you could lose the entire cluster. I agree that swap is often too slow to help, but "self-destruct and hope the rest of the cluster is healthy right now" is not a good failure mode. Don't overcommit in prod.
- mason55 7y ago
  Sure, I agree with that 100%. I guess if you had a scenario where you were operating near your limits and something happened that caused one or a few hosts to start swapping, and because they were up the LB kept sending them traffic instead of redirecting it to other hosts, then you could avoid a thundering herd-type problem. But that still relies on uncorrelated failures. Otherwise that seems like an orthogonal discussion to swap vs. OOM killer. It's something to be aware of and you need circuit breakers in place but I don't think it's an argument for letting hosts swap instead of having the OOM killer deal with it.
gnode 7y ago
> The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. Putting rarely used (or never to be used again) pages in swap frees up physical RAM to be used as disk cache, which can improve performance for file access. If you have enough RAM to cover all application memory, and cache your entire filesystem, there is no gain to be had. I agree that slowing down low memory scenarios is silly though. In many cases killing a process is easier to deal with than unexpected poor performance. The slow death of out of memory systems doesn't seem to be unique to having a swapfile though; Linux seems to discard any recoverable pages (such as application / library code backed by files on disc) in the run-up to OOM, resulting in thrashing. Ideally I'd want to have swap, but automatically kill processes when performance deteriorates due to swapping, rather than when the system is strictly out of memory.
pfranz 7y ago
> If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. One thing I think the author was getting at, but never explicitly stated is that we spec most things for the average expected workload with a bit of headroom--not the peak workload since that is very difficult to assess. Peak workload might also be few and far between and not worth paying for upfront. Swap allows you to deal with peak workloads more gracefully for many applications. Having your process randomly die isn't a big deal with stateless processes (which is a more and more common design), but really sucks for stateful things. Not just the possibility of lost/corrupt state, but process startup and loading of state can take a long time. We have a few servers that don't have swap and I was hoping to find an excuse to add it in this article. Instead, it reenforced that decision because of type of things we run on it.
- eikenberry 7y ago
  > Swap allows you to deal with peak workloads more gracefully for many applications. This is almost always false in practice though. Peak workloads means more than just lots of memory, it means high CPU and high IO (disk and network). If you under-spec your servers such that they need to hit swap during peak loads they are going to generally cause more problems than if they just die. A server dying is something you have to prepare for and deal with anyways, and is much simpler to plan for than the weird latency issues you'll get with a server hits swap. Think of it as similar to Erlang's fail-fast model. Better for things to just die and be recovered than to continue on in a bad state.
  - pfranz 7y ago
    I'm not really sure what you're getting at. You and I are both emphasizing a different behavior for different classes of applications. I feel like my comment and the article were pretty clear about that. I've had the opposite experience than "almost always false in practice" but my domain is fairly different than most here so I try and be very careful when generalizing. If your hardware is underspeced for the workload, there's still a business case to be made before you buy new hardware. Sometimes the preferred failure mode is to slow down, other times it's preferred to die and restart. Hardware speced for peak loads can sometimes cost multiple times otherwise appropriate hardware...assuming you even know what peak loads will be. I have one class of application that is fairly idempotent, we don't have hard drives on these machines, so we don't use swap and expect it to die if it exceeds the amount of memory requested. This article reenforced this. I have another class of application that queues up things to process. The specs work on the current hardware 99% of the time, but on holiday weekends when large projects are delivering things slow down because work is submitted and people leave so nobody cares. Reloading the state from disk can take 20m+ and without swap if you were to exceed memory you're dead in the water until you purchase more hardware. Our contingency hardware has identical specs as the original.

RoutinePlayer 7y ago

Awesome comment! Especially as it relates to VMs in modern cloud environments. There really is no need for swap space, and letting them fail quickly is usually best. Using swap space may make sense for a traditional physical, stand-alone server, where having it fail quickly doesn't really help. Or maybe for a workstation.

deathanatos 7y ago

> If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. The memory freed from a page that got pushed to swap might be better used for serving disk cache, for example. If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when they do become busy, you won't have to wait. At least, that's the theory as I understand it. Whether or not it works in practice, IDK. > And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping. I have successfully recovered hosts that began swapping, and even hosts that filled their swap. But it's beyond painful, and in most of the cases where I have a host doing such swapping, an OOM kill would have been a welcome reprieve. > Second, I find it's much harder to detect degraded performance than it is to detect a dead process. Agreed, and restarting something that's been OOM'd is simple enough too. Our hosts tend to stop reporting metrics when they start swapping, simply b/c so little is actually getting done, so they get labelled as "down". Linux also exposes a metric called "major page faults" (IIRC, it's "pgmajfault" in /proc/vmstat) that records the number of major (required service from disk) page faults; if the rate of that value is too steep for too long, that ≅ swap thrashing.

mnw21cam 7y ago
The key point is that if you have no swap, and you run low on memory, then the OS has no choice but to throw away stuff that is clean. This includes stuff that is being actively used right now, like the code that is in your processes. So your system will still thrash, even though it has no swap, because it is trying to load pages of running code in on-demand, while constantly throwing them away again because it doesn't have enough space. This will incapacitate a server even more thoroughly than if there is swap, and the working set is larger than physical memory. The more RAM a server has, the more severe this incapacitation will be. For this reason, you want an OOM killer to kill stuff way earlier than when you actually run out of RAM, because if you ever reach that point it is too late. I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects.
- scottlamb 7y ago
  > I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects. Along those lines, take a look at this: [1] > Pressure stall information were added to Linux 4.20 as a way to quantify resource pressure in the system in a better way than the traditional load average. PSI aggregates and reports the overall wallclock time in which the tasks in a system (or cgroup) wait for cpu, io or memory. > This release [Linux 5.2] lets users to configure sensitive thresholds and use poll() and friends to be notified when a certain pressure thresold is breached within a user-defined time window. With this mechanism, Android can monitor for, and ward off, mounting memory shortages before they cause problems for the user. For example, using memory stall, monitors in userspace like the low memory killer daemon (lmkd) can detect mounting pressure and kill less important processes before device becomes visibly sluggish. In memory stress testing psi memory monitors produce roughly 10x less false positives compared to vmpressure. I haven't tried it yet, but it sounds promising. [1] https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_Information_for_better_resource_monitoring https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_I...
7y ago

wrycoder 7y ago

From TFA: "The Linux Kernel will move memory pages which are hardly ever used into swap space to ensure that even more cachable space is made available in-memory for more frequently used memory pages (a page is a piece of memory). Swap usage becomes a performance problem when the Kernel is pressured to continuously move memory pages in and out of memory and swap space."

mason55 7y ago
I still don't get how that answers the question. If I have enough RAM for both the "hardly ever used" pages and the "more frequently used" pages then what good is the swap doing me? The only way that makes sense is if there is some cache the kernel has which is faster than RAM. BUT the kernel cannot move things between this cache and RAM, it can only treat the cache + RAM as a single blob, so to free up space in the cache the kernel has to move pages to disk. So by having some swap space the kernel can move hardly ever used pages to disk which is the only way to free up space in this cache. It's possible that what I just described really exists, I don't know that much about the internals of how the kernel manages memory. But if that is what's happening then THAT is the detail that would have been useful in this post.
- wrycoder 7y ago
  It does work that way. I had a small, very lightly loaded VPS that had no swap. Occasionally, it would not start all its services on boot, in particular the db, due to OOM. Adding a small amount of swap allowed it to boot cleanly. The swap was not subsequently used. The alternative was to upgrade the server RAM at double the monthly cost.

erik_seaberg 7y ago

Redundancy helps with uncorrelated failures, but if you start getting requests that cause memory pressure and OOM kills, you could lose the entire cluster. I agree that swap is often too slow to help, but "self-destruct and hope the rest of the cluster is healthy right now" is not a good failure mode. Don't overcommit in prod.

mason55 7y ago
Sure, I agree with that 100%. I guess if you had a scenario where you were operating near your limits and something happened that caused one or a few hosts to start swapping, and because they were up the LB kept sending them traffic instead of redirecting it to other hosts, then you could avoid a thundering herd-type problem. But that still relies on uncorrelated failures. Otherwise that seems like an orthogonal discussion to swap vs. OOM killer. It's something to be aware of and you need circuit breakers in place but I don't think it's an argument for letting hosts swap instead of having the OOM killer deal with it.

gnode 7y ago

> The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. Putting rarely used (or never to be used again) pages in swap frees up physical RAM to be used as disk cache, which can improve performance for file access. If you have enough RAM to cover all application memory, and cache your entire filesystem, there is no gain to be had. I agree that slowing down low memory scenarios is silly though. In many cases killing a process is easier to deal with than unexpected poor performance. The slow death of out of memory systems doesn't seem to be unique to having a swapfile though; Linux seems to discard any recoverable pages (such as application / library code backed by files on disc) in the run-up to OOM, resulting in thrashing. Ideally I'd want to have swap, but automatically kill processes when performance deteriorates due to swapping, rather than when the system is strictly out of memory.

pfranz 7y ago

> If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. One thing I think the author was getting at, but never explicitly stated is that we spec most things for the average expected workload with a bit of headroom--not the peak workload since that is very difficult to assess. Peak workload might also be few and far between and not worth paying for upfront. Swap allows you to deal with peak workloads more gracefully for many applications. Having your process randomly die isn't a big deal with stateless processes (which is a more and more common design), but really sucks for stateful things. Not just the possibility of lost/corrupt state, but process startup and loading of state can take a long time. We have a few servers that don't have swap and I was hoping to find an excuse to add it in this article. Instead, it reenforced that decision because of type of things we run on it.

eikenberry 7y ago
> Swap allows you to deal with peak workloads more gracefully for many applications. This is almost always false in practice though. Peak workloads means more than just lots of memory, it means high CPU and high IO (disk and network). If you under-spec your servers such that they need to hit swap during peak loads they are going to generally cause more problems than if they just die. A server dying is something you have to prepare for and deal with anyways, and is much simpler to plan for than the weird latency issues you'll get with a server hits swap. Think of it as similar to Erlang's fail-fast model. Better for things to just die and be recovered than to continue on in a bad state.
- pfranz 7y ago
  I'm not really sure what you're getting at. You and I are both emphasizing a different behavior for different classes of applications. I feel like my comment and the article were pretty clear about that. I've had the opposite experience than "almost always false in practice" but my domain is fairly different than most here so I try and be very careful when generalizing. If your hardware is underspeced for the workload, there's still a business case to be made before you buy new hardware. Sometimes the preferred failure mode is to slow down, other times it's preferred to die and restart. Hardware speced for peak loads can sometimes cost multiple times otherwise appropriate hardware...assuming you even know what peak loads will be. I have one class of application that is fairly idempotent, we don't have hard drives on these machines, so we don't use swap and expect it to die if it exceeds the amount of memory requested. This article reenforced this. I have another class of application that queues up things to process. The specs work on the current hardware 99% of the time, but on holiday weekends when large projects are delivering things slow down because work is submitted and people leave so nobody cares. Reloading the state from disk can take 20m+ and without swap if you were to exceed memory you're dead in the water until you purchase more hardware. Our contingency hardware has identical specs as the original.