4 ms·

> If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if

> If I have enough RAM why do I want things to swap to begin with? The author never explained why putting rarely used pages in swap will improve performance if I have adequate RAM. The memory freed from a page that got pushed to swap might be better used for serving disk cache, for example. If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when they do become busy, you won't have to wait. At least, that's the theory as I understand it. Whether or not it works in practice, IDK. > And using swap as some kind of "RAM emergency cover" does not make any sense to me. Personally, I have never had a case where I could gracefully recover a host that had started swapping. I have successfully recovered hosts that began swapping, and even hosts that filled their swap. But it's beyond painful, and in most of the cases where I have a host doing such swapping, an OOM kill would have been a welcome reprieve. > Second, I find it's much harder to detect degraded performance than it is to detect a dead process. Agreed, and restarting something that's been OOM'd is simple enough too. Our hosts tend to stop reporting metrics when they start swapping, simply b/c so little is actually getting done, so they get labelled as "down". Linux also exposes a metric called "major page faults" (IIRC, it's "pgmajfault" in /proc/vmstat) that records the number of major (required service from disk) page faults; if the rate of that value is too steep for too long, that ≅ swap thrashing.

mnw21cam 7y ago
The key point is that if you have no swap, and you run low on memory, then the OS has no choice but to throw away stuff that is clean. This includes stuff that is being actively used right now, like the code that is in your processes. So your system will still thrash, even though it has no swap, because it is trying to load pages of running code in on-demand, while constantly throwing them away again because it doesn't have enough space. This will incapacitate a server even more thoroughly than if there is swap, and the working set is larger than physical memory. The more RAM a server has, the more severe this incapacitation will be. For this reason, you want an OOM killer to kill stuff way earlier than when you actually run out of RAM, because if you ever reach that point it is too late. I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects.
- scottlamb 7y ago
  > I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects. Along those lines, take a look at this: [1] > Pressure stall information were added to Linux 4.20 as a way to quantify resource pressure in the system in a better way than the traditional load average. PSI aggregates and reports the overall wallclock time in which the tasks in a system (or cgroup) wait for cpu, io or memory. > This release [Linux 5.2] lets users to configure sensitive thresholds and use poll() and friends to be notified when a certain pressure thresold is breached within a user-defined time window. With this mechanism, Android can monitor for, and ward off, mounting memory shortages before they cause problems for the user. For example, using memory stall, monitors in userspace like the low memory killer daemon (lmkd) can detect mounting pressure and kill less important processes before device becomes visibly sluggish. In memory stress testing psi memory monitors produce roughly 10x less false positives compared to vmpressure. I haven't tried it yet, but it sounds promising. [1] https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_Information_for_better_resource_monitoring https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_I...
mason55 7y ago
> That page that got pushed to swap might be better used for serving disk cache, for example. > If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when the do become busy, you won't have to wait. > At least, that's the theory as I understand it. Whether or not it works in practice, IDK. These are the ideas that I hoped the original post was going to be about. Disk cache is a good one. I'm not knowledgeable enough about this, will the Linux kernel automatically start using RAM as disk cache if there's enough free space? Is it smart enough to prioritize disk cache over infrequently used pages and then know to discard the disk cache if it does need to page stuff back in from swap?
- FBT 7y ago
  The original post does talk about it, but it skips the basics, presumably on the assumption that they're already known. > will the Linux kernel automatically start using RAM as disk cache if there's enough free space? Yes, exactly. > Is it smart enough to prioritize disk cache over infrequently used pages and then know to discard the disk cache if it does need to page stuff back in from swap? It is, and this is what's controlled by the vm.vfs_cache_pressure sysctl that the post discusses how best to configure.

mnw21cam 7y ago

The key point is that if you have no swap, and you run low on memory, then the OS has no choice but to throw away stuff that is clean. This includes stuff that is being actively used right now, like the code that is in your processes. So your system will still thrash, even though it has no swap, because it is trying to load pages of running code in on-demand, while constantly throwing them away again because it doesn't have enough space. This will incapacitate a server even more thoroughly than if there is swap, and the working set is larger than physical memory. The more RAM a server has, the more severe this incapacitation will be. For this reason, you want an OOM killer to kill stuff way earlier than when you actually run out of RAM, because if you ever reach that point it is too late. I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects.

scottlamb 7y ago
> I use a program called earlyoom, which has saved my server from having to have the big red button pushed quite a few times, when fumble-fingered PhD students "accidentally" consume all the RAM in their pet projects. Along those lines, take a look at this: [1] > Pressure stall information were added to Linux 4.20 as a way to quantify resource pressure in the system in a better way than the traditional load average. PSI aggregates and reports the overall wallclock time in which the tasks in a system (or cgroup) wait for cpu, io or memory. > This release [Linux 5.2] lets users to configure sensitive thresholds and use poll() and friends to be notified when a certain pressure thresold is breached within a user-defined time window. With this mechanism, Android can monitor for, and ward off, mounting memory shortages before they cause problems for the user. For example, using memory stall, monitors in userspace like the low memory killer daemon (lmkd) can detect mounting pressure and kill less important processes before device becomes visibly sluggish. In memory stress testing psi memory monitors produce roughly 10x less false positives compared to vmpressure. I haven't tried it yet, but it sounds promising. [1] https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_Information_for_better_resource_monitoring https://kernelnewbies.org/Linux_5.2#Improved_Presure_Stall_I...

mason55 7y ago

> That page that got pushed to swap might be better used for serving disk cache, for example. > If you don't have enough RAM (even for spikes), a preemptive push when things aren't busy might mean that when the do become busy, you won't have to wait. > At least, that's the theory as I understand it. Whether or not it works in practice, IDK. These are the ideas that I hoped the original post was going to be about. Disk cache is a good one. I'm not knowledgeable enough about this, will the Linux kernel automatically start using RAM as disk cache if there's enough free space? Is it smart enough to prioritize disk cache over infrequently used pages and then know to discard the disk cache if it does need to page stuff back in from swap?

FBT 7y ago
The original post does talk about it, but it skips the basics, presumably on the assumption that they're already known. > will the Linux kernel automatically start using RAM as disk cache if there's enough free space? Yes, exactly. > Is it smart enough to prioritize disk cache over infrequently used pages and then know to discard the disk cache if it does need to page stuff back in from swap? It is, and this is what's controlled by the vm.vfs_cache_pressure sysctl that the post discusses how best to configure.