4 ms·

> I’ve actually had Linux become completely unresponsive when running many large node instances. This is almost certainly not a scheduling problem but a swappi

> I’ve actually had Linux become completely unresponsive when running many large node instances. This is almost certainly not a scheduling problem but a swapping problem. Linux is famously bad at handling these kinds of situations. About five years ago I decided to never use swap anymore, and I couldn't be happier with that decision.

iforgotpassword 4y ago
> About five years ago I decided to never use swap anymore, and I couldn't be happier with that decision. How does that make sense? Either your workload fits into RAM, so no swapping occurs and everything is responsive. Or your workload is too large to fit into RAM so a) you have a swap partition and you system starts swapping and everything becomes really unresponsive, but the task will finish. b) you don't have swap so the kernel kills the process using most memory, which is most likely the thing you're just trying to run. How can you be happy with option b? I mean obviously get more RAM either way, but I'd rather have that safety net for the occasional memory intensive task. Obviously not a solution if you're short on RAM by a huge margin but if it's just a GB or two it's acceptable, because it's likely that the kernel can just swap out stuff from background programs that don't actively access their stuff right now.
- Asooka 4y ago
  Because the failure mode is gentler. With swap, once you start a process that needs more RAM than you have, you enter a swapping death spiral, the system becomes completely unresponsive and you have to hard reset. Without swap, you only get one process killed.
  - iforgotpassword 4y ago
    No, as I described that is not the case. Not every process running needs all the memory it allocated all the time. A GB or two can easily be swapped out from your browser, Networkmanager, file manager, email client, etcpp. swapping only becomes a problem if pressure is so high that you constantly swap in and out the same pages. You can see this well from the kennel's proactive swapping. If I run a bog standard desktop Linux distro and leave swappiness at the default 60, after a few hours of normal usage where I don't even get close to filling up my RAM with actual memory allocations from processes, the kernel will have swapped out almost a GB of data from several background services. Because a lot of programs allocate some memory for something and then never look at it in a long time. So it makes more sense to swap that to disk and instead use that ram as page cache. Also, with current NVMe speeds, swapping has become a lot more bearable. Really the only place where I can see why you wouldn't use swap is servers with a well defined, dedicated workload. Here I indeed want a fast and noticable fail mode in case a new version of whatever I'm running might have a memory leak, or just vastly different behavior regarding memory usage.
    - atq2119 4y ago
      Your theory is correct, but as usual, the difference between theory and practice is larger in practice than in theory. When a workload runs out of RAM, it often really runs out of RAM, and would just be swapping continuously. It may even run out of swap space, if it was left to run for long enough. In my experience, even with NVMe drives, it just wasn't worth it because it usually wouldn't be able to finish in a reasonable amount of time. In part, I'm sure that's because the Linux kernel behaved much worse than is theoretically possible. This may obviously depend on your workload, but I've heard similar sentiments from others (in fact, it was somebody else who suggested disabling swap to me based on their experience). The other point is that a sibling comment mentions some recent kernel changes that may have led to improvements, but I haven't given those a chance.
      - iforgotpassword 4y ago
        I guess it's really that then, different workloads. Swap has served me well over time, but mostly because I don't encounter those "some tool goes berserk"-cases really, but just having opened to many applications at once that I don't necessarily all use at the same time, because I leave my workstation running 24/7.
arglebargle123 4y ago
The multi-gen LRU swap patchset helped a lot with this issue but didn't entirely resolve it. I think it was merged a year ago or so and it's worth checking out if you haven't yet—it makes a pretty significant usability impact during swap pressure.
Osiris 4y ago
That may be true. The system did eventually recover after about 15-20 minutes.

iforgotpassword 4y ago

> About five years ago I decided to never use swap anymore, and I couldn't be happier with that decision. How does that make sense? Either your workload fits into RAM, so no swapping occurs and everything is responsive. Or your workload is too large to fit into RAM so a) you have a swap partition and you system starts swapping and everything becomes really unresponsive, but the task will finish. b) you don't have swap so the kernel kills the process using most memory, which is most likely the thing you're just trying to run. How can you be happy with option b? I mean obviously get more RAM either way, but I'd rather have that safety net for the occasional memory intensive task. Obviously not a solution if you're short on RAM by a huge margin but if it's just a GB or two it's acceptable, because it's likely that the kernel can just swap out stuff from background programs that don't actively access their stuff right now.

Asooka 4y ago
Because the failure mode is gentler. With swap, once you start a process that needs more RAM than you have, you enter a swapping death spiral, the system becomes completely unresponsive and you have to hard reset. Without swap, you only get one process killed.
- iforgotpassword 4y ago
  No, as I described that is not the case. Not every process running needs all the memory it allocated all the time. A GB or two can easily be swapped out from your browser, Networkmanager, file manager, email client, etcpp. swapping only becomes a problem if pressure is so high that you constantly swap in and out the same pages. You can see this well from the kennel's proactive swapping. If I run a bog standard desktop Linux distro and leave swappiness at the default 60, after a few hours of normal usage where I don't even get close to filling up my RAM with actual memory allocations from processes, the kernel will have swapped out almost a GB of data from several background services. Because a lot of programs allocate some memory for something and then never look at it in a long time. So it makes more sense to swap that to disk and instead use that ram as page cache. Also, with current NVMe speeds, swapping has become a lot more bearable. Really the only place where I can see why you wouldn't use swap is servers with a well defined, dedicated workload. Here I indeed want a fast and noticable fail mode in case a new version of whatever I'm running might have a memory leak, or just vastly different behavior regarding memory usage.
  - atq2119 4y ago
    Your theory is correct, but as usual, the difference between theory and practice is larger in practice than in theory. When a workload runs out of RAM, it often really runs out of RAM, and would just be swapping continuously. It may even run out of swap space, if it was left to run for long enough. In my experience, even with NVMe drives, it just wasn't worth it because it usually wouldn't be able to finish in a reasonable amount of time. In part, I'm sure that's because the Linux kernel behaved much worse than is theoretically possible. This may obviously depend on your workload, but I've heard similar sentiments from others (in fact, it was somebody else who suggested disabling swap to me based on their experience). The other point is that a sibling comment mentions some recent kernel changes that may have led to improvements, but I haven't given those a chance.
    - iforgotpassword 4y ago
      I guess it's really that then, different workloads. Swap has served me well over time, but mostly because I don't encounter those "some tool goes berserk"-cases really, but just having opened to many applications at once that I don't necessarily all use at the same time, because I leave my workstation running 24/7.

arglebargle123 4y ago

The multi-gen LRU swap patchset helped a lot with this issue but didn't entirely resolve it. I think it was merged a year ago or so and it's worth checking out if you haven't yet—it makes a pretty significant usability impact during swap pressure.

Osiris 4y ago

That may be true. The system did eventually recover after about 15-20 minutes.