4 ms·

I don't know, the only thing I ever noticed whne using swap was : > Disabling swap doesn't prevent pathological behaviour at near-OOM, although it's true that

by shakow 4y ago

I don't know, the only thing I ever noticed whne using swap was : > Disabling swap doesn't prevent pathological behaviour at near-OOM, although it's true that having swap may prolong it. Basically, the memory manager will just bulldoze through moving all the other applications to the swap to let the leaking program pursue its leak. > you are left with a system in an unpredictable state. Having no swap doesn't avoid this. Yes, it does: the program that leaked my 16GB is killed at the very beginning of the situation, and I don't have to either wait dozens of minutes for it to fill the swap on top of that to finally be killed, or just hard reboot the machine and lose the whole state.

tsimionescu 4y ago
> Yes, it does: the program that leaked my 16GB is killed at the very beginning of the situation How, and by whom? In a situation where you are running out of memory, it is programs that allocate the most often that have the highest chance of seeing an OOM, not those that allocate the most total amount. Also, on Linux, just because you don't have a swap partition doesn't mean things won't get swapped out to disk. Any memory mapped files will be swapped back to disk before issuing OOM problems - and the most common case of memory mapped files is executables and libraries. Which is of course the very thing that leads to a kind of livelock of swapless Linux systems that live close to 100% memory utilization: every time a thread is woken up, the instruction pipeline stalls until the next CPU instruction can be read from disk, slowing everything to a crawl.
- shakow 4y ago
  > How, and by whom? By the kernel OOM killer? Is that supposed to be a trick question? > In a situation where you are running out of memory, it is programs that allocate the most often that have the highest chance of seeing an OOM, not those that allocate the most total amount. I don't really care; it still beats the machine being unresponsive for hours, especially if it's a remote one I dont have access to. > Also, on Linux, just because you don't have a swap partition doesn't mean things won't get swapped out to disk. Any memory mapped files will be swapped back to disk before issuing OOM problems That's not what is commonly understood by swapping though; that's a cache purge. > every time a thread is woken up, the instruction pipeline stalls until the next CPU instruction can be read from disk, slowing everything to a crawl. When you have swap enabled, yes; that the whole problem. However, IME, OoM just leads to the problematic program being quickly killed, then the user being able to go back to work.
  - LarryMullins 4y ago
    > However, IME, OoM just leads to the problematic program being quickly killed, I think this is the divide. In my experience, OOM killer rarely kills the culprit. It doesn't actually know which programs are behaving irregularly (that's the 'trick' in his 'trick question') and the OOM score heuristic isn't reliable in desktop usecases. If you know ahead of time which process will likely cause you trouble, you can adjust oom_score_adj for that process to make it the first target, but this is not great for common desktop use.
  - tsimionescu 4y ago
    The OOMKiller is a pretty blunt and simple tool, that starts killing things until memory pressure is gone. Also, depending on how large the memory mapped areas are and specific OOM settings, your executables might start getting swapped out before the OOMKiller is actually called. > That's not what is commonly understood by swapping though; that's a cache purge. No, it's exactly a form of swapping. I'm not talking about purging Linux's IO caches. I am talking about taking a memory resident page of your process and unmapping it, such that the next it is accessed it causes a page fault and the VMM loads it back from disk. When you have swap space, this can happen to any page. When you don't, this can still happen to read-only pages that already have a corresponding file on disk - which is usually your executable and any shared libraries you loaded. Now, whether you'll really hit this situation or not really depends on how much of your memory is being taken up by the code of running processes. In many common server workloads, this should be very rare. However, in desktop workloads, where you might have lots of programs running at the same time, the total amount of memory being taken up by all their code may end up high enough to enter this state before calling the OOMKiller.
toast0 4y ago
The trick is not disabling swap, but having a reasonable size swap. The old rule of thumb of 2x RAM was reasonable in the 90s, but after that, it should be 2x RAM or 512MB, whichever is smaller. Nobody needs a 2TB swap volume on their 1TB ram server. If you've got something leaking a lot and fast, 512MB swap is going to fill up in seconds, not minutes. If you have a slower leak, swap usage provides a clear signal something is wrong and may give you a chance to intervene before the OOM killer. Even in fast death, swap usage may be useful forensics, if you record it at a high enough time resolution.
- seppel 4y ago
  > The old rule of thumb of 2x RAM was reasonable in the 90s, but after that, it should be 2x RAM or 512MB, whichever is smaller. This is exactly why I don't have swap for over a decade on my machines (since my machines have 16+ GB RAM). The mere 1/32 improvement is not worth the headaches of swap.

by shakow 4y ago

tsimionescu 4y ago

> Yes, it does: the program that leaked my 16GB is killed at the very beginning of the situation How, and by whom? In a situation where you are running out of memory, it is programs that allocate the most often that have the highest chance of seeing an OOM, not those that allocate the most total amount. Also, on Linux, just because you don't have a swap partition doesn't mean things won't get swapped out to disk. Any memory mapped files will be swapped back to disk before issuing OOM problems - and the most common case of memory mapped files is executables and libraries. Which is of course the very thing that leads to a kind of livelock of swapless Linux systems that live close to 100% memory utilization: every time a thread is woken up, the instruction pipeline stalls until the next CPU instruction can be read from disk, slowing everything to a crawl.

shakow 4y ago
> How, and by whom? By the kernel OOM killer? Is that supposed to be a trick question? > In a situation where you are running out of memory, it is programs that allocate the most often that have the highest chance of seeing an OOM, not those that allocate the most total amount. I don't really care; it still beats the machine being unresponsive for hours, especially if it's a remote one I dont have access to. > Also, on Linux, just because you don't have a swap partition doesn't mean things won't get swapped out to disk. Any memory mapped files will be swapped back to disk before issuing OOM problems That's not what is commonly understood by swapping though; that's a cache purge. > every time a thread is woken up, the instruction pipeline stalls until the next CPU instruction can be read from disk, slowing everything to a crawl. When you have swap enabled, yes; that the whole problem. However, IME, OoM just leads to the problematic program being quickly killed, then the user being able to go back to work.
- LarryMullins 4y ago
  > However, IME, OoM just leads to the problematic program being quickly killed, I think this is the divide. In my experience, OOM killer rarely kills the culprit. It doesn't actually know which programs are behaving irregularly (that's the 'trick' in his 'trick question') and the OOM score heuristic isn't reliable in desktop usecases. If you know ahead of time which process will likely cause you trouble, you can adjust oom_score_adj for that process to make it the first target, but this is not great for common desktop use.
- tsimionescu 4y ago
  The OOMKiller is a pretty blunt and simple tool, that starts killing things until memory pressure is gone. Also, depending on how large the memory mapped areas are and specific OOM settings, your executables might start getting swapped out before the OOMKiller is actually called. > That's not what is commonly understood by swapping though; that's a cache purge. No, it's exactly a form of swapping. I'm not talking about purging Linux's IO caches. I am talking about taking a memory resident page of your process and unmapping it, such that the next it is accessed it causes a page fault and the VMM loads it back from disk. When you have swap space, this can happen to any page. When you don't, this can still happen to read-only pages that already have a corresponding file on disk - which is usually your executable and any shared libraries you loaded. Now, whether you'll really hit this situation or not really depends on how much of your memory is being taken up by the code of running processes. In many common server workloads, this should be very rare. However, in desktop workloads, where you might have lots of programs running at the same time, the total amount of memory being taken up by all their code may end up high enough to enter this state before calling the OOMKiller.

toast0 4y ago

The trick is not disabling swap, but having a reasonable size swap. The old rule of thumb of 2x RAM was reasonable in the 90s, but after that, it should be 2x RAM or 512MB, whichever is smaller. Nobody needs a 2TB swap volume on their 1TB ram server. If you've got something leaking a lot and fast, 512MB swap is going to fill up in seconds, not minutes. If you have a slower leak, swap usage provides a clear signal something is wrong and may give you a chance to intervene before the OOM killer. Even in fast death, swap usage may be useful forensics, if you record it at a high enough time resolution.

seppel 4y ago
> The old rule of thumb of 2x RAM was reasonable in the 90s, but after that, it should be 2x RAM or 512MB, whichever is smaller. This is exactly why I don't have swap for over a decade on my machines (since my machines have 16+ GB RAM). The mere 1/32 improvement is not worth the headaches of swap.