4 ms·

Recognition that older linux swap strategies were unhelpful sometimes, which this piece of writing does, validates out past sense it wasn't working well. Regain

by ggm 1y ago

Recognition that older linux swap strategies were unhelpful sometimes, which this piece of writing does, validates out past sense it wasn't working well. Regaining trust takes time. Sometimes I think if backing store and swap were more clearly delineated we might have got to decent algorithms sooner. Having a huge amount of swap pre-emptively claimed was making it look like starvation, when it was just a runtime planning strategy. It's also confusing how top and vmstat report things. Also, as a BSD mainly person, I think the differences stand out. I haven't noticed an OOM killer approach on BSD. Ancient model: twice as much swap as memory Old model: same amount of swap as memory New model: amount of swap your experience tells you this job mix demands to manage memory pressure fairly, which is a bit of a tall ask sometimes, but basically pick a number up to memory size.

deleted 1y ago
[deleted]
kijin 1y ago
For modern Linux servers with large amounts of RAM, my rule of thumb is between 1/8 and 1/32 of RAM, depending on what the machine is for. For example, one of my database servers has 128GB of RAM and 8GB of swap. It tends to stabilize around 108GB of RAM and 5GB of swap usage under normal load, so I know that a 4GB swap would have been less than optimal. A larger swap would have been a waste as well.
- man8alexd 1y ago
  The proper rule of thumb is to make the swap large enough to keep all inactive anonymous pages after the workload has stabilized, but not too large to cause swap thrashing and a delayed OOM kill if a fast memory leak happens. Another rule of thumb is that performance degradation due to the active working set spilling into the swap is exponential - 0.1% excess causes 2x degradation, 1% - 10x degradation, 10% - 100x degradation (assuming 10^3 difference in latency between RAM and SSD).
  - kijin 1y ago
    I would approach the issue from the other direction. Start by buying enough RAM to contain the active working set for the foreseeable future. Afterward, you can start experimenting with different swap sizes (swapfiles are easier to resize, and they perform exactly as well as swap partitions!) to see how many inactive anonymous pages you can safely swap out. If you can swap out several gigabytes, that's a bonus! But don't take that for granted. Always be prepared to move everything back into RAM when needed.
- ChocolateGod 1y ago
  I no longer use disk swap for servers, instead opting for Zram with a maximum is 50% of RAM capacity and a high swapiness value. It'd be cool if Zram could apply to the RAM itself (like macOS) rather than needing a fake swap device.
  - cmurf 1y ago
    zswap https://docs.kernel.org/admin-guide/mm/zswap.html https://docs.kernel.org/admin-guide/mm/zswap.html The cgroup accounting also now works in zswap.
    - ChocolateGod 1y ago
      Zswap requires a backing disk swap, Zram does not.
      - cmurf 1y ago
        The backing disk or file will only be written to if cache eviction on the basis of LRU comes into play, which is fine because that's probably worth the write hit. The likelihood of thrashing, the biggest complaint about disk based swap, is far reduced. zram based swap isn't free. Its efficiency depends on the compression ratio (and cost).
  - LargoLasskhyfv 1y ago
    Lookie lookie! Isn't it spooky? https://github.com/CachyOS/CachyOS-Settings/blob/master/usr/lib/udev/rules.d/30-zram.rules https://github.com/CachyOS/CachyOS-Settings/blob/master/usr/... Resulting in https://i.postimg.cc/hP37vvpJ/screenieshottie.png https://i.postimg.cc/hP37vvpJ/screenieshottie.png Good enough...
    - ChocolateGod 1y ago
      Yeh. I haven't yet figured out how to get zram to apply transparently to containers though, anything in another memory cgroup will never get compressed unless swap is explicitly exposed to it.
creshal 1y ago
> Also, as a BSD mainly person, I think the differences stand out. I haven't noticed an OOM killer approach on BSD. BSD allocators simply return errors if no more memory is available; for backwards compatibility reasons Linux is stuck with a fatally flawed API that doesn't.
- man8alexd 1y ago
  I assumed the same, but just discovered that FreeBSD has vm.overcommit too. But I'm not sure about its working.
  - toast0 1y ago
    Overcommit is subtle. If you allocate a bunch of address space and don't touch it, that's one thing. If you allocate and touch everything, and then try to allocate more, it's better to get an allocation error than an unsatifyable page fault later. My understanding (which could very well be wrong) is Linux overcommit will continue to allocate address space when asked regardless of memory pressure; but FreeBSD overcommit will refuse allocations when there's too much memory pressure. I'm pretty sure I've seen FreeBSD's OOM killer, but it needs a specific pattern of memory use, it's much more likely for an application to get a failed allocation and exit, freeing memory; than for all the applications to have unused allocations that they then use. All that said, I prefer to run with a small swap, somewhere around 0.5-2GB. Memory pressure is hard to measure (although recent linux has a measure that I haven't used), but swap % and swap i/o are easy to measure. If your swap grows quickly, you might not have time to do any operations to fix it, but your stats should tell the tale. If your swap grows slowly enough, you can set thresholds and analyze the situation. If you have a lot of swap i/o that provides a measure of urgency.
    - jcalvinowens 1y ago
      > If you allocate and touch everything, and then try to allocate more, it's better to get an allocation error than an unsatifyable page fault later. It depends, but generally speaking I'd disagree with that. The only time you actually want to see the allocation failures is if you're writing high reliability software where you've gone to the trouble to guarantee some sort of meaningful forward progress when memory is exhausted. That is VERY VERY hard, and quickly becomes impossible when you have non-trivial library dependencies. If all you do is raise std::bad_alloc or call abort(), handling NULL return from malloc() is arguably a waste of icache: just let it crash. Dereferencing NULL is guaranteed to crash on Linux, only root can mmap() the lowest page. Admittedly I'm anal, and I write the explicit code to check for it and call abort(), but I know very experienced programmers I respect who don't.

by ggm 1y ago

deleted 1y ago

[deleted]

kijin 1y ago

For modern Linux servers with large amounts of RAM, my rule of thumb is between 1/8 and 1/32 of RAM, depending on what the machine is for. For example, one of my database servers has 128GB of RAM and 8GB of swap. It tends to stabilize around 108GB of RAM and 5GB of swap usage under normal load, so I know that a 4GB swap would have been less than optimal. A larger swap would have been a waste as well.

man8alexd 1y ago
The proper rule of thumb is to make the swap large enough to keep all inactive anonymous pages after the workload has stabilized, but not too large to cause swap thrashing and a delayed OOM kill if a fast memory leak happens. Another rule of thumb is that performance degradation due to the active working set spilling into the swap is exponential - 0.1% excess causes 2x degradation, 1% - 10x degradation, 10% - 100x degradation (assuming 10^3 difference in latency between RAM and SSD).
- kijin 1y ago
  I would approach the issue from the other direction. Start by buying enough RAM to contain the active working set for the foreseeable future. Afterward, you can start experimenting with different swap sizes (swapfiles are easier to resize, and they perform exactly as well as swap partitions!) to see how many inactive anonymous pages you can safely swap out. If you can swap out several gigabytes, that's a bonus! But don't take that for granted. Always be prepared to move everything back into RAM when needed.
ChocolateGod 1y ago
I no longer use disk swap for servers, instead opting for Zram with a maximum is 50% of RAM capacity and a high swapiness value. It'd be cool if Zram could apply to the RAM itself (like macOS) rather than needing a fake swap device.
- cmurf 1y ago
  zswap https://docs.kernel.org/admin-guide/mm/zswap.html https://docs.kernel.org/admin-guide/mm/zswap.html The cgroup accounting also now works in zswap.
  - ChocolateGod 1y ago
    Zswap requires a backing disk swap, Zram does not.
    - cmurf 1y ago
      The backing disk or file will only be written to if cache eviction on the basis of LRU comes into play, which is fine because that's probably worth the write hit. The likelihood of thrashing, the biggest complaint about disk based swap, is far reduced. zram based swap isn't free. Its efficiency depends on the compression ratio (and cost).
- LargoLasskhyfv 1y ago
  Lookie lookie! Isn't it spooky? https://github.com/CachyOS/CachyOS-Settings/blob/master/usr/lib/udev/rules.d/30-zram.rules https://github.com/CachyOS/CachyOS-Settings/blob/master/usr/... Resulting in https://i.postimg.cc/hP37vvpJ/screenieshottie.png https://i.postimg.cc/hP37vvpJ/screenieshottie.png Good enough...
  - ChocolateGod 1y ago
    Yeh. I haven't yet figured out how to get zram to apply transparently to containers though, anything in another memory cgroup will never get compressed unless swap is explicitly exposed to it.

creshal 1y ago

> Also, as a BSD mainly person, I think the differences stand out. I haven't noticed an OOM killer approach on BSD. BSD allocators simply return errors if no more memory is available; for backwards compatibility reasons Linux is stuck with a fatally flawed API that doesn't.

man8alexd 1y ago
I assumed the same, but just discovered that FreeBSD has vm.overcommit too. But I'm not sure about its working.
- toast0 1y ago
  Overcommit is subtle. If you allocate a bunch of address space and don't touch it, that's one thing. If you allocate and touch everything, and then try to allocate more, it's better to get an allocation error than an unsatifyable page fault later. My understanding (which could very well be wrong) is Linux overcommit will continue to allocate address space when asked regardless of memory pressure; but FreeBSD overcommit will refuse allocations when there's too much memory pressure. I'm pretty sure I've seen FreeBSD's OOM killer, but it needs a specific pattern of memory use, it's much more likely for an application to get a failed allocation and exit, freeing memory; than for all the applications to have unused allocations that they then use. All that said, I prefer to run with a small swap, somewhere around 0.5-2GB. Memory pressure is hard to measure (although recent linux has a measure that I haven't used), but swap % and swap i/o are easy to measure. If your swap grows quickly, you might not have time to do any operations to fix it, but your stats should tell the tale. If your swap grows slowly enough, you can set thresholds and analyze the situation. If you have a lot of swap i/o that provides a measure of urgency.
  - jcalvinowens 1y ago
    > If you allocate and touch everything, and then try to allocate more, it's better to get an allocation error than an unsatifyable page fault later. It depends, but generally speaking I'd disagree with that. The only time you actually want to see the allocation failures is if you're writing high reliability software where you've gone to the trouble to guarantee some sort of meaningful forward progress when memory is exhausted. That is VERY VERY hard, and quickly becomes impossible when you have non-trivial library dependencies. If all you do is raise std::bad_alloc or call abort(), handling NULL return from malloc() is arguably a waste of icache: just let it crash. Dereferencing NULL is guaranteed to crash on Linux, only root can mmap() the lowest page. Admittedly I'm anal, and I write the explicit code to check for it and call abort(), but I know very experienced programmers I respect who don't.