4 ms·

Instead of using swap (on disk) at all, try deleting the swap partition and just use zram [1]. It essentially puts swap in-memory but compressed. This is likely

Instead of using swap (on disk) at all, try deleting the swap partition and just use zram [1]. It essentially puts swap in-memory but compressed. This is likely faster than disk anyway, and avoids worrying about endurance at all. I don’t bother with swap partitions anymore. [1]: https://www.kernel.org/doc/html/latest/admin-guide/blockdev/zram.html https://www.kernel.org/doc/html/latest/admin-guide/blockdev/...

etbe 5y ago
zram is a good thing. However in my investigations in 2014 "my conclusion was that swap wouldn’t be a problem as no normally operating systems that I run had swap using any significant fraction of total disk writes". During the last 8 years the amount of RAM in all my systems has increased significantly so swap is even less of an issue.
- tpetry 5y ago
  Sure but setting up swap is still recommended. I guess some internal linux memory algorithms prefer to have some safety net? Setting up a 1GB zram swap was effective for me, it‘s not much wasted memory as servers have so much memory these days and because of compression it can fit more than 1GB.
  - wahern 5y ago
    What I've seen is that under memory pressure kernel tasks trying to evict or free pages to satisfy an allocation request can race with other tasks dirtying and filling pages even faster, especially via the buffer cache. This can induce patterns of lock contention on low-level VM data structures and flushing procedures that effectively behave like a deadlock. Eventually various loop limits and lock timeouts will help unstick things, but in the worst cases the system gets caught in higher order loops and I've seen systems lock up for minutes. The systems I've seen this on never had any swap. Some of the heuristics designed to minimize pathological contention latency seem to implicitly assume that the swapping subsystem--both in its ability to help free space, and the latency it introduces when loading and evicting pages--will help mitigate the chance tasks will get caught in a tight contention loop. IOW, the I/O latency of swap effectively induces back pressure on load, helping operations freeing pages to progress faster than operations consuming pages. (Predictably, the faster your swap, the less well this works. When people began putting swap on SSDs, heuristics had to be retuned.) Arguably the root of the problem is the legacy of overcommit. Even though it can be nominally disabled, many aspects of the kernel were designed with the notion that the only direction to move under memory pressure is forward, relying on the promise of the OOM killer eventually freeing up enough memory to maintain forward progress on the current operation, rather than unwinding and returning a failure condition. The dynamic seems similar to buffer bloat.
    - _0w8t 5y ago
      Overcommit on POSIX systems is a consequence of fork/exec architecture of starting new processes. Fork must double the memory accounting if one disables overcommit and for a memory hog that starts a helper process that may lead to OOM when when in fact there are enough memory. Windows, that does not have fork, does not suffer from this.
      - deleted 5y ago
        [deleted]
      - zozbot234 5y ago
        POSIX is not restricted to a fork/exec model, there's also posix_spawn(). Additionally it may be possible to somehow rely on COW to avoid double accounting after fork, provided that the new process execs "soon after". This would cover cases where posix_spawn() cannot be directly used, because some fixup needs to occur before the new process exec's.
        _0w8t 5y ago
        Without double-accounting after the fork it is in general impossible to guarantee that no running program needs to be killed on OOM as the kernel does not know what the process is going to do after the fork. And fork still has legitimate users that does not involve exec. With multi-process architecture that many programs like browsers use for security it is a common optimization to have a seed process with sandbox initialized that is forked as necessary into a worker process.
- _0w8t 5y ago
  A fast swap like zram or modern SSD is very beneficial on a development machine as one allows to keep active file cache for much longer so grep and git on huge trees works instantly even if one have a lot of memory hogs like VMs or language servers around.
_0w8t 5y ago
Modern SSD are much faster at read than lz4, the fastest compression algorithm, at decompression and just 2-3 times slower in practice when writing than lz4 at compression. And they do not tax CPU while doing that, which is important since heavy memory usage may come from CPU-intensive tasks. I did some benchmarks in my laptop and for heavy compilation tasks that may occasionally spike the memory usage over the amount of RAM memory compression was not beneficial compared with using SSD for swap. The picture changes if one needs encrypted storage and cannot use hardware encryption on SSD. The software encryption will tax CPU on roughly same scale as lz4 and lz4 is still faster overall for read/write combination.
- etbe 5y ago
  When I was using a Thinkpad X1 Carbon Gen6 with Linux it would hit a thermal limit and throttle the CPU speed if I read from NVMe at maximum speed for more than about 30 seconds. That was partly due to an issue with the way Linux and the BIOS interacted which gave a lower thermal limit than expected. But it really depends on what NVMe devices you use, I've seen cheap ones bottleneck at 30MB/s when doing continuous writes with no downtime. RAM is getting so cheap and easy to get that in recent years I only really use swap on laptops. Using a CPU core for compression isn't a big deal in a laptop because you usually have 4 cores and only 1 of them in use most of the time.
- amarshall 5y ago
  Huh? What numbers are you looking at? On my laptop (i7-1165G7, DDR4 3200 MHz) lz4 decompression is ~3000 MB/s, compression is ~485 MB/s (on my 3960X desktop it’s ~1.5x that) (numbers from lzbench). Zram by default uses one stream per CPU, so theoretical max is (I think) 8x those for the laptop and, well, a lot more for the Threadripper. Non-Optane SSDs only really have a potential advantage in sequential writing. Max throughput aside, Zram will generally have better latency (and thus likely IOPS as well), and better consistency since most SSDs get their max performance from DRAM caches. In the end, though, quibbling over synthetic numbers doesn’t matter much if you have benchmarked your actual use case. I don’t know what “not beneficial” means (the same? 10x worse?), but part of my original point was that
  - 5y ago
RyEgswuCsn 5y ago
Another potential offender is the hibernation process. It's probably compressed but still writing several GB to the SSD every time the machine goes into hibernation certainly doesn't help.
- londons_explore 5y ago
  I am disappointed that in Linux the hibernation process is so dumb. For example, if you hibernate a Linux machine, then boot it up and immediately re-hibernate, there should be barely any memory to write out (since very little will have changed since the last hibernation). Yet the written data is the whole image with no regards for the fact most of the data is already on disk and doesn't need to be rewritten.
zozbot234 5y ago
The code and feature duplication between "zram" and "zswap/zcache" seems quite unfortunate. Is there no prospect of unifying these kernel-side tools in the future, so that a single configuration choice can be used to set up memory compression regardless of it being "backed" by a physical swap device?

etbe 5y ago

zram is a good thing. However in my investigations in 2014 "my conclusion was that swap wouldn’t be a problem as no normally operating systems that I run had swap using any significant fraction of total disk writes". During the last 8 years the amount of RAM in all my systems has increased significantly so swap is even less of an issue.

tpetry 5y ago
Sure but setting up swap is still recommended. I guess some internal linux memory algorithms prefer to have some safety net? Setting up a 1GB zram swap was effective for me, it‘s not much wasted memory as servers have so much memory these days and because of compression it can fit more than 1GB.
- wahern 5y ago
  What I've seen is that under memory pressure kernel tasks trying to evict or free pages to satisfy an allocation request can race with other tasks dirtying and filling pages even faster, especially via the buffer cache. This can induce patterns of lock contention on low-level VM data structures and flushing procedures that effectively behave like a deadlock. Eventually various loop limits and lock timeouts will help unstick things, but in the worst cases the system gets caught in higher order loops and I've seen systems lock up for minutes. The systems I've seen this on never had any swap. Some of the heuristics designed to minimize pathological contention latency seem to implicitly assume that the swapping subsystem--both in its ability to help free space, and the latency it introduces when loading and evicting pages--will help mitigate the chance tasks will get caught in a tight contention loop. IOW, the I/O latency of swap effectively induces back pressure on load, helping operations freeing pages to progress faster than operations consuming pages. (Predictably, the faster your swap, the less well this works. When people began putting swap on SSDs, heuristics had to be retuned.) Arguably the root of the problem is the legacy of overcommit. Even though it can be nominally disabled, many aspects of the kernel were designed with the notion that the only direction to move under memory pressure is forward, relying on the promise of the OOM killer eventually freeing up enough memory to maintain forward progress on the current operation, rather than unwinding and returning a failure condition. The dynamic seems similar to buffer bloat.
  - _0w8t 5y ago
    Overcommit on POSIX systems is a consequence of fork/exec architecture of starting new processes. Fork must double the memory accounting if one disables overcommit and for a memory hog that starts a helper process that may lead to OOM when when in fact there are enough memory. Windows, that does not have fork, does not suffer from this.
    - deleted 5y ago
      [deleted]
    - zozbot234 5y ago
      POSIX is not restricted to a fork/exec model, there's also posix_spawn(). Additionally it may be possible to somehow rely on COW to avoid double accounting after fork, provided that the new process execs "soon after". This would cover cases where posix_spawn() cannot be directly used, because some fixup needs to occur before the new process exec's.
      - _0w8t 5y ago
        Without double-accounting after the fork it is in general impossible to guarantee that no running program needs to be killed on OOM as the kernel does not know what the process is going to do after the fork. And fork still has legitimate users that does not involve exec. With multi-process architecture that many programs like browsers use for security it is a common optimization to have a seed process with sandbox initialized that is forked as necessary into a worker process.
_0w8t 5y ago
A fast swap like zram or modern SSD is very beneficial on a development machine as one allows to keep active file cache for much longer so grep and git on huge trees works instantly even if one have a lot of memory hogs like VMs or language servers around.

_0w8t 5y ago

Modern SSD are much faster at read than lz4, the fastest compression algorithm, at decompression and just 2-3 times slower in practice when writing than lz4 at compression. And they do not tax CPU while doing that, which is important since heavy memory usage may come from CPU-intensive tasks. I did some benchmarks in my laptop and for heavy compilation tasks that may occasionally spike the memory usage over the amount of RAM memory compression was not beneficial compared with using SSD for swap. The picture changes if one needs encrypted storage and cannot use hardware encryption on SSD. The software encryption will tax CPU on roughly same scale as lz4 and lz4 is still faster overall for read/write combination.

etbe 5y ago
When I was using a Thinkpad X1 Carbon Gen6 with Linux it would hit a thermal limit and throttle the CPU speed if I read from NVMe at maximum speed for more than about 30 seconds. That was partly due to an issue with the way Linux and the BIOS interacted which gave a lower thermal limit than expected. But it really depends on what NVMe devices you use, I've seen cheap ones bottleneck at 30MB/s when doing continuous writes with no downtime. RAM is getting so cheap and easy to get that in recent years I only really use swap on laptops. Using a CPU core for compression isn't a big deal in a laptop because you usually have 4 cores and only 1 of them in use most of the time.
amarshall 5y ago
Huh? What numbers are you looking at? On my laptop (i7-1165G7, DDR4 3200 MHz) lz4 decompression is ~3000 MB/s, compression is ~485 MB/s (on my 3960X desktop it’s ~1.5x that) (numbers from lzbench). Zram by default uses one stream per CPU, so theoretical max is (I think) 8x those for the laptop and, well, a lot more for the Threadripper. Non-Optane SSDs only really have a potential advantage in sequential writing. Max throughput aside, Zram will generally have better latency (and thus likely IOPS as well), and better consistency since most SSDs get their max performance from DRAM caches. In the end, though, quibbling over synthetic numbers doesn’t matter much if you have benchmarked your actual use case. I don’t know what “not beneficial” means (the same? 10x worse?), but part of my original point was that
- 5y ago

RyEgswuCsn 5y ago

Another potential offender is the hibernation process. It's probably compressed but still writing several GB to the SSD every time the machine goes into hibernation certainly doesn't help.

londons_explore 5y ago
I am disappointed that in Linux the hibernation process is so dumb. For example, if you hibernate a Linux machine, then boot it up and immediately re-hibernate, there should be barely any memory to write out (since very little will have changed since the last hibernation). Yet the written data is the whole image with no regards for the fact most of the data is already on disk and doesn't need to be rewritten.

zozbot234 5y ago

The code and feature duplication between "zram" and "zswap/zcache" seems quite unfortunate. Is there no prospect of unifying these kernel-side tools in the future, so that a single configuration choice can be used to set up memory compression regardless of it being "backed" by a physical swap device?