4 ms·

Because Chrome's program text itself is reclaimable and counts as "available".

by blueflow 1y ago

man8alexd 1y ago
active pages are not reclaimable.
- blueflow 1y ago
  Pages will not stay in the active LRU if the accessor is stalled. Edit: You keep repeating that all over the thread - i started my week an hour ago and i had two of these memory stall events during the weekend. Again its the machines with big binaries running and no swap. Maybe you could provide a better explanation than "this doesn't happen"?
  - man8alexd 1y ago
    You keep repeating that the program text pages are being evicted as if that were the only thing that happens under memory pressure. It is the last thing that happens after the kernel drops all regular page cache, causing I/O starvation, and without swap, the kernel can't do anything about inactive anon pages. This means that at this moment, you are almost out of memory and the kernel can't do anything other than shuffle the remaining pages, thrashing the I/O. The kernel can't predict the future and doesn't know if it is a temporary situation due to some load spike or if it is a memory leak and resources won't be available again. The kernel doesn't know SLOs and operator priorities - whether the workload should survive at all costs, or the stalls are not acceptable and the workload should be OOM-killed. It is the job of the operator to give the kernel appropriate information by setting cgroup limits and monitoring memory pressure. If you want the workload to be OOM-killed before it consumes all the memory and starts causing I/O thrashing - set `memory.max` and `memory.swap.max` cgroup limits. If you want the workload memory not to be reclaimed by the kernel - set `memory.min`.
ciupicri 1y ago
Though I can deactivate the swap partition from the HDD and afterwards it's running acceptable. Perhaps it's using the faster zram, but I don't remember noticing zram usage increasing as shown my swapon -s. Or am I missing something?
- blueflow 1y ago
  Do you have the pressure stall information data in /proc/pressure? If you have it enabled, reboot, use chrome until your get your stall again, and then look which of the files in there has the highest total= number, thats likely it. I doubt that disabling swap reduces your stalls, but i'd like to see the numbers for that.
  - ciupicri 1y ago
    I haven't rebooted yet, but Chrome went downhill again. # cat /proc/pressure/memory some avg10=3.34 avg60=6.40 avg300=3.22 total=65245554 full avg10=3.18 avg60=6.14 avg300=3.10 total=64646112 # free -m total used free shared buff/cache available Mem: 15874 8099 309 370 8162 7775 Swap: 25599 10873 14726 Swap is composed from 4+5 GBs of zRAM (lzo&zstd) and 16 GBs on HDD. There's also some background I/O from time to time caused by torrent seeding. vm.swappiness = 13. I misread your comment, so I didn't check all the files, but here's how /proc/pressure/ looks now after managing to close Chrome: cpu some avg10=0.00 avg60=0.00 avg300=0.07 total=188606342 cpu full avg10=0.00 avg60=0.00 avg300=0.00 total=0 io some avg10=2.07 avg60=1.50 avg300=9.23 total=2149496347 io full avg10=2.07 avg60=1.47 avg300=8.79 total=2045347731 irq full avg10=0.11 avg60=0.20 avg300=0.23 total=288863055 memory some avg10=0.00 avg60=0.00 avg300=1.22 total=94771913 memory full avg10=0.00 avg60=0.00 avg300=1.18 total=93379177
    - blueflow 1y ago
      Okay this is a legit case where having swap on a slow HDD is the main reason for stalls. 10 GB swapped, $expletive.
    - 1y ago

Because Chrome's program text itself is reclaimable and counts as "available".

man8alexd 1y ago

active pages are not reclaimable.

blueflow 1y ago
Pages will not stay in the active LRU if the accessor is stalled. Edit: You keep repeating that all over the thread - i started my week an hour ago and i had two of these memory stall events during the weekend. Again its the machines with big binaries running and no swap. Maybe you could provide a better explanation than "this doesn't happen"?
- man8alexd 1y ago
  You keep repeating that the program text pages are being evicted as if that were the only thing that happens under memory pressure. It is the last thing that happens after the kernel drops all regular page cache, causing I/O starvation, and without swap, the kernel can't do anything about inactive anon pages. This means that at this moment, you are almost out of memory and the kernel can't do anything other than shuffle the remaining pages, thrashing the I/O. The kernel can't predict the future and doesn't know if it is a temporary situation due to some load spike or if it is a memory leak and resources won't be available again. The kernel doesn't know SLOs and operator priorities - whether the workload should survive at all costs, or the stalls are not acceptable and the workload should be OOM-killed. It is the job of the operator to give the kernel appropriate information by setting cgroup limits and monitoring memory pressure. If you want the workload to be OOM-killed before it consumes all the memory and starts causing I/O thrashing - set `memory.max` and `memory.swap.max` cgroup limits. If you want the workload memory not to be reclaimed by the kernel - set `memory.min`.

ciupicri 1y ago

Though I can deactivate the swap partition from the HDD and afterwards it's running acceptable. Perhaps it's using the faster zram, but I don't remember noticing zram usage increasing as shown my swapon -s. Or am I missing something?

blueflow 1y ago
Do you have the pressure stall information data in /proc/pressure? If you have it enabled, reboot, use chrome until your get your stall again, and then look which of the files in there has the highest total= number, thats likely it. I doubt that disabling swap reduces your stalls, but i'd like to see the numbers for that.
- ciupicri 1y ago
  I haven't rebooted yet, but Chrome went downhill again. # cat /proc/pressure/memory some avg10=3.34 avg60=6.40 avg300=3.22 total=65245554 full avg10=3.18 avg60=6.14 avg300=3.10 total=64646112 # free -m total used free shared buff/cache available Mem: 15874 8099 309 370 8162 7775 Swap: 25599 10873 14726 Swap is composed from 4+5 GBs of zRAM (lzo&zstd) and 16 GBs on HDD. There's also some background I/O from time to time caused by torrent seeding. vm.swappiness = 13. I misread your comment, so I didn't check all the files, but here's how /proc/pressure/ looks now after managing to close Chrome: cpu some avg10=0.00 avg60=0.00 avg300=0.07 total=188606342 cpu full avg10=0.00 avg60=0.00 avg300=0.00 total=0 io some avg10=2.07 avg60=1.50 avg300=9.23 total=2149496347 io full avg10=2.07 avg60=1.47 avg300=8.79 total=2045347731 irq full avg10=0.11 avg60=0.20 avg300=0.23 total=288863055 memory some avg10=0.00 avg60=0.00 avg300=1.22 total=94771913 memory full avg10=0.00 avg60=0.00 avg300=1.18 total=93379177
  - blueflow 1y ago
    Okay this is a legit case where having swap on a slow HDD is the main reason for stalls. 10 GB swapped, $expletive.
  - 1y ago