3 ms·

> If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediate

> If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately! You are not low on ram! I'm running RKE2 on my desktop and it'll start killing pods due to low memory pressure, even though the memory was only used for disk caching. I wonder if there is any way to make it stop doing that and instead only start killing pods if it's due to "real" low memory pressure.

blueflow 3y ago
Think about it: A processes executable code comes from a file. You will need the size of the executable available as disk cache or the program execution will cause heavy thrashing and I/O. So some part of it is "real" memory pressure.
- neurostimulant 3y ago
  I also run an nfs server in the same machine, so after a period of heavy nfs use, most of the ram were eaten by the disk cache and rke2/kubernetes start having memory pressure taint. After a fresh restart with all pods running, the memory usage is below 10%, so I doubt the disk cache was full with executable files cache.
okwhateverdude 3y ago
Assuming Linux, oddly enough I came across this exact behavior[0] while researching resource management for an on-prem k8s cluster. Take a look at that thread for more info, but TL;DR, you need to actually finesse page cache constraints if you want avoid the behavior. You actually can have really fine grained control over page cache via cgroups v2[1][2] and systemd[3]. [0]: https://github.com/kubernetes/kubernetes/issues/43916 https://github.com/kubernetes/kubernetes/issues/43916 [1]: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-in... [2]: https://biriukov.dev/docs/page-cache/6-cgroup-v2-and-page-cache/ https://biriukov.dev/docs/page-cache/6-cgroup-v2-and-page-ca... [3]: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html https://www.freedesktop.org/software/systemd/man/systemd.res...
- neurostimulant 3y ago
  Thank you for the pointers. That's a lot of things to learn since I never look into cgroup before. I'll see if there is something better there than my current "fix" (periodically run `echo 1 > /proc/sys/vm/drop_caches`).

blueflow 3y ago

Think about it: A processes executable code comes from a file. You will need the size of the executable available as disk cache or the program execution will cause heavy thrashing and I/O. So some part of it is "real" memory pressure.

neurostimulant 3y ago
I also run an nfs server in the same machine, so after a period of heavy nfs use, most of the ram were eaten by the disk cache and rke2/kubernetes start having memory pressure taint. After a fresh restart with all pods running, the memory usage is below 10%, so I doubt the disk cache was full with executable files cache.

okwhateverdude 3y ago

Assuming Linux, oddly enough I came across this exact behavior[0] while researching resource management for an on-prem k8s cluster. Take a look at that thread for more info, but TL;DR, you need to actually finesse page cache constraints if you want avoid the behavior. You actually can have really fine grained control over page cache via cgroups v2[1][2] and systemd[3]. [0]: https://github.com/kubernetes/kubernetes/issues/43916 https://github.com/kubernetes/kubernetes/issues/43916 [1]: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-in... [2]: https://biriukov.dev/docs/page-cache/6-cgroup-v2-and-page-cache/ https://biriukov.dev/docs/page-cache/6-cgroup-v2-and-page-ca... [3]: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html https://www.freedesktop.org/software/systemd/man/systemd.res...

neurostimulant 3y ago
Thank you for the pointers. That's a lot of things to learn since I never look into cgroup before. I'll see if there is something better there than my current "fix" (periodically run `echo 1 > /proc/sys/vm/drop_caches`).