4 ms·

This is called an oomkiller. The kernel has one but it kicks in very late and the kernel prefers to do page trashing instead of killing processes. systemd-oomd

This is called an oomkiller. The kernel has one but it kicks in very late and the kernel prefers to do page trashing instead of killing processes. systemd-oomd should be integrated in systemd, you can configure it to your liking and see if it improves your problem.

ButlerianJihad 22d ago
I believe that you mean: https://en.wikipedia.org/wiki/Thrashing_(computer_science) https://en.wikipedia.org/wiki/Thrashing_(computer_science) Chris Siebenmann discusses when the OOM killer triggers: https://utcc.utoronto.ca/~cks/space/blog/linux/OOMKillerWhen https://utcc.utoronto.ca/~cks/space/blog/linux/OOMKillerWhen Chris disables systemd-oomd after it obliterates his X session with no explanation: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNowDisabled https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...
- rovr138 22d ago
  > First off, this is exactly how systemd-oomd is supposed to behave under memory pressure. The documentation is specific on this; systemd-oomd itself says: > > [...] If the configured limits are exceeded, systemd-oomd will select a cgroup to terminate, and send SIGKILL to all processes in it. [...] > By having the user@.service template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes (across all of your sessions, if you have more than one). ... Maybe *Fedora* has fixed or improved in the last 4 years. Or maybe they don't run Fedora.
  - BenjiWiebe 21d ago
    One thing Fedora does now, is use zram. In my experience it works really well. I wonder why my computer is a bit sluggish, and find out I have several gigs in zram. If that was in swap on a disk, it would be really painful.
    - MrDrMcCoy 21d ago
      If it were swap on disk fronted by zswap, it'd be even better ;)
  - rwmj 21d ago
    I'm a Fedora developer and I can assure you that Fedora's behaviour when it runs out of memory is still terrible.
miladyincontrol 21d ago
Adding to this, glad systemd-oomd finally added solid rulesets in 261 Made it far easier to target any containers that got too hot rather than ever risk anything higher priority.
SoftTalker 21d ago
I wish there was an easy way to configure it to say "target user processes first, specifically java (or these days python)" as in my experience they are always the culprits. Processes owned by system accounts or root should be the last ones killed.
- bigstrat2003 21d ago
  Similarly, in the past I have wished for the ability to exempt a process from the oomkiller. I've run servers where the top memory user was also the server's entire reason for existence, and if that process gets killed the server may as well be down. It would literally have been better for any other process to get killed, but it was always the application process because of the memory usage.
- mixmastamyk 20d ago
  systemd-oomd works reasonably well and there is source code. Perhaps claude can help add more detailed policy support to it. Just found this comment: https://news.ycombinator.com/item?id=49663299 https://news.ycombinator.com/item?id=49663299

ButlerianJihad 22d ago

I believe that you mean: https://en.wikipedia.org/wiki/Thrashing_(computer_science) https://en.wikipedia.org/wiki/Thrashing_(computer_science) Chris Siebenmann discusses when the OOM killer triggers: https://utcc.utoronto.ca/~cks/space/blog/linux/OOMKillerWhen https://utcc.utoronto.ca/~cks/space/blog/linux/OOMKillerWhen Chris disables systemd-oomd after it obliterates his X session with no explanation: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNowDisabled https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...

rovr138 22d ago
> First off, this is exactly how systemd-oomd is supposed to behave under memory pressure. The documentation is specific on this; systemd-oomd itself says: > > [...] If the configured limits are exceeded, systemd-oomd will select a cgroup to terminate, and send SIGKILL to all processes in it. [...] > By having the user@.service template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes (across all of your sessions, if you have more than one). ... Maybe *Fedora* has fixed or improved in the last 4 years. Or maybe they don't run Fedora.
- BenjiWiebe 21d ago
  One thing Fedora does now, is use zram. In my experience it works really well. I wonder why my computer is a bit sluggish, and find out I have several gigs in zram. If that was in swap on a disk, it would be really painful.
  - MrDrMcCoy 21d ago
    If it were swap on disk fronted by zswap, it'd be even better ;)
- rwmj 21d ago
  I'm a Fedora developer and I can assure you that Fedora's behaviour when it runs out of memory is still terrible.

miladyincontrol 21d ago

Adding to this, glad systemd-oomd finally added solid rulesets in 261 Made it far easier to target any containers that got too hot rather than ever risk anything higher priority.

SoftTalker 21d ago

I wish there was an easy way to configure it to say "target user processes first, specifically java (or these days python)" as in my experience they are always the culprits. Processes owned by system accounts or root should be the last ones killed.

bigstrat2003 21d ago
Similarly, in the past I have wished for the ability to exempt a process from the oomkiller. I've run servers where the top memory user was also the server's entire reason for existence, and if that process gets killed the server may as well be down. It would literally have been better for any other process to get killed, but it was always the application process because of the memory usage.
mixmastamyk 20d ago
systemd-oomd works reasonably well and there is source code. Perhaps claude can help add more detailed policy support to it. Just found this comment: https://news.ycombinator.com/item?id=49663299 https://news.ycombinator.com/item?id=49663299