7 ms·

> That work did not address one other unfortunate characteristic of the OOM killer, though: its opinion of what is the least important process on the system ten

> That work did not address one other unfortunate characteristic of the OOM killer, though: its opinion of what is the least important process on the system tends to differ from that of the system's users. My experience of the linux OOM killer is not that its opinion differs from mine but that it has no opinion at all for a long, long time after the system is in deep trouble. The OOM killer simply does not act quickly enough to save systems. Sadly it's not customisable but 'earlyoom' (packaged for debian and probably everything else) is. I turned it on when I was messing about when debugging a badly behaved bit of software which went into a memory allocation loop and have just left it on. It's saved me a few times and I now plan to leave it on forever. Looks like oomd is an idea along the same lines but with slightly different goals. It's not in my distro so not an easy option for me.

freedomben 5y ago
This is my experience also. By the time the OOM killer kicks in, the system has been locked for 15 to 20 minutes already. If it was production you've already terminated the instance. If it's your laptop or desktop, you've already held the power button. Fedora has earlyoom enabled by default but so far it hasn't saved me. I really need to look into configuring it. How did you get started? Man pages? Blog post?
- cmurf 5y ago
  earlyoom is enable on Fedora 32 and 33 Workstation edition; and 33 KDE spin. On Fedora 34, all editions and spins have systemd-oomd enabled. It does take some initial configuration of systemd service units since oomd works by cgroupsv2 based accounting and killing of entire cgroups, not PIDs. This work is still a work in progress, with uresourced setting up the initial resource allocations (with planned obsolescence). It should be safe to run uresourced on any edition or spin but right now it's only enabled by default on Workstation edition. If the results you're getting aren't expected, there's still some chance it's a bug somewhere, so you should report it against systemd, attach `journalctl -b -o short-monotonic --no-hostname` or at least ~10 minutes of logging prior to the unexpected behavior you're reporting.
- calpaterson 5y ago
  The manpage combined with some (forced) experimentation with the wonky code mentioned was enough for me. I run with this config as even the earlyoom defaults were not strict enough: -r 30 -m 5 -s 80 I run with 16gb of memory and do use swap. In practice if swap is growing at all once memory is near full, I'm in trouble and action needs to be taken.
- garaetjjte 5y ago
  >If it's your laptop or desktop, you've already held the power button. Note that if enabled you can use SysRq keys to invoke OOM killer manually.
- binarybanana 5y ago
  While that is true, you can invoke the OOM killer at any time you want by pressing Ctrl+Alt+Print("SysRq")+F(uck?). Given the letter they picked it seems like they're fully aware of the deficiency. As to why it was never really addressed, that's a good question. The recent le9[1] patch addresses the most annoying symptom of running out of memory by keeping some amount of RAM reserved for clean pages (like static program code) so that those won't get swapped in and out so much. That greatly improves responsiveness under low memory conditions. [1]: https://github.com/hakavlad/le9-patch https://github.com/hakavlad/le9-patch
GranPC 5y ago
I used to experience this a long time ago, and I know many people who still do - but on my system (running an Ubuntu 18.04 derived distribution) the OOM killer takes at most 3-4 seconds to step in and kill whichever process is consuming the most memory. Does anyone know if Ubuntu/Debian tunes the OOM killer differently to try and stop this from happening?
- disabled 5y ago
  This post sums it up best: https://news.ycombinator.com/item?id=27652599 https://news.ycombinator.com/item?id=27652599
  - GranPC 5y ago
    I don't see how any of that relates to the OOM killer, am I missing something?
jeffbee 5y ago
The reason Linux systems lock up like that is because the kernel will let a process fill all memory up with dirty pages that need writeback, then as soon as it needs some memory the first thing it does it drops all of the in-memory copies of file-backed pages, which includes all of your programs. Then whenever one of your programs wants to run, or continue running by branching to a far address, it has to swap in that code from disk again. Even though you thought your system does not "have swap", it does have swap in effect. The workaround for this is to copy important programs into memory and pin them there with mlock. It is particularly important that if you rely on a userspace OOM killer it gets locked into memory.
- phsau 5y ago
  I've also found that under these conditions kswapd will effectively consume all your CPU time. The time it spends running is probably proportional to your maximum memory too - in our case it parses through 500+GB of LRU. The blocking writeback behaviour can be managed effectively with dirty page writeback ratio tuning. You don't want to block trying to write 50GB to disk at once when you hit the high dirty page watermark.
- scottlamb 5y ago
  > The workaround for this is to copy important programs into memory and pin them there with mlock. Nit: I don't think it's necessary to copy the program into anonymous memory to use mlock. You might be thinking of huge pages (transparent or otherwise), which are supported on anonymous pages and unfortunately aren't yet supported on ext4- or btrfs-backed file pages.
- pengaru 5y ago
  The resource control aspects of cgroups actually improve this situation, if you take advantage of them. You'll still see that page fault thrashing but it becomes isolated to the cgroup experiencing the memory pressure. It doesn't bring down the entire system in my experience.
- chrsig 5y ago
  I'd never considered the program itself getting dropped and re-read from disk -- but it makes perfect sense. I'm curious if executing the programs from a ramdisk would achieve the same effect.
saurik 5y ago
FWIW, I have a very distinct memory of the time OOM decided to kill init (pid 1), which was the most incredibly incompetent choice it possibly could have made ;P.
hjek 5y ago
You should try `nohang`. It comes with nice notifications about memory status and when it kills a process. I find that it reacts timely.

freedomben 5y ago

This is my experience also. By the time the OOM killer kicks in, the system has been locked for 15 to 20 minutes already. If it was production you've already terminated the instance. If it's your laptop or desktop, you've already held the power button. Fedora has earlyoom enabled by default but so far it hasn't saved me. I really need to look into configuring it. How did you get started? Man pages? Blog post?

cmurf 5y ago
earlyoom is enable on Fedora 32 and 33 Workstation edition; and 33 KDE spin. On Fedora 34, all editions and spins have systemd-oomd enabled. It does take some initial configuration of systemd service units since oomd works by cgroupsv2 based accounting and killing of entire cgroups, not PIDs. This work is still a work in progress, with uresourced setting up the initial resource allocations (with planned obsolescence). It should be safe to run uresourced on any edition or spin but right now it's only enabled by default on Workstation edition. If the results you're getting aren't expected, there's still some chance it's a bug somewhere, so you should report it against systemd, attach `journalctl -b -o short-monotonic --no-hostname` or at least ~10 minutes of logging prior to the unexpected behavior you're reporting.
calpaterson 5y ago
The manpage combined with some (forced) experimentation with the wonky code mentioned was enough for me. I run with this config as even the earlyoom defaults were not strict enough: -r 30 -m 5 -s 80 I run with 16gb of memory and do use swap. In practice if swap is growing at all once memory is near full, I'm in trouble and action needs to be taken.
garaetjjte 5y ago
>If it's your laptop or desktop, you've already held the power button. Note that if enabled you can use SysRq keys to invoke OOM killer manually.
binarybanana 5y ago
While that is true, you can invoke the OOM killer at any time you want by pressing Ctrl+Alt+Print("SysRq")+F(uck?). Given the letter they picked it seems like they're fully aware of the deficiency. As to why it was never really addressed, that's a good question. The recent le9[1] patch addresses the most annoying symptom of running out of memory by keeping some amount of RAM reserved for clean pages (like static program code) so that those won't get swapped in and out so much. That greatly improves responsiveness under low memory conditions. [1]: https://github.com/hakavlad/le9-patch https://github.com/hakavlad/le9-patch

GranPC 5y ago

I used to experience this a long time ago, and I know many people who still do - but on my system (running an Ubuntu 18.04 derived distribution) the OOM killer takes at most 3-4 seconds to step in and kill whichever process is consuming the most memory. Does anyone know if Ubuntu/Debian tunes the OOM killer differently to try and stop this from happening?

disabled 5y ago
This post sums it up best: https://news.ycombinator.com/item?id=27652599 https://news.ycombinator.com/item?id=27652599
- GranPC 5y ago
  I don't see how any of that relates to the OOM killer, am I missing something?

jeffbee 5y ago

The reason Linux systems lock up like that is because the kernel will let a process fill all memory up with dirty pages that need writeback, then as soon as it needs some memory the first thing it does it drops all of the in-memory copies of file-backed pages, which includes all of your programs. Then whenever one of your programs wants to run, or continue running by branching to a far address, it has to swap in that code from disk again. Even though you thought your system does not "have swap", it does have swap in effect. The workaround for this is to copy important programs into memory and pin them there with mlock. It is particularly important that if you rely on a userspace OOM killer it gets locked into memory.

phsau 5y ago
I've also found that under these conditions kswapd will effectively consume all your CPU time. The time it spends running is probably proportional to your maximum memory too - in our case it parses through 500+GB of LRU. The blocking writeback behaviour can be managed effectively with dirty page writeback ratio tuning. You don't want to block trying to write 50GB to disk at once when you hit the high dirty page watermark.
scottlamb 5y ago
> The workaround for this is to copy important programs into memory and pin them there with mlock. Nit: I don't think it's necessary to copy the program into anonymous memory to use mlock. You might be thinking of huge pages (transparent or otherwise), which are supported on anonymous pages and unfortunately aren't yet supported on ext4- or btrfs-backed file pages.
pengaru 5y ago
The resource control aspects of cgroups actually improve this situation, if you take advantage of them. You'll still see that page fault thrashing but it becomes isolated to the cgroup experiencing the memory pressure. It doesn't bring down the entire system in my experience.
chrsig 5y ago
I'd never considered the program itself getting dropped and re-read from disk -- but it makes perfect sense. I'm curious if executing the programs from a ramdisk would achieve the same effect.

saurik 5y ago

FWIW, I have a very distinct memory of the time OOM decided to kill init (pid 1), which was the most incredibly incompetent choice it possibly could have made ;P.

hjek 5y ago

You should try `nohang`. It comes with nice notifications about memory status and when it kills a process. I find that it reacts timely.