6 ms·

Earlyoom – Early OOM Daemon for Linux

50 points by harporoeder 3y ago

ecef9-8c0f-4374 3y ago
I use it on a 4GB PI in combination with XSuspender. It has its drawbacks, but works surprisingly well. XSuspender - Automatically suspend inactive X11 applications When an application window loses focus, XSuspender tries to match it to one of the rules in its configuration. If a match is found, the application is sent a SIGSTOP signal (preventing the process from obtaining further CPU time). Upon windows regaining focus, the process is seamlessly continued where it had left off.
hamilyon2 3y ago
Came here to tell about zram - swap in compressed RAM. Disable (or never set up in first place) on-disk swap and use compressed RAM as swap. This gives you swap, so you have advantages of that. But when you actually run out of memory, kernel oom has a chance to do it's thing, because system is still responsive in this situation.
marcodiego 3y ago
Sincere doubt: is it really needed after MGLRU?
- Timber-6539 3y ago
  Yes. MGLRU only keeps the system responsive enough so that the OOM daemon can do its job when it comes to it.
jauntywundrkind 3y ago
Why this and not Meta's oomd?
- saagarjha 3y ago
  Or Android’s lmkd?
- TheDong 3y ago
  Or systemd's systemd-oomd https://github.com/systemd/systemd/pull/15206 https://github.com/systemd/systemd/pull/15206 The answer for both of those is "if you're unfortunate enough to be running a 2.6 kernel linux server from 2005, then you can't use cgroupsv2 and thus can't use oomd or systemd-oomd". That is the only thing earlyoom is good for at this point I think, is old systems without cgroupsv2
  - xk3 3y ago
    I spent a few days trying to configure systemd-oomd to work with my machine but I kept getting hangs when running out of memory. earlyoom is a lot simpler and thus more reliable. It only took ~5 minutes to install and test to make sure it worked and I've never had a hang since then.
  - mgrandl 3y ago
    Actually no there are other reasons. Meta oomd requires you to have swap enabled, which makes it a non-starter for us. systemd-oomd has its own set of drawbacks. We evaluated all options at length and earlyoom came out as the clear winner for our usecase.
    - circularfoyers 3y ago
      Why weren't you using swap?
      - mpol 3y ago
        I'm not the parent. But in the last 5 or 8 years, Linux swap has mostly been useless to me. I run a desktop with 16GB ram, have swappiness set to 0, and even when using maybe 5GB or 7GB for running programs, it wants to swap out all kinds of stuff. When I switch to another running program I have to wait for it to get taken out of swap. But I was using that running program just a while ago, why do I have to wait? And it doesn't help in thrashing in Linux, it does it anyway.
        voxadam 3y ago
vlovich123 3y ago
Does this handle cgroups such that it would know to kill chrome even though each individual tab is only using a little bit? Could be mistaken but looking for the single largest process can be misleading if you’re dealing with applications that have forked a ton of processes…
- dale_glass 3y ago
  systemd-oomd is another thing that exists, and does of course take cgroups into account. But since chrome should survive a tab dying, just killing an individual tab should be a perfectly good strategy in most cases.
  - account42 3y ago
    The problem is not with killing a single tab rather all of chrome, the problem is with identifying that chrome is the memory hog in the first place which you won't see if you just look at individual processess because the memory usage is spread out.
    - simiones 3y ago
      What is the use case where you'd want to kill all of Chrome just because there are many tabs open? If there is a memory leak in Chrome, it will almost certainly affect a single tab process, and that can be killed. If you just need a bit more memory, killing one of Chrome's tabs may be a good choice. Otherwise, it's very likely that Chrome is more important than other processes running on the system, in any use case where a heavily used Chrome is present, isn't it? You won't have Chrome with a thousand tabs on a server system, and if you're running something like a massive compilation, you probably just want it to fail if it can't fit in RAM, not kill the whole Chrome process group to continue.
      - resoluteteeth 3y ago
        > What is the use case where you'd want to kill all of Chrome just because there are many tabs open? If there is a memory leak in Chrome, it will almost certainly affect a single tab process, and that can be killed. If you just need a bit more memory, killing one of Chrome's tabs may be a good choice. They aren't saying they want to kill all of chrome. They're saying that in order to know that the problem is that chrome is using all of the memory in order to decide to kill a single chrome tab, the daemon needs to be able to identify that all of the processes belong to one program (chrome).
fbdab103 3y ago
I have taken to running this anytime I start a jupyter notebook using more than a trivial amount of data. It has only fired off a couple of times, but I have been grateful for the safety net. Anytime I can consume my 64G of ram, I was probably doing something stupid anyway.
- number6 3y ago
  Advent of code? :)
guappa 3y ago
If you want stuff to die inexplicably, use this! (I used to, bad idea)
- omeid2 3y ago
  Does it has to be inexplicably? Is there any reason the OOM Daemon can't nicely prompt you about what process they just killed? Or even ask you if you want to kill, similar to how Firefox will ask you if you want to kill a misbehaving tab?
  - guappa 3y ago
    It just picks a big one at random. Like the shell in which you're running make, and then you have a mess. Even worse when it kills containers.
  - Santosh83 3y ago
    I guess they could attempt to do so, but under extremely severe low memory conditions, the prompt (especially if its graphical) has a good chance of not reaching the user's screen in any reasonable time. That's why it has to decide without external I/O.
    - omeid2 3y ago
      You can suspend a process, drop some caches, and even swap them to disk, if the memory approaches critical levels, or better yet, you can set a watermark that suspend the process and prompts you before it gets to that.
- harporoeder 3y ago
  I would prefer a process on my desktop be killed rather than my system become unusable to the point of requiring a hard reset of the machine as is common on Linux systems during low memory scenarios.
  - arp242 3y ago
    I've never had an OOM on Linux requiring a hard reset, but you do need some patience for the kernel's OOM killer to kick in – up to a minute or two, perhaps even a bit bit longer (I never timed it exactly).
dmitrygr 3y ago
Wouldn’t this suck for power on laptops? Doing something at a fixed frequency (especially as high as 10Hz) will cause CPU wake ups where previously you wouldn’t need any. Say goodbye to your battery life.
- sweetjuly 3y ago
  For an awake power state you almost certainly have at least one core waking up more frequently than 10Hz just to handle poking devices like the GPU. HW power management (such as through wait for interrupt instructions) operate on a much smaller time scale and so you wouldn't really be perturbing anything on a real interactive system.
  - dmitrygr 3y ago
    On phones in android we even aggregated “once per hour” wake ups to save power noticeably, so it does matter actually.
    - ungamedplayer 3y ago
      I think you might be talking about different wake ups.
    - sweetjuly 3y ago
      Wake from suspend-to-RAM (and likely with a radio power up) is going to be gobs more expensive than a wake from retention because you're both coming from a much lower power state (you were doing great before and now you're not, so a big change in energy usage over baseline) and it has a high overhead (the core cluster/SoC has to go through power on reset and get reconfigured by SW). This greatly contrasts with a wake from WFI where at minimum all architectural state is retained but generally caches and predictors are retained for quite a while.

50 points by harporoeder 3y ago

I use it on a 4GB PI in combination with XSuspender. It has its drawbacks, but works surprisingly well. XSuspender - Automatically suspend inactive X11 applications When an application window loses focus, XSuspender tries to match it to one of the rules in its configuration. If a match is found, the application is sent a SIGSTOP signal (preventing the process from obtaining further CPU time). Upon windows regaining focus, the process is seamlessly continued where it had left off.

hamilyon2 3y ago

Came here to tell about zram - swap in compressed RAM. Disable (or never set up in first place) on-disk swap and use compressed RAM as swap. This gives you swap, so you have advantages of that. But when you actually run out of memory, kernel oom has a chance to do it's thing, because system is still responsive in this situation.

marcodiego 3y ago

Sincere doubt: is it really needed after MGLRU?

Timber-6539 3y ago
Yes. MGLRU only keeps the system responsive enough so that the OOM daemon can do its job when it comes to it.

jauntywundrkind 3y ago

Why this and not Meta's oomd?

saagarjha 3y ago
Or Android’s lmkd?
TheDong 3y ago
Or systemd's systemd-oomd https://github.com/systemd/systemd/pull/15206 https://github.com/systemd/systemd/pull/15206 The answer for both of those is "if you're unfortunate enough to be running a 2.6 kernel linux server from 2005, then you can't use cgroupsv2 and thus can't use oomd or systemd-oomd". That is the only thing earlyoom is good for at this point I think, is old systems without cgroupsv2
- xk3 3y ago
  I spent a few days trying to configure systemd-oomd to work with my machine but I kept getting hangs when running out of memory. earlyoom is a lot simpler and thus more reliable. It only took ~5 minutes to install and test to make sure it worked and I've never had a hang since then.
- mgrandl 3y ago
  Actually no there are other reasons. Meta oomd requires you to have swap enabled, which makes it a non-starter for us. systemd-oomd has its own set of drawbacks. We evaluated all options at length and earlyoom came out as the clear winner for our usecase.
  - circularfoyers 3y ago
    Why weren't you using swap?
    - mpol 3y ago
      I'm not the parent. But in the last 5 or 8 years, Linux swap has mostly been useless to me. I run a desktop with 16GB ram, have swappiness set to 0, and even when using maybe 5GB or 7GB for running programs, it wants to swap out all kinds of stuff. When I switch to another running program I have to wait for it to get taken out of swap. But I was using that running program just a while ago, why do I have to wait? And it doesn't help in thrashing in Linux, it does it anyway.
      - voxadam 3y ago

vlovich123 3y ago

Does this handle cgroups such that it would know to kill chrome even though each individual tab is only using a little bit? Could be mistaken but looking for the single largest process can be misleading if you’re dealing with applications that have forked a ton of processes…

dale_glass 3y ago
systemd-oomd is another thing that exists, and does of course take cgroups into account. But since chrome should survive a tab dying, just killing an individual tab should be a perfectly good strategy in most cases.
- account42 3y ago
  The problem is not with killing a single tab rather all of chrome, the problem is with identifying that chrome is the memory hog in the first place which you won't see if you just look at individual processess because the memory usage is spread out.
  - simiones 3y ago
    What is the use case where you'd want to kill all of Chrome just because there are many tabs open? If there is a memory leak in Chrome, it will almost certainly affect a single tab process, and that can be killed. If you just need a bit more memory, killing one of Chrome's tabs may be a good choice. Otherwise, it's very likely that Chrome is more important than other processes running on the system, in any use case where a heavily used Chrome is present, isn't it? You won't have Chrome with a thousand tabs on a server system, and if you're running something like a massive compilation, you probably just want it to fail if it can't fit in RAM, not kill the whole Chrome process group to continue.
    - resoluteteeth 3y ago
      > What is the use case where you'd want to kill all of Chrome just because there are many tabs open? If there is a memory leak in Chrome, it will almost certainly affect a single tab process, and that can be killed. If you just need a bit more memory, killing one of Chrome's tabs may be a good choice. They aren't saying they want to kill all of chrome. They're saying that in order to know that the problem is that chrome is using all of the memory in order to decide to kill a single chrome tab, the daemon needs to be able to identify that all of the processes belong to one program (chrome).

fbdab103 3y ago

I have taken to running this anytime I start a jupyter notebook using more than a trivial amount of data. It has only fired off a couple of times, but I have been grateful for the safety net. Anytime I can consume my 64G of ram, I was probably doing something stupid anyway.

number6 3y ago
Advent of code? :)

guappa 3y ago

If you want stuff to die inexplicably, use this! (I used to, bad idea)

omeid2 3y ago
Does it has to be inexplicably? Is there any reason the OOM Daemon can't nicely prompt you about what process they just killed? Or even ask you if you want to kill, similar to how Firefox will ask you if you want to kill a misbehaving tab?
- guappa 3y ago
  It just picks a big one at random. Like the shell in which you're running make, and then you have a mess. Even worse when it kills containers.
- Santosh83 3y ago
  I guess they could attempt to do so, but under extremely severe low memory conditions, the prompt (especially if its graphical) has a good chance of not reaching the user's screen in any reasonable time. That's why it has to decide without external I/O.
  - omeid2 3y ago
    You can suspend a process, drop some caches, and even swap them to disk, if the memory approaches critical levels, or better yet, you can set a watermark that suspend the process and prompts you before it gets to that.
harporoeder 3y ago
I would prefer a process on my desktop be killed rather than my system become unusable to the point of requiring a hard reset of the machine as is common on Linux systems during low memory scenarios.
- arp242 3y ago
  I've never had an OOM on Linux requiring a hard reset, but you do need some patience for the kernel's OOM killer to kick in – up to a minute or two, perhaps even a bit bit longer (I never timed it exactly).

dmitrygr 3y ago

Wouldn’t this suck for power on laptops? Doing something at a fixed frequency (especially as high as 10Hz) will cause CPU wake ups where previously you wouldn’t need any. Say goodbye to your battery life.

sweetjuly 3y ago
For an awake power state you almost certainly have at least one core waking up more frequently than 10Hz just to handle poking devices like the GPU. HW power management (such as through wait for interrupt instructions) operate on a much smaller time scale and so you wouldn't really be perturbing anything on a real interactive system.
- dmitrygr 3y ago
  On phones in android we even aggregated “once per hour” wake ups to save power noticeably, so it does matter actually.
  - ungamedplayer 3y ago
    I think you might be talking about different wake ups.
  - sweetjuly 3y ago
    Wake from suspend-to-RAM (and likely with a radio power up) is going to be gobs more expensive than a wake from retention because you're both coming from a much lower power state (you were doing great before and now you're not, so a big change in energy usage over baseline) and it has a high overhead (the core cluster/SoC has to go through power on reset and get reconfigured by SW). This greatly contrasts with a wake from WFI where at minimum all architectural state is retained but generally caches and predictors are retained for quite a while.