5 ms·

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running ou

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

stathibus 3y ago
early-oom
manymatter 3y ago
I've never had that problem. Maybe you don't setup much swap space as I would expect the system to run well enough to save your work and shutdown for most common leaks.
- gkbrk 3y ago
  Swap usually makes it worse, without swap there is some chance that the Linux OOM killer does something useful and saves the system. With swap, it becomes a frozen system that never manages to kill anything due to all the swapping. You can wait 5 minutes, 10 minutes, or 15 minutes, but the system never recovers without a reboot.
  - manymatter 3y ago
    That would normally mean you ignore the system getting slower for a long time.
    - gkbrk 3y ago
      Not a long time. With swap enabled, when a process consumes too much memory your system goes from perfect performance to cursor lagging to everything is frozen and you can't even switch to a TTY within 5-10 seconds. Without swap, the system lags for a couple seconds, OOM killer frees up memory and you're good to go again. The only slowdown is any pages that were kicked out from the file cache. But those quickly come back after the OOM killer does its thing.
      - manymatter 3y ago
        What if the thing you kill is in the critical stack to saving your work? If it isn't I don't really understand why you would be swapping it in a lot I would view the OOm solution as a compute as cattle thinb, but here we are talking about a user desktop where the user can take the best action for themselves once they realize there's a problem.
        gkbrk 3y ago
        > If it isn't I don't really understand why you would be swapping it in a lot. The user doesn't decide which processes are swapped in. If the process gets CPU time and tries to access its data, that data will get swapped in. > We are talking about a user desktop where the user can take the best action for themselves once they realize there's a problem. You can't do that with swap, because once you realize there's a problem, you cannot even move your cursor or run commands to take any actions.
        manymatter 3y ago
        This sounds nothing like the gradual leak problem described.. An OOM killer is great once you actually run out of resources, removing virtual resources to always run out and play roulette is using it as a fad hammer.
    - grumbel 3y ago
      You have only about 10sec between the system getting slower and the system locking up completely. If you manage to hit the Magic SysRq key combination to trigger OOM manually, that can save the system, but you have to be quick. [1] https://en.wikipedia.org/wiki/Magic_SysRq_key https://en.wikipedia.org/wiki/Magic_SysRq_key
  - ynik 3y ago
    That only works if the system is accessing mostly anonymous pages. If the load on the system is accessing plenty of mmapped code/data pages, it can still trash those even if swap is disabled. I've still seen systems hanging for 10+ minutes without recovering even though swap was already disabled. The Linux kernel OOM killer only acts if there's nothing left that can be discarded, which often happens way too late to save the system. You need a user-mode OOM-killer like earlyoom if you want to keep the system responsive.
    - gkbrk 3y ago
      Yes, I don't disagree that user-mode OOM-killers are helpful, or that the system can still hang without swap. I'm just saying turning on swap, or increasing the swap capacity does not fix the problem, and it usually makes it worse.
    - lokar 3y ago
      This is why on production systems important binaries should mlock all of their code pages into ram at startup.
      - Always_Anon 3y ago
        [dead]
- chrismorgan 3y ago
  My previous Linux laptop with 8GB of RAM (2014–2017 or so, I think) I never even got round to setting up swap, and I only ran into problems two or three times total (when running two or three Firefox instances and Chromium and a VM taking 2GB and a bunch of other things running—some of the biggest consumers, namely browsers, actually notice if they’re using too much of your RAM and adjust how they work so they don’t ask for so much RAM, at what I believe is a fairly slight performance cost). On my current laptop and with my current habits, I’m consuming a lot more memory, and so the technique I use to avoid OOM is simply having 40GB of RAM. (As it happens, I do actually have swap set up at present, because certain circumstances meant I wanted to hibernate it occasionally; should disable swap again now I’m back to not needing to hibernate.)
  - isthatafact 3y ago
    > some of the biggest consumers, namely browsers, actually notice if they’re using too much of your RAM and adjust how they work so they don’t ask for so much RAM Maybe there are reasons, but it is not nearly good enough -- I frequently run out of RAM and encounter OOM kills (prefer not to deal with linux swap), usually requiring a reboot. I really wish that I could just set an upper limit on (e.g.) firefox RAM usage -- 8GB for example -- instead of its insistence on using all of the unused RAM minus a couple to several hundred MB, which does not leave enough room for memory usage spikes. There might even be a way to set this buried somewhere in the config parameters, but I could never find it. It must be technically possible to set some limit because otherwise the browser would not be able to maintain a somewhat consistent usage just below the total system RAM.
TheDong 3y ago
systemd-oomd [0]. You can also use facebook's oomd, but the systemd one is basically the same and slightly better. If you want, you can also write your own custom memory pressure handler using the recent PSI features: https://docs.kernel.org/accounting/psi.html https://docs.kernel.org/accounting/psi.html The other, easier, answer is to buy more ram, if you simply have more memory than you will ever use (such as having 128GiB of memory when you only regularly use 30GiB), you'll rarely run into OOM issues. [0]: https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html https://www.freedesktop.org/software/systemd/man/latest/syst...
- nucleardog 3y ago
  > The other, easier, answer is to buy more ram, if you simply have more memory than you will ever use (such as having 128GiB of memory when you only regularly use 30GiB), you'll rarely run into OOM issues. Wanna emphasize this mostly because it took me way too long to think of. Was dealing with memory pressure regularly and constantly getting frustrated… then dropped an entire $60 or something to throw another 32GB in the machine and never think about my memory again. Like a week later when I ran out of disk space again and was struggling to make room… same realization. Spent $75 for a 1TB NVME SSD dropped in my mailbox two days later and doubled my storage. So we’re talking $135 to double my RAM and storage and stop wasting time and brain space on this stuff.
  - CableNinja 3y ago
    In the words of a past employer.. We ran out of space, and then we realized we are a storage company.
melvyn2 3y ago
For desktop use, nohang does what the name says. https://github.com/hakavlad/nohang https://github.com/hakavlad/nohang
magicalhippo 3y ago
This is one area where Windows is far superior to Linux as a desktop experience. By accident, I once wrote an infinite loop that just allocated a bunch of memory. As I ran the program my Windows system quickly became laggy and unresponsive. However, not completely. With a bit of patience I got Task Manager up and managed to kill the process and the system was back to normal within a minute. On Linux, I experience the same as you. Everything is fine until suddenly it's completely unresponsive. Most of the time I can't even manage to shut down the system cleanly and have to hard reset.
- gnramires 3y ago
  Indeed! I wonder why the system can't just stop giving the process memory when it's starting to become critical? (say, you're over 95% memory used) I really think a more general solution is treating memory (and CPU time as welL!) as a scarce resource, and programs should either: (1) Deal with having severely denied resources (default behavior); (2) Use a communication protocol to negotiate memory with the system. Negotiation could mean (a) The process freeing memory spontaneously when there's pressure (another process requires it) and it's inactive, (b) Requesting the system free memory from other processes when user requires. This would impose more memory management burden on developers (trying to fulfill OS requests), but in turn it would make for a far better experience when there's memory contention. Disclaimer: I have no idea how memory allocation works in detail. Maybe this is reinventing the wheel?
  - willcipriano 3y ago
    > I wonder why the system can't just stop giving the process memory when it's starting to become critical If you turn swap off the allocator will error out when you are out of memory. I don't think most software handles that error but all the wiring is there to do that.
  - roblabla 3y ago
    > Maybe this is reinventing the wheel? I believe something like that exists on mobile platforms for memory, at least on iOS you get a message (applicationDidReceiveMemoryWarning:) when the system is memory starved and wants your app to free memory. If you don't release enough memory and the memory pressure doesn't go down, the system will start killing apps.
Kwpolska 3y ago
Buying more RAM?
- microtonal 3y ago
  At the risk of this being tongue-in-cheeck: if you have a rogue process, any amount of RAM is going to be eaten up. (It sometimes happens in software development and it's nice if it doesn't take your machine in swap hell as a result.)
- alexeiz 3y ago
  Better yet, download more RAM.
Toorkit 3y ago
I recently installed Linux on my main desktop again. After freezing up twice in a week, I looked around. I enabled systemd-oom, but it hasn't been necessary, because I also enabled zram as a swap device. I've seen it compress multiple gb of ram into a few hundred mb and haven't frozen my desktop since.
seba_dos1 3y ago
You want to enable thrashing prevention: https://docs.kernel.org/next/admin-guide/mm/multigen_lru.html#thrashing-prevention https://docs.kernel.org/next/admin-guide/mm/multigen_lru.htm... Make sure you use zram (or zswap) as well. May also consider enabling userspace oom handler like others already suggested, but it's less important than the rest.
mgdlbp 3y ago
Enable Alt+SysRq+F and do it manually. Works in seconds every time; the kernel can take days because it's not eager enough when thrashing. Can have security implications, but the target is chosen in the typical way and screen locks should be setting a score_adj to avoid being picked.
- deleted 3y ago
  [deleted]
wakeupcall 3y ago
I'm using earlyoom on desktop, systemd-oomd on servers. Due to cgroupv2, systemd-oomd tends to kill entire process groups instead of a single run-away process [1]. It's perfect for servers where almost every process is part of a system service already. For desktop usage I found this behavior unacceptable. I would sometimes get my entire X session killed when trying something dumb (which is the reason I run such a daemon in the first place), or my editor when spawning a helper subprocess... earlyoom was the simplest daemon of the bunch I tried, and worked out of the box without special configuration. [1] Or at least this was the state ~1yr ago.
grumbel 3y ago
More RAM. It's the only thing that can reliably prevent it. Magic SysRq key[1] can trigger the OOM manually, which can sometimes rescue the system if you notice it soon enough. But if you run into the situation regularly under normal use, not programs running amok, more RAM is a must. If you can't upgrade the RAM, zram[2] is also worth a try. [1] https://en.wikipedia.org/wiki/Magic_SysRq_key https://en.wikipedia.org/wiki/Magic_SysRq_key [2] https://en.wikipedia.org/wiki/Zram https://en.wikipedia.org/wiki/Zram
marcodiego 3y ago
Is this still the case on a modern distro using MGLRU and systemd-oomd?
- seba_dos1 3y ago
  Not really, but not all distros have caught up yet.
DefineOutside 3y ago
Look into oom_kill_allocating_task. There's likely a better solution such as an earlier OOM, but killing the wrong process seems better than locking the system.
larve 3y ago
In a previous embedded job, I built a daemon watching a couple of “at risk of eating too much memory” processes that were not critical for the actual function of the machine, and killed them proactively when memory started getting low. This prevented the system from going into starvation and spending 10 seconds doing whatever each time an Ethernet packet came in. As for my pragmatic solution at home: max out the physical RAM…
mgdlbp 3y ago
Tangentially, an issue with the desktop use case is that the main process of a long-running browser is often the first to be killed, instead of one of its content processes, which it can survive losing. This is the case since 2019 as the kernel no longer prefers killing children.[0] As far as I know, unless a browser manages its oom_score_adj, which none do, a wrapper is insufficient as a workaround and a daemon would have to do so on its behalf, because the value is inherited when forking. [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bbbe48029720d2c6b6733f78d02571a281511adb https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
mattpallissard 3y ago
Wrap everything in systemd-run. I documented some of it here on my shitty site. Some of the memory limits have changed over time, notably the browsers unit https://pallissard.net/2022/06/27/limiting_application_resources/ https://pallissard.net/2022/06/27/limiting_application_resou... Edit: I like your user space oomd idea though, that would be killer
alexeiz 3y ago
I'm on openSUSE TW, and I considered systemd-oom, but the setup/configuration procedure for it is pretty elaborate. So I opted for earlyoom, which worked just a fine and required no manual configuration whatsoever. In my opinion, if your distro already integrates systemd-oom, like Fedora, it's easier to just use that. Otherwise earlyoom is the next best choice. Also it's recommended to enable zram. It's much faster than a swap partition or file. The combination of earlyoom and zram made my system much more responsive when it's running short of memory.

stathibus 3y ago

early-oom

manymatter 3y ago

I've never had that problem. Maybe you don't setup much swap space as I would expect the system to run well enough to save your work and shutdown for most common leaks.

gkbrk 3y ago
Swap usually makes it worse, without swap there is some chance that the Linux OOM killer does something useful and saves the system. With swap, it becomes a frozen system that never manages to kill anything due to all the swapping. You can wait 5 minutes, 10 minutes, or 15 minutes, but the system never recovers without a reboot.
- manymatter 3y ago
  That would normally mean you ignore the system getting slower for a long time.
  - gkbrk 3y ago
    Not a long time. With swap enabled, when a process consumes too much memory your system goes from perfect performance to cursor lagging to everything is frozen and you can't even switch to a TTY within 5-10 seconds. Without swap, the system lags for a couple seconds, OOM killer frees up memory and you're good to go again. The only slowdown is any pages that were kicked out from the file cache. But those quickly come back after the OOM killer does its thing.
    - manymatter 3y ago
      What if the thing you kill is in the critical stack to saving your work? If it isn't I don't really understand why you would be swapping it in a lot I would view the OOm solution as a compute as cattle thinb, but here we are talking about a user desktop where the user can take the best action for themselves once they realize there's a problem.
      - gkbrk 3y ago
        > If it isn't I don't really understand why you would be swapping it in a lot. The user doesn't decide which processes are swapped in. If the process gets CPU time and tries to access its data, that data will get swapped in. > We are talking about a user desktop where the user can take the best action for themselves once they realize there's a problem. You can't do that with swap, because once you realize there's a problem, you cannot even move your cursor or run commands to take any actions.
        manymatter 3y ago
        This sounds nothing like the gradual leak problem described.. An OOM killer is great once you actually run out of resources, removing virtual resources to always run out and play roulette is using it as a fad hammer.
  - grumbel 3y ago
    You have only about 10sec between the system getting slower and the system locking up completely. If you manage to hit the Magic SysRq key combination to trigger OOM manually, that can save the system, but you have to be quick. [1] https://en.wikipedia.org/wiki/Magic_SysRq_key https://en.wikipedia.org/wiki/Magic_SysRq_key
- ynik 3y ago
  That only works if the system is accessing mostly anonymous pages. If the load on the system is accessing plenty of mmapped code/data pages, it can still trash those even if swap is disabled. I've still seen systems hanging for 10+ minutes without recovering even though swap was already disabled. The Linux kernel OOM killer only acts if there's nothing left that can be discarded, which often happens way too late to save the system. You need a user-mode OOM-killer like earlyoom if you want to keep the system responsive.
  - gkbrk 3y ago
    Yes, I don't disagree that user-mode OOM-killers are helpful, or that the system can still hang without swap. I'm just saying turning on swap, or increasing the swap capacity does not fix the problem, and it usually makes it worse.
  - lokar 3y ago
    This is why on production systems important binaries should mlock all of their code pages into ram at startup.
    - Always_Anon 3y ago
      [dead]
chrismorgan 3y ago
My previous Linux laptop with 8GB of RAM (2014–2017 or so, I think) I never even got round to setting up swap, and I only ran into problems two or three times total (when running two or three Firefox instances and Chromium and a VM taking 2GB and a bunch of other things running—some of the biggest consumers, namely browsers, actually notice if they’re using too much of your RAM and adjust how they work so they don’t ask for so much RAM, at what I believe is a fairly slight performance cost). On my current laptop and with my current habits, I’m consuming a lot more memory, and so the technique I use to avoid OOM is simply having 40GB of RAM. (As it happens, I do actually have swap set up at present, because certain circumstances meant I wanted to hibernate it occasionally; should disable swap again now I’m back to not needing to hibernate.)
- isthatafact 3y ago
  > some of the biggest consumers, namely browsers, actually notice if they’re using too much of your RAM and adjust how they work so they don’t ask for so much RAM Maybe there are reasons, but it is not nearly good enough -- I frequently run out of RAM and encounter OOM kills (prefer not to deal with linux swap), usually requiring a reboot. I really wish that I could just set an upper limit on (e.g.) firefox RAM usage -- 8GB for example -- instead of its insistence on using all of the unused RAM minus a couple to several hundred MB, which does not leave enough room for memory usage spikes. There might even be a way to set this buried somewhere in the config parameters, but I could never find it. It must be technically possible to set some limit because otherwise the browser would not be able to maintain a somewhat consistent usage just below the total system RAM.

TheDong 3y ago

systemd-oomd [0]. You can also use facebook's oomd, but the systemd one is basically the same and slightly better. If you want, you can also write your own custom memory pressure handler using the recent PSI features: https://docs.kernel.org/accounting/psi.html https://docs.kernel.org/accounting/psi.html The other, easier, answer is to buy more ram, if you simply have more memory than you will ever use (such as having 128GiB of memory when you only regularly use 30GiB), you'll rarely run into OOM issues. [0]: https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html https://www.freedesktop.org/software/systemd/man/latest/syst...

nucleardog 3y ago
> The other, easier, answer is to buy more ram, if you simply have more memory than you will ever use (such as having 128GiB of memory when you only regularly use 30GiB), you'll rarely run into OOM issues. Wanna emphasize this mostly because it took me way too long to think of. Was dealing with memory pressure regularly and constantly getting frustrated… then dropped an entire $60 or something to throw another 32GB in the machine and never think about my memory again. Like a week later when I ran out of disk space again and was struggling to make room… same realization. Spent $75 for a 1TB NVME SSD dropped in my mailbox two days later and doubled my storage. So we’re talking $135 to double my RAM and storage and stop wasting time and brain space on this stuff.
- CableNinja 3y ago
  In the words of a past employer.. We ran out of space, and then we realized we are a storage company.

melvyn2 3y ago

For desktop use, nohang does what the name says. https://github.com/hakavlad/nohang https://github.com/hakavlad/nohang

magicalhippo 3y ago

This is one area where Windows is far superior to Linux as a desktop experience. By accident, I once wrote an infinite loop that just allocated a bunch of memory. As I ran the program my Windows system quickly became laggy and unresponsive. However, not completely. With a bit of patience I got Task Manager up and managed to kill the process and the system was back to normal within a minute. On Linux, I experience the same as you. Everything is fine until suddenly it's completely unresponsive. Most of the time I can't even manage to shut down the system cleanly and have to hard reset.

gnramires 3y ago
Indeed! I wonder why the system can't just stop giving the process memory when it's starting to become critical? (say, you're over 95% memory used) I really think a more general solution is treating memory (and CPU time as welL!) as a scarce resource, and programs should either: (1) Deal with having severely denied resources (default behavior); (2) Use a communication protocol to negotiate memory with the system. Negotiation could mean (a) The process freeing memory spontaneously when there's pressure (another process requires it) and it's inactive, (b) Requesting the system free memory from other processes when user requires. This would impose more memory management burden on developers (trying to fulfill OS requests), but in turn it would make for a far better experience when there's memory contention. Disclaimer: I have no idea how memory allocation works in detail. Maybe this is reinventing the wheel?
- willcipriano 3y ago
  > I wonder why the system can't just stop giving the process memory when it's starting to become critical If you turn swap off the allocator will error out when you are out of memory. I don't think most software handles that error but all the wiring is there to do that.
- roblabla 3y ago
  > Maybe this is reinventing the wheel? I believe something like that exists on mobile platforms for memory, at least on iOS you get a message (applicationDidReceiveMemoryWarning:) when the system is memory starved and wants your app to free memory. If you don't release enough memory and the memory pressure doesn't go down, the system will start killing apps.

Kwpolska 3y ago

Buying more RAM?

microtonal 3y ago
At the risk of this being tongue-in-cheeck: if you have a rogue process, any amount of RAM is going to be eaten up. (It sometimes happens in software development and it's nice if it doesn't take your machine in swap hell as a result.)
alexeiz 3y ago
Better yet, download more RAM.

Toorkit 3y ago

I recently installed Linux on my main desktop again. After freezing up twice in a week, I looked around. I enabled systemd-oom, but it hasn't been necessary, because I also enabled zram as a swap device. I've seen it compress multiple gb of ram into a few hundred mb and haven't frozen my desktop since.

seba_dos1 3y ago

You want to enable thrashing prevention: https://docs.kernel.org/next/admin-guide/mm/multigen_lru.html#thrashing-prevention https://docs.kernel.org/next/admin-guide/mm/multigen_lru.htm... Make sure you use zram (or zswap) as well. May also consider enabling userspace oom handler like others already suggested, but it's less important than the rest.

mgdlbp 3y ago

Enable Alt+SysRq+F and do it manually. Works in seconds every time; the kernel can take days because it's not eager enough when thrashing. Can have security implications, but the target is chosen in the typical way and screen locks should be setting a score_adj to avoid being picked.

deleted 3y ago
[deleted]

wakeupcall 3y ago

I'm using earlyoom on desktop, systemd-oomd on servers. Due to cgroupv2, systemd-oomd tends to kill entire process groups instead of a single run-away process [1]. It's perfect for servers where almost every process is part of a system service already. For desktop usage I found this behavior unacceptable. I would sometimes get my entire X session killed when trying something dumb (which is the reason I run such a daemon in the first place), or my editor when spawning a helper subprocess... earlyoom was the simplest daemon of the bunch I tried, and worked out of the box without special configuration. [1] Or at least this was the state ~1yr ago.

grumbel 3y ago

More RAM. It's the only thing that can reliably prevent it. Magic SysRq key[1] can trigger the OOM manually, which can sometimes rescue the system if you notice it soon enough. But if you run into the situation regularly under normal use, not programs running amok, more RAM is a must. If you can't upgrade the RAM, zram[2] is also worth a try. [1] https://en.wikipedia.org/wiki/Magic_SysRq_key https://en.wikipedia.org/wiki/Magic_SysRq_key [2] https://en.wikipedia.org/wiki/Zram https://en.wikipedia.org/wiki/Zram

marcodiego 3y ago

Is this still the case on a modern distro using MGLRU and systemd-oomd?

seba_dos1 3y ago
Not really, but not all distros have caught up yet.

DefineOutside 3y ago

Look into oom_kill_allocating_task. There's likely a better solution such as an earlier OOM, but killing the wrong process seems better than locking the system.

larve 3y ago

In a previous embedded job, I built a daemon watching a couple of “at risk of eating too much memory” processes that were not critical for the actual function of the machine, and killed them proactively when memory started getting low. This prevented the system from going into starvation and spending 10 seconds doing whatever each time an Ethernet packet came in. As for my pragmatic solution at home: max out the physical RAM…

mgdlbp 3y ago

Tangentially, an issue with the desktop use case is that the main process of a long-running browser is often the first to be killed, instead of one of its content processes, which it can survive losing. This is the case since 2019 as the kernel no longer prefers killing children.[0] As far as I know, unless a browser manages its oom_score_adj, which none do, a wrapper is insufficient as a workaround and a daemon would have to do so on its behalf, because the value is inherited when forking. [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bbbe48029720d2c6b6733f78d02571a281511adb https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

mattpallissard 3y ago

Wrap everything in systemd-run. I documented some of it here on my shitty site. Some of the memory limits have changed over time, notably the browsers unit https://pallissard.net/2022/06/27/limiting_application_resources/ https://pallissard.net/2022/06/27/limiting_application_resou... Edit: I like your user space oomd idea though, that would be killer

alexeiz 3y ago

I'm on openSUSE TW, and I considered systemd-oom, but the setup/configuration procedure for it is pretty elaborate. So I opted for earlyoom, which worked just a fine and required no manual configuration whatsoever. In my opinion, if your distro already integrates systemd-oom, like Fedora, it's easier to just use that. Otherwise earlyoom is the next best choice. Also it's recommended to enable zram. It's much faster than a swap partition or file. The combination of earlyoom and zram made my system much more responsive when it's running short of memory.