5 ms·

It's funny to read this. My system did something similar to me today, since I run a python script that sacrifices (kills) memory hogs that end up contributing t

It's funny to read this. My system did something similar to me today, since I run a python script that sacrifices (kills) memory hogs that end up contributing to total system lock up in Linux: https://askubuntu.com/a/1018733 https://askubuntu.com/a/1018733 So there I was, working on an illustration in Inkscape, zoomed in a bunch, selected a path, and after a brief lockup, no mouse cursor movement even available, boom. Inkscape was auto-killed and the script sent a notifier that it had intervened in this way for the greater good. Mouse cursor moveable again. And OK, so I was running up against the limit for sure, tons of memory use. There is supposedly built-in system tooling that handles this in Linux, but it doesn't. In the meantime I'm happy for my little script I found online, because the whole point is to break just one or two things and hand some control back to me, instead of breaking _everything_. https://askubuntu.com/questions/1258371/oom-killer-never-runs https://askubuntu.com/questions/1258371/oom-killer-never-run...

yjftsjthsd-h 5y ago
In case you haven't already seen it, there's a relatively widely packaged program called earlyoom to do this. Especially helpful because it lets you tell it what to try and kill first, and what to try and spare.
- themodelplumber 5y ago
  Oh that's cool. Yeah I'd much rather it kill Firefox or all my VLC windows than my artwork. Thank you.
  - yjftsjthsd-h 5y ago
    Happy to help:) Yeah, I set it to kill Firefox since that's "safe" and that's usually the memory hog in the first place:)
charcircuit 5y ago
Linux's OOM killer always seems to run too late (once it's already locked up meaning it takes like half an hour for it to free up the memory. It's been like this for years, so I hope someone eventually fixes it.
- themodelplumber 5y ago
  For sure, it seems like a really strange problem. However I've noticed it's not consistently an issue across various Linux systems I run here. So I wonder if this variance due to different configurations (or whatever it is) has led to a lack of a louder "please fix ASAP" sentiment.
  - charcircuit 5y ago
    I wonder if they could implemented some sort of logging where people could submit cases where it's failed for them.
- _3u10 5y ago
  Add some swap to your system, drastically reduces OOM killer frequency.
  - charcircuit 5y ago
    I'm sure this would help, but I have 64GB of RAM. Linux should be able to figure out how to save some of that massive amount of memory so that it can save itself.
- rnhmjoj 5y ago
  When the system runs out of memory the OOM killer is actually invoked right away. The problem is that systems with modern fast storage can run for hours with little to no memory left without really going OOM. If you're interested in this, I've explained how this happens in this thread: https://news.ycombinator.com/item?id=28771003 https://news.ycombinator.com/item?id=28771003
  - charcircuit 5y ago
    I understand this, but I want to point out while it's locked up thrashing you can't use sysrq to invoke the OOM so you have to wait anyways. By too late, I meant that manually invoking (sysrq) the OOM before it starts thrashing prevents it from totally locking up.
NavinF 5y ago
Oh that’s a known problem. There are many kernel patches floating around that fix it by triggering the OOM killer when the system is thrashing. Here’s one based on ChromeOS’s solution: https://github.com/hakavlad/le9-patch https://github.com/hakavlad/le9-patch I’ve never had this specific problem back when I was daily driving desktop Linux, but I did run into 1000 other ones that needed bandaid solutions just like this. It’s death of a thousand cuts: https://news.ycombinator.com/item?id=28490753 https://news.ycombinator.com/item?id=28490753 I see that nothing’s changed in the last couple of years.
- themodelplumber 5y ago
  Just curious, do you believe there's an OS out there right now without this thousand cuts problem?
  - NavinF 5y ago
    Oh yes, 4 of them. Windows, macos, android, and ios only have a handful of cuts in my experience. I'm not saying that's entirely linux's fault or anything. It's just that desktop linux has a market share of around 1% (depending on your source) and even that's fragmented across several distros. Hardly anyone tests their software or hardware on linux, let alone the distro you happen to use. Some of the problems I listed used to be an issue on other OSs. For example Android used to have ridiculous audio latency on the order of 400ms, but a few years ago Google got their shit together and fixed that: https://source.android.com/devices/audio/latency/measurements#examples https://source.android.com/devices/audio/latency/measurement... The same can't be said for desktop linux. I don't think any of the distros even have a device farm for end-to-end latency measurements. The other 4 OSs have extensive testing and the devs care about user metrics.

yjftsjthsd-h 5y ago

In case you haven't already seen it, there's a relatively widely packaged program called earlyoom to do this. Especially helpful because it lets you tell it what to try and kill first, and what to try and spare.

themodelplumber 5y ago
Oh that's cool. Yeah I'd much rather it kill Firefox or all my VLC windows than my artwork. Thank you.
- yjftsjthsd-h 5y ago
  Happy to help:) Yeah, I set it to kill Firefox since that's "safe" and that's usually the memory hog in the first place:)

charcircuit 5y ago

Linux's OOM killer always seems to run too late (once it's already locked up meaning it takes like half an hour for it to free up the memory. It's been like this for years, so I hope someone eventually fixes it.

themodelplumber 5y ago
For sure, it seems like a really strange problem. However I've noticed it's not consistently an issue across various Linux systems I run here. So I wonder if this variance due to different configurations (or whatever it is) has led to a lack of a louder "please fix ASAP" sentiment.
- charcircuit 5y ago
  I wonder if they could implemented some sort of logging where people could submit cases where it's failed for them.
_3u10 5y ago
Add some swap to your system, drastically reduces OOM killer frequency.
- charcircuit 5y ago
  I'm sure this would help, but I have 64GB of RAM. Linux should be able to figure out how to save some of that massive amount of memory so that it can save itself.
rnhmjoj 5y ago
When the system runs out of memory the OOM killer is actually invoked right away. The problem is that systems with modern fast storage can run for hours with little to no memory left without really going OOM. If you're interested in this, I've explained how this happens in this thread: https://news.ycombinator.com/item?id=28771003 https://news.ycombinator.com/item?id=28771003
- charcircuit 5y ago
  I understand this, but I want to point out while it's locked up thrashing you can't use sysrq to invoke the OOM so you have to wait anyways. By too late, I meant that manually invoking (sysrq) the OOM before it starts thrashing prevents it from totally locking up.

NavinF 5y ago

Oh that’s a known problem. There are many kernel patches floating around that fix it by triggering the OOM killer when the system is thrashing. Here’s one based on ChromeOS’s solution: https://github.com/hakavlad/le9-patch https://github.com/hakavlad/le9-patch I’ve never had this specific problem back when I was daily driving desktop Linux, but I did run into 1000 other ones that needed bandaid solutions just like this. It’s death of a thousand cuts: https://news.ycombinator.com/item?id=28490753 https://news.ycombinator.com/item?id=28490753 I see that nothing’s changed in the last couple of years.

themodelplumber 5y ago
Just curious, do you believe there's an OS out there right now without this thousand cuts problem?
- NavinF 5y ago
  Oh yes, 4 of them. Windows, macos, android, and ios only have a handful of cuts in my experience. I'm not saying that's entirely linux's fault or anything. It's just that desktop linux has a market share of around 1% (depending on your source) and even that's fragmented across several distros. Hardly anyone tests their software or hardware on linux, let alone the distro you happen to use. Some of the problems I listed used to be an issue on other OSs. For example Android used to have ridiculous audio latency on the order of 400ms, but a few years ago Google got their shit together and fixed that: https://source.android.com/devices/audio/latency/measurements#examples https://source.android.com/devices/audio/latency/measurement... The same can't be said for desktop linux. I don't think any of the distros even have a device farm for end-to-end latency measurements. The other 4 OSs have extensive testing and the devs care about user metrics.