4 ms·

I'd much rather have OOMkiller trigger than have my system sit completely unresponsive for huge chunks of time, while I slowly struggle to kill something.

by throwaway2048 7y ago

dahfizz 7y ago
While this might add convenience when all you do is use a web browser, Linux is huge in the server space. Having the OS suddenly and somewhat unpredictability killing programs is absolutely not an option for most workloads.
- quazeekotl 7y ago
  I don't think having a system sit completely unresponsive to anything for a huge amount of time is exactly optimal either.
- _jal 7y ago
  Absolutely, which is why configurable knobs are things that exist. I already have to mess with the OOMkiller on my servers; I'm fine with fiddling with knobs on my desktop box too. This is Linux, after all. Wasn't one of the constant talking points in the war over the Init System That Shall Not Be Named that Linux needs to scale from embedded to big iron?
- eikenberry 7y ago
  If you work in the cloud for any length of time you learn this is not true. Every system must be able to handle processes being killed unpredictably otherwise you will have problems. Better to deal with it and plan for it than expect a system to have 100% uptime. Because you never have 100% uptime.
- TheDong 7y ago
  The oom killer already exists on servers and already can kill programs. If you want to turn off overcommit and have the system power off when it runs out of memory, the kernel allows that. Whatever knob they add will certainly be configurable, and ubuntu desktop can configure it one way while ubuntu server configures it the other, if it turns out people would prefer that. In practice, people running servers seem to want oom killers to kick in before the server barfs. One example of this is facebook's oomd [1]. I assure you, they're running that on their servers, not their web-browser-machines. [1]: https://github.com/facebookincubator/oomd https://github.com/facebookincubator/oomd
- swiley 7y ago
  With no script it’s rare for Firefox to eat all my memory (with 4GB of ram) more often It’s GCC. I really just need to be involved when that happens, going and killing things isn’t the answer. One thing I do want to try is sand boxing the browser using UML (which I’ve been using a lot of lately.) This way since it gets its own kernel I can totally limit everything.
- toast0 7y ago
  Having the system thrashing as described is worse than most alternatives, including those below for most workloads. There are probably a few workloads where thrashing is better, as maybe an operator can get in despite the thrashing and do the right thing. a) killing the malfunctioning process (this is often hard) b) killing the biggest process c) kernel panic / locking up d) killing more or less random processes until memory pressure is relieved Thrashing in a distributed environment is probably going to end up with partially failed health checks and all the nastiness that comes with flapping. If you're lucky, failing health checks will reduce load and memory usage, and get you back to a happy place; if you're realistic, you probably got into the thrashing situation because of a burst of latency or traffic that resulted in slower processing for a bit and the resulting retries killed the system.
- Dylan16807 7y ago
  Then make it very predictable. Set a minimum amount of disk cache. Now you know that the OOM killer will trigger exactly X hundred megabytes sooner, and also that your performance won't tank because running code is all paged out.
- saalaa 7y ago
  I've had that argument countless times. The last thing you want during operations (whatever your actual job description) is a server that's essentially not serving, not remotely available for troubleshooting and not crashing.
- blattimwind 7y ago
  Connection refused is way better for the average server stack than timeouts or even worse accepted connections with transfer rates of 30 seconds per byte. "Connection refused" is a huge signal flare going off instantly, timeouts and extreme slowness will by definition take longer to notice while impacting systems, even if your monitoring were perfectly set up to look for it.
  - jorangreef 7y ago
    Sure, for modern systems "transfer rates of 30 seconds per byte" would also be "a huge signal flare going off instantly" where metrics track basic software-hardware interaction (latency and throughput, not just on/off error rates). These metrics also need not necessarily by definition take longer to notice gray failures. You can detect a gray failure transfer rate without having to wait 30 seconds. Another example, but a second or two is enough to notice disk latency issues (and retry to a faster disk), which is probably in the same granularity as the window for detecting "Connection refused".

I'd much rather have OOMkiller trigger than have my system sit completely unresponsive for huge chunks of time, while I slowly struggle to kill something.

dahfizz 7y ago

While this might add convenience when all you do is use a web browser, Linux is huge in the server space. Having the OS suddenly and somewhat unpredictability killing programs is absolutely not an option for most workloads.

quazeekotl 7y ago
I don't think having a system sit completely unresponsive to anything for a huge amount of time is exactly optimal either.
_jal 7y ago
Absolutely, which is why configurable knobs are things that exist. I already have to mess with the OOMkiller on my servers; I'm fine with fiddling with knobs on my desktop box too. This is Linux, after all. Wasn't one of the constant talking points in the war over the Init System That Shall Not Be Named that Linux needs to scale from embedded to big iron?
eikenberry 7y ago
If you work in the cloud for any length of time you learn this is not true. Every system must be able to handle processes being killed unpredictably otherwise you will have problems. Better to deal with it and plan for it than expect a system to have 100% uptime. Because you never have 100% uptime.
TheDong 7y ago
The oom killer already exists on servers and already can kill programs. If you want to turn off overcommit and have the system power off when it runs out of memory, the kernel allows that. Whatever knob they add will certainly be configurable, and ubuntu desktop can configure it one way while ubuntu server configures it the other, if it turns out people would prefer that. In practice, people running servers seem to want oom killers to kick in before the server barfs. One example of this is facebook's oomd [1]. I assure you, they're running that on their servers, not their web-browser-machines. [1]: https://github.com/facebookincubator/oomd https://github.com/facebookincubator/oomd
swiley 7y ago
With no script it’s rare for Firefox to eat all my memory (with 4GB of ram) more often It’s GCC. I really just need to be involved when that happens, going and killing things isn’t the answer. One thing I do want to try is sand boxing the browser using UML (which I’ve been using a lot of lately.) This way since it gets its own kernel I can totally limit everything.
toast0 7y ago
Having the system thrashing as described is worse than most alternatives, including those below for most workloads. There are probably a few workloads where thrashing is better, as maybe an operator can get in despite the thrashing and do the right thing. a) killing the malfunctioning process (this is often hard) b) killing the biggest process c) kernel panic / locking up d) killing more or less random processes until memory pressure is relieved Thrashing in a distributed environment is probably going to end up with partially failed health checks and all the nastiness that comes with flapping. If you're lucky, failing health checks will reduce load and memory usage, and get you back to a happy place; if you're realistic, you probably got into the thrashing situation because of a burst of latency or traffic that resulted in slower processing for a bit and the resulting retries killed the system.
Dylan16807 7y ago
Then make it very predictable. Set a minimum amount of disk cache. Now you know that the OOM killer will trigger exactly X hundred megabytes sooner, and also that your performance won't tank because running code is all paged out.
saalaa 7y ago
I've had that argument countless times. The last thing you want during operations (whatever your actual job description) is a server that's essentially not serving, not remotely available for troubleshooting and not crashing.
blattimwind 7y ago
Connection refused is way better for the average server stack than timeouts or even worse accepted connections with transfer rates of 30 seconds per byte. "Connection refused" is a huge signal flare going off instantly, timeouts and extreme slowness will by definition take longer to notice while impacting systems, even if your monitoring were perfectly set up to look for it.
- jorangreef 7y ago
  Sure, for modern systems "transfer rates of 30 seconds per byte" would also be "a huge signal flare going off instantly" where metrics track basic software-hardware interaction (latency and throughput, not just on/off error rates). These metrics also need not necessarily by definition take longer to notice gray failures. You can detect a gray failure transfer rate without having to wait 30 seconds. Another example, but a second or two is enough to notice disk latency issues (and retry to a faster disk), which is probably in the same granularity as the window for detecting "Connection refused".