5 ms·

> If you can't on your particular Unix, I'd actually say that your Unix is probably not letting you get full use out of your RAM. What use is "full use" if you

by wahern 7y ago

> If you can't on your particular Unix, I'd actually say that your Unix is probably not letting you get full use out of your RAM. What use is "full use" if your system live locks!? This is similar logic to that of overcommit--more "efficient" use of RAM in most cases, with the tiny, insignificant cost of your processes randomly[1] being killed? What happens in practice is that people resort to over provisioning RAM anyhow. But even then that's no guarantee that your processes won't be OOM-killed. We're seeing precisely that in serious production environment--OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of gigabytes) of unallocated memory because it can't evict the buffer cache quickly enough--where quickly enough is defined by some magic heuristics deep in the eviction code. [1] And before you say that it's not random but can be controlled... you really have no idea of the depth of the problem. Non-strict, overcommit-dependent logic is baked deep into the Linux kernel. There are plenty of ways to wedge the kernel where it will end up shooting down the processes that are supposed to be the most protected, sometimes shooting down much of the system. In the many cases people simply reboot the server entirely rather than wait around to see what the wreckage looks like. This is 1990s Windows revisited--reboot and move along, unreliability is simply the nature of computers....

the8472 7y ago
You can disable the oom killer if you think its heuristics are misfiring. > OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of gigabytes) of unallocated memory because it can't evict the buffer cache quickly enough Is that due to dirty pages? Have you tried tweaking the dirty ratio to get dirty files flushed sooner? I believe there were also some improvements in the default behavior towards the end of the 4.x series that are supposed to result in more steady flushing.
- wahern 7y ago
  > You can disable the oom killer if you think its heuristics are misfiring. You can't completely disable the OOM killer. Some in-kernel allocations will trigger an OOM reap and kill regardless. Too many parts of Linux were written with the assumption of overcommit. Heck, as this issue shows even when there's technically free memory the OOM killer can still kick in. I had heard about these issues before but never saw that in practice--at least not often enough that I looked closely enough. I knew Google and Facebook and others have been relying on userspace OOM killers for years to improve reliability and consistency. But only after having recently joined a DevOps team at a large enterprise with large enterprise, big data customers did I begin to see the depth and breadth of the problem. Before getting into the hell-hole of helpless despair that is DevOps I primarily did systems programming. And, for the record, I always made sure to handle allocation failure in those languages where I could (C, Lua, etc) because as a software engineer it was always clear to me that, much as with security, an over reliance on policy-based mechanisms invariably resulted in the worst possible QoS. (Of course, in Linux even disabling overcommit is no guarantee that page fault won't trigger an OOM kill, but there are other techniques to improve reliability under load.) > Is that due to dirty pages? Yes, people have tweaked and turned those knobs and reduced incidence rate by maybe 20-30%. Yet nobody is sleeping better at night. Computing shouldn't be a black art. The irony is that overcommit and loose memory accounting more generally was originally intended to remove the necessity for professional system administrators to babysit a system to keep it humming along. But now we've come completely full circle. And it was entirely predictable. And before anyone says that cloud computing means you increase reliability by scaling horizontally, well "cloud scale" systems is exactly what are being run. But unless you massively over provision, and especially when certain tasks, even when split up across multiple nodes, can take minutes or even hours, an OOM kill incident can reverberate and cascade. Again, ironic because the whole idea of loose memory accounting is supposedly to obviate the need to over provision.
eridius 7y ago
> What use is "full use" if your system live locks!? By "full use" it doesn't mean "get to the point of live-locking". It means if the heuristic is sufficiently conservative so as to prevent live-locking, it will also necessarily prevent usage patterns that wouldn't live-lock.
- mrob 7y ago
  If you refuse to drive without wearing a seatbelt, you're not making full use of the vehicle. The conservative heuristic "always wear a seatbelt" will protect you in a crash, but it will also necessarily inconvenience you even in usage patterns where you wouldn't crash. Safety and reliability improvements are very rarely free. It's my opinion, and I assume wahern's, that Linux should be reliable by default, even if this reduces efficiency in the non-failure case.
  - dtech 7y ago
    A seat-belt is a really bad comparison because there's almost zero cost to enabling it. A better comparison is a car that will only go to 100 km/h on a 120 km/h road. It's safer, less accident prone and more energy efficient and you will never need to brake because of hitting the speed limit, but you're not making full use of the car.
    - AstralStorm 7y ago
      Yes, because something like an engineering margin or extra capacity for unpredictable situations is a bad thing. /s People do not learn. No safety margin means things will fail so you need to expect failure. Even if you do expect failure, there is a chance of unexpected kind of failure. Failures tend to be much more costly than the extra capacity. And that's in predictable systems, not something as unpredictable as say an ssh server.
    - brazzy 7y ago
      > A seat-belt is a really bad comparison because there's almost zero cost to enabling it. Watch some documentaries about the time wearing seatbelts became mandatory and listen to people bitch how inconvenient, uncomfortable and dangerous (what if the car catches fire and the seatbelt gets stuck???) they are...
toast0 7y ago
> This is 1990s Windows revisited--reboot and move along, unreliability is simply the nature of computers.... Ugh, so much this. Where I work (until the 16th) we've moved from an environment of stability, where problems are investigated and fixed and stay fixed, to our acquirer's environment where things break randomly at all times, and it's not worth investigating system problems because nothing will stay fixed anyway. #totallynotbitter
- dylan604 7y ago
  This is that part of being professional where you want to just keep working until your last day, or come in with your feet kicked up and do nothing with the "what are they going to do, fire me?" attitude. If there's severance, I'd go with the former. If everything is settled and it's just waiting out your time, try the latter?
  - halfmatthalfcat 7y ago
    If you’re trying to be professional, the latter is never an option.
- AtlasBarfed 7y ago
  Or we kick the can down the road a different way: XPoint type stuff (assuming prices come down) so that swap is fast enough and big enough again. Although XPoint has been a disappointment so far.
nh2 7y ago
My hunch regarding overcommit is that Linux should sort out this situation, making disabled-overcommit a first-class scenario. We (application developers) will follow and adjust our programs to correctly handle malloc() failure -- after all it's quite easy to fix that even in existing applications. One thing that's needed is efficient ways to ask for, and release, memory from the OS. I feel like Linux isn't doing so well on that front. For example, Haskell (GHC) switched from calling brk() to allocating 1 TB virtual memory upfront using mmap(), so that allocations can be done without system calls, and deallocations can be done cheaply (using madvise(MADV_FREE)). Of course this approach makes it impossible to observe memory pressure. Many GNOME applications do similar large memory mappings, apparently for security reasons (Gigacage). It seems to me that these programs have to be given some good, fast APIs so that we can go into a world without overcommit.
- wmf 7y ago
  Also, fork() essentially requires overcommit but many people resist spawn() because it isn't "minimal" or whatever.
- koala_man 7y ago
  >We (application developers) will follow and adjust our programs to correctly handle malloc() failure -- after all it's quite easy to fix that even in existing applications. Can you elaborate on why this is easy? It seems really difficult to me. Wouldn't you need to add several checks to even trivial code like `string greeting = "Hello "; greeting += name;`, because you need to allocate space for a string, allocate stack space to call a constructor, allocate stack space to call an append function, allocate space for the new string? Even Erlang with its memory safety and its crash&restart philosophy kills the entire VM when running out of memory. >Haskell (GHC) switched from calling brk() to allocating 1 TB virtual memory upfront The choice of 1TB was a clever one. Noobs frequently confuse VM for RAM, so this improbably large value has probably prevented a lot of outraged posts about Haskell's memory usage.
  - toast0 7y ago
    > Even Erlang with its memory safety and its crash&restart philosophy kills the entire VM when running out of memory. That's the correct thing to do for almost all failed allocations. Crash, with a nice crashdump if possible/configured.
bscphil 7y ago
Well, Linux isn't supposed to live lock under memory pressure. It's supposed to OOM kill the process that's causing the most problems. The bug (discussed in the mailing list thread the post links to) is that the Linux kernel isn't properly detecting live lock conditions and activating OOM-killer. The reason seems to be that for SSD users paging out file-backed memory happens quickly enough that the kernel still detects it as activity and doesn't know the system has locked up. Arguably we currently have three options to prevent livelocking (short of fixing the kernel). All three have significant cons. 1. Have a big swap partition. Historically people have recommended not to have swap at all because (arguably) the kernel was too swappy and would swap out stuff that was needed. And also because some people prefer misbehaving processes to get OOM-killed instead of slowing the system to a crawl on old HDDs. I was in the latter camp, but I'm now experiencing this bug too (nothing gets OOM-killed, the system locks up instead) and considering reenabling swap. 2. Use the new memory pressure information the kernel provides with a userspace tool like earlyoom to kill misbehaving processes before the pressure is significant enough to slow the system. I tried this one out, but earlyoom repeatedly killed the X server under low memory conditions. On a desktop system, X is likely the parent process of everything you want to run, but this still might be a bug? 3. Disabling overcommit entirely. This supposedly breaks a bunch of userspace programs, and even assumptions made in the kernel itself. Worse still, it still results in processes probably killing themselves anyway when memory requests fail, but it doesn't have the advantage (on a correctly working Linux system) that the worst behaving process on the system gets killed instead of whoever happened to request too much memory.
- hrez 7y ago
  Just do sysctl vm.panic_on_oom=1 Because what are you going to OOM-kill, some non-important process? Why would you run non-important processes in the first place :) And random OOM-killing is plain crazy. Just reboot and start anew.
  - segfaultbuserr 7y ago
    > And random OOM-killing is plain crazy. Just reboot and start anew. Are you arguing that killing 100% of the running programs and losing all the data is superior to kill <100% of the programs? > Because what are you going to OOM-kill, some non-important process? Why would you run non-important processes in the first place. Because I misclicked my mouse. It's very common to create a disaster by mistakes, for example, accidentally open a 10 GiB file in a web browser or a text editor. In this case, the OOM algorithm will always kill Firefox without affecting others, and it's preciously what I need. Killing Firefox (due to a webpage with runaway JavaScript) while keeping my unsaved text editor running is desirable, even if it's not guaranteed, but statistically much better than killing the computer. I often use Alt + Sysrq to manually activate the OOM killer, in case the kernel doesn't automatically detect memory exhaustion fast enough. And it works pretty nice for me. I found that, at the worst case scenario, nearly the entire desktop will be killed (rarely occurs, often it simply kills the offending process), but I can still restart my desktop in a minute, instead of spending five minutes to reboot the hardware.

by wahern 7y ago

the8472 7y ago

You can disable the oom killer if you think its heuristics are misfiring. > OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of gigabytes) of unallocated memory because it can't evict the buffer cache quickly enough Is that due to dirty pages? Have you tried tweaking the dirty ratio to get dirty files flushed sooner? I believe there were also some improvements in the default behavior towards the end of the 4.x series that are supposed to result in more steady flushing.

wahern 7y ago
> You can disable the oom killer if you think its heuristics are misfiring. You can't completely disable the OOM killer. Some in-kernel allocations will trigger an OOM reap and kill regardless. Too many parts of Linux were written with the assumption of overcommit. Heck, as this issue shows even when there's technically free memory the OOM killer can still kick in. I had heard about these issues before but never saw that in practice--at least not often enough that I looked closely enough. I knew Google and Facebook and others have been relying on userspace OOM killers for years to improve reliability and consistency. But only after having recently joined a DevOps team at a large enterprise with large enterprise, big data customers did I begin to see the depth and breadth of the problem. Before getting into the hell-hole of helpless despair that is DevOps I primarily did systems programming. And, for the record, I always made sure to handle allocation failure in those languages where I could (C, Lua, etc) because as a software engineer it was always clear to me that, much as with security, an over reliance on policy-based mechanisms invariably resulted in the worst possible QoS. (Of course, in Linux even disabling overcommit is no guarantee that page fault won't trigger an OOM kill, but there are other techniques to improve reliability under load.) > Is that due to dirty pages? Yes, people have tweaked and turned those knobs and reduced incidence rate by maybe 20-30%. Yet nobody is sleeping better at night. Computing shouldn't be a black art. The irony is that overcommit and loose memory accounting more generally was originally intended to remove the necessity for professional system administrators to babysit a system to keep it humming along. But now we've come completely full circle. And it was entirely predictable. And before anyone says that cloud computing means you increase reliability by scaling horizontally, well "cloud scale" systems is exactly what are being run. But unless you massively over provision, and especially when certain tasks, even when split up across multiple nodes, can take minutes or even hours, an OOM kill incident can reverberate and cascade. Again, ironic because the whole idea of loose memory accounting is supposedly to obviate the need to over provision.

eridius 7y ago

> What use is "full use" if your system live locks!? By "full use" it doesn't mean "get to the point of live-locking". It means if the heuristic is sufficiently conservative so as to prevent live-locking, it will also necessarily prevent usage patterns that wouldn't live-lock.

mrob 7y ago
If you refuse to drive without wearing a seatbelt, you're not making full use of the vehicle. The conservative heuristic "always wear a seatbelt" will protect you in a crash, but it will also necessarily inconvenience you even in usage patterns where you wouldn't crash. Safety and reliability improvements are very rarely free. It's my opinion, and I assume wahern's, that Linux should be reliable by default, even if this reduces efficiency in the non-failure case.
- dtech 7y ago
  A seat-belt is a really bad comparison because there's almost zero cost to enabling it. A better comparison is a car that will only go to 100 km/h on a 120 km/h road. It's safer, less accident prone and more energy efficient and you will never need to brake because of hitting the speed limit, but you're not making full use of the car.
  - AstralStorm 7y ago
    Yes, because something like an engineering margin or extra capacity for unpredictable situations is a bad thing. /s People do not learn. No safety margin means things will fail so you need to expect failure. Even if you do expect failure, there is a chance of unexpected kind of failure. Failures tend to be much more costly than the extra capacity. And that's in predictable systems, not something as unpredictable as say an ssh server.
  - brazzy 7y ago
    > A seat-belt is a really bad comparison because there's almost zero cost to enabling it. Watch some documentaries about the time wearing seatbelts became mandatory and listen to people bitch how inconvenient, uncomfortable and dangerous (what if the car catches fire and the seatbelt gets stuck???) they are...

toast0 7y ago

> This is 1990s Windows revisited--reboot and move along, unreliability is simply the nature of computers.... Ugh, so much this. Where I work (until the 16th) we've moved from an environment of stability, where problems are investigated and fixed and stay fixed, to our acquirer's environment where things break randomly at all times, and it's not worth investigating system problems because nothing will stay fixed anyway. #totallynotbitter

dylan604 7y ago
This is that part of being professional where you want to just keep working until your last day, or come in with your feet kicked up and do nothing with the "what are they going to do, fire me?" attitude. If there's severance, I'd go with the former. If everything is settled and it's just waiting out your time, try the latter?
- halfmatthalfcat 7y ago
  If you’re trying to be professional, the latter is never an option.
AtlasBarfed 7y ago
Or we kick the can down the road a different way: XPoint type stuff (assuming prices come down) so that swap is fast enough and big enough again. Although XPoint has been a disappointment so far.

nh2 7y ago

My hunch regarding overcommit is that Linux should sort out this situation, making disabled-overcommit a first-class scenario. We (application developers) will follow and adjust our programs to correctly handle malloc() failure -- after all it's quite easy to fix that even in existing applications. One thing that's needed is efficient ways to ask for, and release, memory from the OS. I feel like Linux isn't doing so well on that front. For example, Haskell (GHC) switched from calling brk() to allocating 1 TB virtual memory upfront using mmap(), so that allocations can be done without system calls, and deallocations can be done cheaply (using madvise(MADV_FREE)). Of course this approach makes it impossible to observe memory pressure. Many GNOME applications do similar large memory mappings, apparently for security reasons (Gigacage). It seems to me that these programs have to be given some good, fast APIs so that we can go into a world without overcommit.

wmf 7y ago
Also, fork() essentially requires overcommit but many people resist spawn() because it isn't "minimal" or whatever.
koala_man 7y ago
>We (application developers) will follow and adjust our programs to correctly handle malloc() failure -- after all it's quite easy to fix that even in existing applications. Can you elaborate on why this is easy? It seems really difficult to me. Wouldn't you need to add several checks to even trivial code like `string greeting = "Hello "; greeting += name;`, because you need to allocate space for a string, allocate stack space to call a constructor, allocate stack space to call an append function, allocate space for the new string? Even Erlang with its memory safety and its crash&restart philosophy kills the entire VM when running out of memory. >Haskell (GHC) switched from calling brk() to allocating 1 TB virtual memory upfront The choice of 1TB was a clever one. Noobs frequently confuse VM for RAM, so this improbably large value has probably prevented a lot of outraged posts about Haskell's memory usage.
- toast0 7y ago
  > Even Erlang with its memory safety and its crash&restart philosophy kills the entire VM when running out of memory. That's the correct thing to do for almost all failed allocations. Crash, with a nice crashdump if possible/configured.

bscphil 7y ago

Well, Linux isn't supposed to live lock under memory pressure. It's supposed to OOM kill the process that's causing the most problems. The bug (discussed in the mailing list thread the post links to) is that the Linux kernel isn't properly detecting live lock conditions and activating OOM-killer. The reason seems to be that for SSD users paging out file-backed memory happens quickly enough that the kernel still detects it as activity and doesn't know the system has locked up. Arguably we currently have three options to prevent livelocking (short of fixing the kernel). All three have significant cons. 1. Have a big swap partition. Historically people have recommended not to have swap at all because (arguably) the kernel was too swappy and would swap out stuff that was needed. And also because some people prefer misbehaving processes to get OOM-killed instead of slowing the system to a crawl on old HDDs. I was in the latter camp, but I'm now experiencing this bug too (nothing gets OOM-killed, the system locks up instead) and considering reenabling swap. 2. Use the new memory pressure information the kernel provides with a userspace tool like earlyoom to kill misbehaving processes before the pressure is significant enough to slow the system. I tried this one out, but earlyoom repeatedly killed the X server under low memory conditions. On a desktop system, X is likely the parent process of everything you want to run, but this still might be a bug? 3. Disabling overcommit entirely. This supposedly breaks a bunch of userspace programs, and even assumptions made in the kernel itself. Worse still, it still results in processes probably killing themselves anyway when memory requests fail, but it doesn't have the advantage (on a correctly working Linux system) that the worst behaving process on the system gets killed instead of whoever happened to request too much memory.

hrez 7y ago
Just do sysctl vm.panic_on_oom=1 Because what are you going to OOM-kill, some non-important process? Why would you run non-important processes in the first place :) And random OOM-killing is plain crazy. Just reboot and start anew.
- segfaultbuserr 7y ago
  > And random OOM-killing is plain crazy. Just reboot and start anew. Are you arguing that killing 100% of the running programs and losing all the data is superior to kill <100% of the programs? > Because what are you going to OOM-kill, some non-important process? Why would you run non-important processes in the first place. Because I misclicked my mouse. It's very common to create a disaster by mistakes, for example, accidentally open a 10 GiB file in a web browser or a text editor. In this case, the OOM algorithm will always kill Firefox without affecting others, and it's preciously what I need. Killing Firefox (due to a webpage with runaway JavaScript) while keeping my unsaved text editor running is desirable, even if it's not guaranteed, but statistically much better than killing the computer. I often use Alt + Sysrq to manually activate the OOM killer, in case the kernel doesn't automatically detect memory exhaustion fast enough. And it works pretty nice for me. I found that, at the worst case scenario, nearly the entire desktop will be killed (rarely occurs, often it simply kills the offending process), but I can still restart my desktop in a minute, instead of spending five minutes to reboot the hardware.