4 ms·

The thing I would really like to figure out is how to prevent a Linux system from essentially livelocking when it close to runs out of memory. We've all seen i

The thing I would really like to figure out is how to prevent a Linux system from essentially livelocking when it close to runs out of memory. We've all seen it. Try to ssh in, connections get established but do not proceed. If you're lucky to have a console shell open from before, it shows gigantic load. Wish there was a way to put a few system critical processes into a container to guarantee them some resources.

nialv7 7y ago
I've heard that is problem is caused by Linux's overcommitting strategy. Basically, initial memory allocation never fails (unless you set special flags), but no memory is actually allocated on the spot. Memory is only allocated when it is accessed. And if Linux runs out of memory when a program accessed a piece of yet to be allocated memory, it will try really _really_ hard to free up memory so that memory access can success. That's what's causing the lock ups. Sounds to me this would be difficult to fix without breaking backward compatibility. In the mean time, you can probably improve your quality of life quite a bit by using something like: https://github.com/facebookincubator/oomd https://github.com/facebookincubator/oomd
- lallysingh 7y ago
  .. or swap?
  - bluedays 7y ago
    or both!?
- diegocg 7y ago
  It's more complex than that. Doing lazy allocation is not the problem, it's a common optimization. The problem comes when Linux does allow programs to (lazily) allocate a total amount of memory than is larger than the available RAM+SWAP before failing allocations. Then, when processes actually try to use that RAM, there is no physical place where to place that memory, and the only solution is to kill a process (OOM). This may certainly seem stupid at first sight. I don't remember the exact reason why Linux does this, but I remember that it was said that not doing it would imply not using all available RAM efficiently and allocations would start failing before expected or something like that. It's actually pretty easy to change this behaviour, there is a sysctl (/proc/sys/vm/overcommit_memory) that defaults to 0, but you can disable the overcommitting behaviour and even tune it. "2" does disable the entire overcommitting logic and it's what some people use to avoid memory trashing situations (but you still can get OOM in some situations IIRC) https://www.kernel.org/doc/html/latest/vm/overcommit-accounting.html https://www.kernel.org/doc/html/latest/vm/overcommit-account...
  - wmf 7y ago
    Overcommit is needed when a large process fork-execs a smaller process. If overcommit is disabled then forking a large process will fail even if it would be safe in practice. A proper implementation of spawn() could fix this but that's not the Unix way.
  - nialv7 7y ago
    Turning off overcommitting can break certain applications that relies on it. For example, the address sanitize allocates huge address spaces as shadow memory. I do not think overcommitting is the problem. I believe the problem is Linux won't allow memory accesses to fail. It would stuck in a loop trying to free up memory, eventually triggers the oom killer. It could have just let the memory access fail.
O_H_E 7y ago
Try a userspace oom. EarlyOOM has been a life saver for me, but there is a few others. AFAIK Fedora and Clear linux have begun shipping EarlyOOM by default. nohang's README have a gist about other projects. https://github.com/hakavlad/nohang https://github.com/hakavlad/nohang
- foobarian 7y ago
  Thanks for that. Sure enough this was even on HN before :) https://news.ycombinator.com/item?id=20620545 https://news.ycombinator.com/item?id=20620545
- TD-Linux 7y ago
  Fedora even enabled it by default: https://fedoraproject.org/wiki/Changes/EnableEarlyoom https://fedoraproject.org/wiki/Changes/EnableEarlyoom
pengaru 7y ago
> Wish there was a way to put a few system critical processes into a container to guarantee them some resources. There is, memcg is a thing, and it has both minimum free and maximum bounds. See the systemd.resource-control(5) man page, you can put important services in their own slice or use nspawn containers. Personally I use nspawn containers for my web browsers, and set memory limits on them. This limits the live-lock behavior you're describing to just within the container. When firefox uses up all the memory, I see thrashing in the form of mostly executable pages being constantly evicted and faulted back in from disk, but the rest of the system stays responsive while the disk is hammered by the reads and some CPU burned while firefox spins its wheels.
linuxhansl 7y ago
This happens when the system is swapping and most processes are swapped out. Then in theory each context switch would need to swap in some pages first. Not having given this much thought, I think some solutions could be: (a) Favor processes that are already swapped and keep them swapped in a bit longer before swapping in other processes. I.e. trade fairness for higher throughput. (b) Kill the worst offenders, EarlyOOM, etc, comes to mind. (c) Simply do not swap :) I realize it's not a solution... But on my systems I simply have swapping disabled.
the8472 7y ago
Besides the userspace oom killers mentioned in sibling comments make sure to run the most recent kernels. This issue has gotten some attention lately and improvements for specific workloads have trickled in over several releases and will probably continue to do so. Additionally some of the userspace solutions rely on the fairly new /proc/pressure interface.
hakavlad 7y ago
Killing the elephant in the room - the userspace low memory handler ability to gracefully handle low memory pressure https://www.reddit.com/r/linux/comments/ee6szk/killing_the_elephant_in_the_room_the_userspace/ https://www.reddit.com/r/linux/comments/ee6szk/killing_the_e... - The discussion was three months ago

nialv7 7y ago

I've heard that is problem is caused by Linux's overcommitting strategy. Basically, initial memory allocation never fails (unless you set special flags), but no memory is actually allocated on the spot. Memory is only allocated when it is accessed. And if Linux runs out of memory when a program accessed a piece of yet to be allocated memory, it will try really _really_ hard to free up memory so that memory access can success. That's what's causing the lock ups. Sounds to me this would be difficult to fix without breaking backward compatibility. In the mean time, you can probably improve your quality of life quite a bit by using something like: https://github.com/facebookincubator/oomd https://github.com/facebookincubator/oomd

lallysingh 7y ago
.. or swap?
- bluedays 7y ago
  or both!?
diegocg 7y ago
It's more complex than that. Doing lazy allocation is not the problem, it's a common optimization. The problem comes when Linux does allow programs to (lazily) allocate a total amount of memory than is larger than the available RAM+SWAP before failing allocations. Then, when processes actually try to use that RAM, there is no physical place where to place that memory, and the only solution is to kill a process (OOM). This may certainly seem stupid at first sight. I don't remember the exact reason why Linux does this, but I remember that it was said that not doing it would imply not using all available RAM efficiently and allocations would start failing before expected or something like that. It's actually pretty easy to change this behaviour, there is a sysctl (/proc/sys/vm/overcommit_memory) that defaults to 0, but you can disable the overcommitting behaviour and even tune it. "2" does disable the entire overcommitting logic and it's what some people use to avoid memory trashing situations (but you still can get OOM in some situations IIRC) https://www.kernel.org/doc/html/latest/vm/overcommit-accounting.html https://www.kernel.org/doc/html/latest/vm/overcommit-account...
- wmf 7y ago
  Overcommit is needed when a large process fork-execs a smaller process. If overcommit is disabled then forking a large process will fail even if it would be safe in practice. A proper implementation of spawn() could fix this but that's not the Unix way.
- nialv7 7y ago
  Turning off overcommitting can break certain applications that relies on it. For example, the address sanitize allocates huge address spaces as shadow memory. I do not think overcommitting is the problem. I believe the problem is Linux won't allow memory accesses to fail. It would stuck in a loop trying to free up memory, eventually triggers the oom killer. It could have just let the memory access fail.

O_H_E 7y ago

Try a userspace oom. EarlyOOM has been a life saver for me, but there is a few others. AFAIK Fedora and Clear linux have begun shipping EarlyOOM by default. nohang's README have a gist about other projects. https://github.com/hakavlad/nohang https://github.com/hakavlad/nohang

foobarian 7y ago
Thanks for that. Sure enough this was even on HN before :) https://news.ycombinator.com/item?id=20620545 https://news.ycombinator.com/item?id=20620545
TD-Linux 7y ago
Fedora even enabled it by default: https://fedoraproject.org/wiki/Changes/EnableEarlyoom https://fedoraproject.org/wiki/Changes/EnableEarlyoom

pengaru 7y ago

> Wish there was a way to put a few system critical processes into a container to guarantee them some resources. There is, memcg is a thing, and it has both minimum free and maximum bounds. See the systemd.resource-control(5) man page, you can put important services in their own slice or use nspawn containers. Personally I use nspawn containers for my web browsers, and set memory limits on them. This limits the live-lock behavior you're describing to just within the container. When firefox uses up all the memory, I see thrashing in the form of mostly executable pages being constantly evicted and faulted back in from disk, but the rest of the system stays responsive while the disk is hammered by the reads and some CPU burned while firefox spins its wheels.

linuxhansl 7y ago

This happens when the system is swapping and most processes are swapped out. Then in theory each context switch would need to swap in some pages first. Not having given this much thought, I think some solutions could be: (a) Favor processes that are already swapped and keep them swapped in a bit longer before swapping in other processes. I.e. trade fairness for higher throughput. (b) Kill the worst offenders, EarlyOOM, etc, comes to mind. (c) Simply do not swap :) I realize it's not a solution... But on my systems I simply have swapping disabled.

the8472 7y ago

Besides the userspace oom killers mentioned in sibling comments make sure to run the most recent kernels. This issue has gotten some attention lately and improvements for specific workloads have trickled in over several releases and will probably continue to do so. Additionally some of the userspace solutions rely on the fairly new /proc/pressure interface.

hakavlad 7y ago

Killing the elephant in the room - the userspace low memory handler ability to gracefully handle low memory pressure https://www.reddit.com/r/linux/comments/ee6szk/killing_the_elephant_in_the_room_the_userspace/ https://www.reddit.com/r/linux/comments/ee6szk/killing_the_e... - The discussion was three months ago