5 ms·

Point 3 is wrong. OOM killing is not random. Each process is given a score according to its memory usage, and the highest score is chosen by the kernel. The way

Point 3 is wrong. OOM killing is not random. Each process is given a score according to its memory usage, and the highest score is chosen by the kernel. The way to mark priority in killing is to adjust this score through /proc. All of this is documented in `man 5 proc` from `/proc/[pid]/oom_adj` to `/proc/[pid]/oom_score_adj`. http://man7.org/linux/man-pages/man5/proc.5.html http://man7.org/linux/man-pages/man5/proc.5.html

bArray 7y ago
> OOM killing is not random. Yeah I know, hence the use of "random". > The way to mark priority in killing is to adjust this score through /proc. Haven't heard about this, thanks for the heads up!
- fluffything 7y ago
  > > The way to mark priority in killing is to adjust this score through /proc. > > Haven't heard about this, thanks for the heads up! While going down that road is technically correct, it is a road full of pain. A slightly less painful strategy is to disable overcommit. That way, if memory pressure is high, and a process calls `malloc`, that call will fail if there is not enough memory, and that process will fail. If you only have a couple of processes in your system that are using most of the memory and you can control them, it is simpler to just making them resilient to these kind of errors, than to try to mess with the process score to control the OOM killer.
  - enneff 7y ago
    But any process can be trying to allocate at the time your system runs out of memory, and most applications are not authored to handle malloc failing. Process failure seems easier to work around, from what I’ve seen. Would love to hear more from someone who has contrary experience.
    - AnIdiotOnTheNet 7y ago
      At least you can then properly blame the software for doing the wrong thing and potentially patch it. The kernel should not implement global behavior that encourages improper memory allocation failure handling.
      - AgentOrange1234 7y ago
        Yes so much. The overcommit approach seems to be to say, “Most people are lazy so we shouldn’t allow anyone to brush their teeth.” Handling malloc failures isn’t rocket science.
        pcwalton 7y ago
        Incorrectly handling malloc failures has been responsible for a lot of security problems. We are all better off acknowledging that C programmers are broadly incapable of writing correct OOM handling at scale and figuring out what to do with this fact, instead of wishing that it weren't this way.
        the_why_of_y 7y ago
        Do you have an example of a nontrivial user-space project that has malloc failure handling that actually works, with tests and all? The one I'm aware of is SQLite. Then there is DBus, whose author disagrees with your assessment. https://blog.ometer.com/2008/02/04/out-of-memory-handling-d-bus-experience/ https://blog.ometer.com/2008/02/04/out-of-memory-handling-d-...
        wahern 7y ago
        Lua, various OS kernels Like any aspect of writing software, how you approach the problem effects the complexity of the final product. If someone doesn't make a habit of handling OOM, then of course their solutions are going to be messy and complex; they're going to "solve" the problem in the most direct and naive way, which is rarely the best way. For example, unless I have reason to use a specialized data structure, I use intrusive list and RB-tree implementations, which cuts down on the number of individual allocations many fold. Once I allocate something like a request context, I know that I can add it to various lists and lookup trees without having to worry about allocation failure. Most of my projects have more points where they do I/O operations than memory allocations. Should people just ignore whether their reads or writes failed, too?
  - thayne 7y ago
    Unfortunately, the fork/exec method of spawning processes doesn't work for memory heavy processes (such as say a java server) without overcommit and copy-on-write memory. Not that I think fork/exec is the best method to spawn processes, but it is the standard way in unix-like systems.
    - tawy12345 7y ago
      The rationale for disabling overcommit only really makes sense if physical and virtual memory consumption are in the same ballpark. That's true for some workloads but not generally true. There are totally legitimate use cases for processes that use significantly more virtual memory than physical memory (since virtual memory is relatively cheap, but physical memory isn't). A lot of programs are going to touch and not release all virtual memory they allocate from the kernel, but there are plenty of important counterexamples. fork()/exec() is one example (which I've been burned by personally), but there are plenty of others. Any program that uses TCMalloc and has fluctuating memory consumption will have a lot of virtual address space allocated but not backed by physical pages. Sophisticated programs like in-memory caches or databases can also safely exploit a larger virtual memory space while keeping the amount of physical memory bounded.
      - bjourne 7y ago
        Virtual memory and overcommit isn't the same thing. Virtual memory means using disk space as memory. Overcommit means that the OS allows more memory to be allocated than it can guarantee is available. It is exactly like an airline booking 1 000 passengers for a flight with 500 seats, hoping that only half of them will actually show up. I don't think overcommit is ever needed in a modern system or is even a useful feature. For example, Windows doesn't support it at all.
        wahern 7y ago
        Windows supports overcommit, it's just not the default, and it's not how typical Windows runtimes allocate memory. And the nice thing about making it opt-in is that processes which didn't ask for overcommit won't get shot down when overcommitted memory can't be faulted in.
simias 7y ago
I haven't toyed with that in a long time (probably about a decade really) but back when I did it was still very difficult to get the OOM to behave the way you wanted. IIRC the scoring is fairly complex and take child processes into account so while it's technically completely deterministic it can still be fairly tricky to anticipate how it's going to work out in practice. And I did know about `oom_adjust`. Often it worked. Sometimes it didn't. Sometimes it would work too well and not kill a process that was clearly using an abnormal amount of memory. Finding the right `oom_adjust` is an art more than a science. Overall I ended up in the camp "you shouldn't throw passengers out of the plane"[1]. The best way to have the OOM killer behave well is not to have it run at all. If I don't have enough RAM just panic and let me figure out what needs to be done. [1] https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/
- SEJeff 7y ago
  And the OOM Killer has been rewritten about twice entirely in the past decade. Christoph Lameter (one of my coworkers, who also wrote the SLUB memory allocator) wrote the very first one. Very little if any of his original code is in the current linux OOM Killer. The current approach does indeed work much better. You can entirely disable the OOM Killer for a given workload with those procfs handles.
yebyen 7y ago
When I have multi-tenant shared servers that are regularly experiencing low memory conditions, that's now one of the first things that I turn off. I set sysctl vm.oom_dump_tasks=0 and vm.oom_kill_allocating_task=1, because without those and a comfortable amount of swap, I've found the likelihood that a rogue process uses up all the memory and the system becomes completely unrecoverable without power cycling, seemingly goes way up. The way that the score is calculated, as I understand it, the processes with large memory footprints that are not particularly long-running, and especially not owned by root, are the most likely to be killed. My shared servers are running unicorn for Rails apps, so they almost all look the same under that lens. I want the process with the runaway memory usage to be killed, so that person becomes aware of the problem when they find their app is down. (There are many better ways to solve this, but in our current system design, there's nothing to tell us that a service is down, so killing some other disused service means it's down until some user comes along and is impacted.) It seems like killing the process that made the last allocation is the most likely way to get this behavior I'm looking for, where the failures are noticed right away. But I'm not convinced I've fixed anything, even if the behavior characteristics seem better, and I haven't had any servers going rogue and killing themselves lately, I am not the sysadmin, I'm pretty sure the sysadmin just comes and makes the machine over-provisioned so this doesn't happen again, which seems to be the best advice out there. And as I've learned from the discussion around this article, the fact of having a fast SSD on which paging stuff out and expiring it from the swap, all happens fast enough to look almost as memory, almost completely neuters the OOM anyway. Our whole situation is wrong, don't even get me started; I'd like to solve this with containers, where we can set a policy that says "all containers must have memory request and limit" and thanks to cgroups, this problem never comes back.
- wahern 7y ago
  > thanks to cgroups, this problem never comes back. cgroups doesn't save you from the OOM killer. In fact, we're seeing persistent OOM issues in production even when there's more than enough physical memory to satisfy all allocation requests. Similar to the issue discussed in the leading LKML thread, I/O contention and stalls when evicting pageable memory (e.g. buffer cache) can result in the OOM killer shooting down processes when an allocation request (particularly an in-kernel request, such as for socket buffers) can't be satisfied quickly enough--quickly according to some complex and opaque heuristics. The fundamental problem is that overcommit has infected everything in the kernel. The assumption of overcommit is baked into Linux in the form of various heuristics and hacks. You can't really get away from it :(

bArray 7y ago

> OOM killing is not random. Yeah I know, hence the use of "random". > The way to mark priority in killing is to adjust this score through /proc. Haven't heard about this, thanks for the heads up!

fluffything 7y ago
> > The way to mark priority in killing is to adjust this score through /proc. > > Haven't heard about this, thanks for the heads up! While going down that road is technically correct, it is a road full of pain. A slightly less painful strategy is to disable overcommit. That way, if memory pressure is high, and a process calls `malloc`, that call will fail if there is not enough memory, and that process will fail. If you only have a couple of processes in your system that are using most of the memory and you can control them, it is simpler to just making them resilient to these kind of errors, than to try to mess with the process score to control the OOM killer.
- enneff 7y ago
  But any process can be trying to allocate at the time your system runs out of memory, and most applications are not authored to handle malloc failing. Process failure seems easier to work around, from what I’ve seen. Would love to hear more from someone who has contrary experience.
  - AnIdiotOnTheNet 7y ago
    At least you can then properly blame the software for doing the wrong thing and potentially patch it. The kernel should not implement global behavior that encourages improper memory allocation failure handling.
    - AgentOrange1234 7y ago
      Yes so much. The overcommit approach seems to be to say, “Most people are lazy so we shouldn’t allow anyone to brush their teeth.” Handling malloc failures isn’t rocket science.
      - pcwalton 7y ago
        Incorrectly handling malloc failures has been responsible for a lot of security problems. We are all better off acknowledging that C programmers are broadly incapable of writing correct OOM handling at scale and figuring out what to do with this fact, instead of wishing that it weren't this way.
      - the_why_of_y 7y ago
        Do you have an example of a nontrivial user-space project that has malloc failure handling that actually works, with tests and all? The one I'm aware of is SQLite. Then there is DBus, whose author disagrees with your assessment. https://blog.ometer.com/2008/02/04/out-of-memory-handling-d-bus-experience/ https://blog.ometer.com/2008/02/04/out-of-memory-handling-d-...
        wahern 7y ago
        Lua, various OS kernels Like any aspect of writing software, how you approach the problem effects the complexity of the final product. If someone doesn't make a habit of handling OOM, then of course their solutions are going to be messy and complex; they're going to "solve" the problem in the most direct and naive way, which is rarely the best way. For example, unless I have reason to use a specialized data structure, I use intrusive list and RB-tree implementations, which cuts down on the number of individual allocations many fold. Once I allocate something like a request context, I know that I can add it to various lists and lookup trees without having to worry about allocation failure. Most of my projects have more points where they do I/O operations than memory allocations. Should people just ignore whether their reads or writes failed, too?
- thayne 7y ago
  Unfortunately, the fork/exec method of spawning processes doesn't work for memory heavy processes (such as say a java server) without overcommit and copy-on-write memory. Not that I think fork/exec is the best method to spawn processes, but it is the standard way in unix-like systems.
  - tawy12345 7y ago
    The rationale for disabling overcommit only really makes sense if physical and virtual memory consumption are in the same ballpark. That's true for some workloads but not generally true. There are totally legitimate use cases for processes that use significantly more virtual memory than physical memory (since virtual memory is relatively cheap, but physical memory isn't). A lot of programs are going to touch and not release all virtual memory they allocate from the kernel, but there are plenty of important counterexamples. fork()/exec() is one example (which I've been burned by personally), but there are plenty of others. Any program that uses TCMalloc and has fluctuating memory consumption will have a lot of virtual address space allocated but not backed by physical pages. Sophisticated programs like in-memory caches or databases can also safely exploit a larger virtual memory space while keeping the amount of physical memory bounded.
    - bjourne 7y ago
      Virtual memory and overcommit isn't the same thing. Virtual memory means using disk space as memory. Overcommit means that the OS allows more memory to be allocated than it can guarantee is available. It is exactly like an airline booking 1 000 passengers for a flight with 500 seats, hoping that only half of them will actually show up. I don't think overcommit is ever needed in a modern system or is even a useful feature. For example, Windows doesn't support it at all.
      - wahern 7y ago
        Windows supports overcommit, it's just not the default, and it's not how typical Windows runtimes allocate memory. And the nice thing about making it opt-in is that processes which didn't ask for overcommit won't get shot down when overcommitted memory can't be faulted in.

simias 7y ago

I haven't toyed with that in a long time (probably about a decade really) but back when I did it was still very difficult to get the OOM to behave the way you wanted. IIRC the scoring is fairly complex and take child processes into account so while it's technically completely deterministic it can still be fairly tricky to anticipate how it's going to work out in practice. And I did know about `oom_adjust`. Often it worked. Sometimes it didn't. Sometimes it would work too well and not kill a process that was clearly using an abnormal amount of memory. Finding the right `oom_adjust` is an art more than a science. Overall I ended up in the camp "you shouldn't throw passengers out of the plane"[1]. The best way to have the OOM killer behave well is not to have it run at all. If I don't have enough RAM just panic and let me figure out what needs to be done. [1] https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/

SEJeff 7y ago
And the OOM Killer has been rewritten about twice entirely in the past decade. Christoph Lameter (one of my coworkers, who also wrote the SLUB memory allocator) wrote the very first one. Very little if any of his original code is in the current linux OOM Killer. The current approach does indeed work much better. You can entirely disable the OOM Killer for a given workload with those procfs handles.

yebyen 7y ago

When I have multi-tenant shared servers that are regularly experiencing low memory conditions, that's now one of the first things that I turn off. I set sysctl vm.oom_dump_tasks=0 and vm.oom_kill_allocating_task=1, because without those and a comfortable amount of swap, I've found the likelihood that a rogue process uses up all the memory and the system becomes completely unrecoverable without power cycling, seemingly goes way up. The way that the score is calculated, as I understand it, the processes with large memory footprints that are not particularly long-running, and especially not owned by root, are the most likely to be killed. My shared servers are running unicorn for Rails apps, so they almost all look the same under that lens. I want the process with the runaway memory usage to be killed, so that person becomes aware of the problem when they find their app is down. (There are many better ways to solve this, but in our current system design, there's nothing to tell us that a service is down, so killing some other disused service means it's down until some user comes along and is impacted.) It seems like killing the process that made the last allocation is the most likely way to get this behavior I'm looking for, where the failures are noticed right away. But I'm not convinced I've fixed anything, even if the behavior characteristics seem better, and I haven't had any servers going rogue and killing themselves lately, I am not the sysadmin, I'm pretty sure the sysadmin just comes and makes the machine over-provisioned so this doesn't happen again, which seems to be the best advice out there. And as I've learned from the discussion around this article, the fact of having a fast SSD on which paging stuff out and expiring it from the swap, all happens fast enough to look almost as memory, almost completely neuters the OOM anyway. Our whole situation is wrong, don't even get me started; I'd like to solve this with containers, where we can set a policy that says "all containers must have memory request and limit" and thanks to cgroups, this problem never comes back.

wahern 7y ago
> thanks to cgroups, this problem never comes back. cgroups doesn't save you from the OOM killer. In fact, we're seeing persistent OOM issues in production even when there's more than enough physical memory to satisfy all allocation requests. Similar to the issue discussed in the leading LKML thread, I/O contention and stalls when evicting pageable memory (e.g. buffer cache) can result in the OOM killer shooting down processes when an allocation request (particularly an in-kernel request, such as for socket buffers) can't be satisfied quickly enough--quickly according to some complex and opaque heuristics. The fundamental problem is that overcommit has infected everything in the kernel. The assumption of overcommit is baked into Linux in the form of various heuristics and hacks. You can't really get away from it :(