6 ms·

Vm.overcommit_memory=2 is the right setting for servers

109 points by signa11 11mo ago

renehsz 11mo ago
Strongly agree with this article. It highlights really well why overcommit is so harmful. Memory overcommit means that once you run out of physical memory, the OOM killer will forcefully terminate your processes with no way to handle the error. This is fundamentally incompatible with the goal of writing robust and stable software which should handle out-of-memory situations gracefully. But it feels like a lost cause these days... So much software breaks once you turn off overcommit, even in situations where you're nowhere close to running out of physical memory. What's not helping the situation is the fact that the kernel has no good page allocation API that differentiates between reserving and committing memory. Large virtual memory buffers that aren't fully committed can be very useful in certain situations. But it should be something a program has to ask for, not the default behavior.
- hparadiz 10mo ago
  That's a normal failure state that happens occasionally. Out of memory errors come up all the time when writing robust async job queues. There are a lot of other reasons a failure could happen but running out of memory is just one of them. Sure I can force the system to use swap but that would degrade performance for everything else so it's better to let it die and log the result and check your dead letter queue after.
- 201984 10mo ago
  > What's not helping the situation is the fact that the kernel has no good page allocation API that differentiates between reserving and committing memory. mmap with PROT_NONE is such a reservation and doesn't count towards the commit limit. A later mmap with MAP_FIXED and PROT_READ | PROT_WRITE can commit parts of the reserved region, and mmap calls with PROT_NONE and MAP_FIXED will decommit.
- barchar 10mo ago
  Even besides the aforementioned fork problems not having overcommit doesn't mean you can handle oom correctly by just handling errors from malloc!
- PunchyHamster 10mo ago
  It's not harmful. It's necessary for modern systems that are not "an ECU in a car" > Memory overcommit means that once you run out of physical memory, the OOM killer will forcefully terminate your processes with no way to handle the error. This is fundamentally incompatible with the goal of writing robust and stable software which should handle out-of-memory situations gracefully. The big software is not written that way. In fact, writing software that way means you will have to sacrifice performance, memory usage, or both because you either * need to allocate exactly what you always need and free it when it gets smaller (if you want to keep memory footprint similar)m and that will add latency * over-allocate, and waste RAM And you'd end up with MORE memory related issues, not less. Writing app where every allocation can fail is just nightmarish waste of time for 99% of the apps that are not "onboard computer of a space ship/plane"
- charcircuit 10mo ago
  >terminate your processes with no way to handle the error. This is fundamentally incompatible with the goal of writing robust and stable software Having an assumption that your process will never crash is not safe. There will always be freak things like CPUs taking the wrong branch or bits randomly flipping. Parting of design a robust system is being tolerant to things like this. Another point also mentioned is this thread is that by the time you run out of memory the system already is going to be in a bad state and now you probably don't have enough memory to even get out of it. Memory should have been freed already by telling programs to lighten up on their memory usage or by killing them and reclaiming the resources.
LordGrey 11mo ago
For anyone not familiar with the meaning of '2' in this context: The Linux kernel supports the following overcommit handling modes 0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default. 1 - Always overcommit. Appropriate for some scientific applications. Classic example is code using sparse arrays and just relying on the virtual memory consisting almost entirely of zero pages. 2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate. Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.
- dbdr 10mo ago
  > exceed swap + a configurable amount (default is 50%) of physical RAM Naive question: why is this default 50%, and more generally why is this not the entire RAM, what happens to the rest?
  - vin10 10mo ago
    it's a (then-)safe default from the age when having 1GB of RAM and 2GB of swap was the norm: https://linux-kernel.vger.kernel.narkive.com/U64kKQbW/should-calculation-of-vm-overcommit-ratio-be-changed# https://linux-kernel.vger.kernel.narkive.com/U64kKQbW/should...
    - throw0101c 10mo ago
      Probably a safe default as there's extra memory of kernel structures, file buffering, SSH sessions to allow logins to debug why your server suddenly has high load and high IOwait (swapping). If you know a a system is going to run (e.g.) a Postgres database, then tweaking the vm.* sysctl values is part of the tuning process.
wmf 11mo ago
This doesn't address the fact that forking large processes requires either overcommit or a lot of swap. That may be the source of the Redis problem.
- loeg 10mo ago
  Why? Most COWed pages will remain untouched. They only need to allocate when touched.
  - pm215 10mo ago
    Because the point of forbidding overcommit is to ensure that the only time you can discover you're out of memory is when you make a syscall that tries (explicitly or implicitly) to allocate more memory. If you don't account the COW pages to both the parent and the child process, you have a situation where you can discover the out of memory condition when the process tries to dirty the RAM and there's no page available to do that with...
    - inkyoto 10mo ago
      The described scenario (and, consequently, a concern) is mostly a philosophical question or a real concern for a very specific workload. Memory allocation is a highly non-deterministic process which highly depends on the code path, and it is generally impossible to predict how the child will handle its own memory space – it can be little or it can be more (relatively to the parent), and it is usually somewhere in the middle. Most daemons, for example, consuming next to zero extra memory after forking. The Ruby garbage collector «mark-and-sweep» (old versions of Ruby – 1.8 and 1.9) and Python reference counting (the Instagram case) bugs are prime examples of pathological cases when a child would walk over its data pages, making each dirty and causing a system collapse, but the bugs have been fixed or workarounds have been applied. Honourable mention goes to Redis in a situation when THP (transparent huge pages) are enabled. No heuristics exist out there that would turn memory allocation into a deterministic process.
  - Tuna-Fish 10mo ago
    If you have overcommit on, that happens. But if you have it off, it has to assume the worst case, otherwise there can be a failure when someone writes to a page.
jleyank 11mo ago
As I recall, this appeared in the 90’s and it was a real pain debugging then as well. Having errors deferred added a Heisenbug component to what should have been a quick, clean crash. Has malloc ever returned zero since then? Or has somebody undone this, erm, feature at times?
- baq 10mo ago
  This is exactly what the article’s title does
Animats 10mo ago
Setting 2 is still pretty generous. It means "Kernel does not allow allocations that exceed swap + (RAM × overcommit_ratio / 100)." It's not a "never swap or overcommit" setting. You can still get into thrashing by memory overload. We may be entering an era when everyone in computing has to get serious about resource consumption. NVidia says GPUs are going to get more expensive for the next five years. DRAM prices are way up, and Samsung says it's not getting better for the next few years. Bulk electricity prices are up due to all those AI data centers. We have to assume for planning purposes that computing gets a little more expensive each year through at least 2030. Somebody may make a breakthrough, but there's nothing in the fab pipeline likely to pay off before 2030, if then.
- silon42 10mo ago
  For me, on the desktop, thrashing overload is the most common way the Linux system effectively crashes... (I've left it overnight a few times, sometimes it recovered, but not always). I'm not disabling overcommit for now, but maybe I should.
  - Tuna-Fish 10mo ago
    disabling overcommit does not fix trashing. Reducing the size of your swap does.
    - silon42 10mo ago
      Yes, but not fully, it may still thrash on mmaped files (especially readonly ones).
  - PunchyHamster 10mo ago
    It's not gonna change anything. But you might get interested into software like earlyoom and similar, that basically tried to preempt oomkiller and kill something before it gets to sluggish state
  - webstrand 10mo ago
    That thrashing is probably executable pages getting evicted, and then having to be reloaded from disk when the process resumes. Even with no swap and overcommit disabled, you'll still get thrashing before the OOM killer gets triggered. I recommend everyone to enable linux's new multi-generational LRU, that can be configured to trigger the OOM when the workingset of the last N deciseconds doesn't fit in memory. And <https://github.com/hakavlad/nohang https://github.com/hakavlad/nohang> has some more suggestions.
c0l0 10mo ago
I realize this is mostly tangential to the article, but a word of warning for those who are about to mess with overcommit for the first time: In my experience, the extreme stance of "always do [thing] with overcommit" is just not defensible, because most (yes, also "server") software is just not written under the assumption that being able to deal with allocation failures in a meaningful way is a necessity. At best, there's an "malloc() or die"-like stanza in the source, and that's that. You can and maybe even should disable overcommit this way when running postgres on the server (and only a minimum of what you would these days call sidecar processes (monitoring and backup agents, etc.) on the same host/kernel), but once you have a typical zoo of stuff using dynamic languages living there, you WILL blow someone's leg off.
- kg 10mo ago
  I run my development VM with overcommit disabled and the way stuff fails when it runs out of memory is really confusing and mysterious sometimes. It's useful for flushing out issues that would otherwise cause system degradation w/overcommit enabled, so I keep it that way, but yeah... doing it in production with a bunch of different applications running is probably asking for trouble.
  - vin10 10mo ago
    > he way stuff fails when it runs out of memory is really confusing have you checked what your `vm.overcommit_ratio` is? If its < 100%, then you will get OOM kills even if plenty of RAM is free since the default is 50 i.e. 50% of RAM can be COMMITTED and no more. curious what kind of failures you are alluding to.
    - kg 10mo ago
      The main scenario that caused me a lot of grief is temporary RAM usage spikes, like a single process run during a build that uses ~8gb of RAM or more for a mere few seconds and then exits. In some cases the oom killer was reaping the wrong process or the build was just failing cryptically and if I examined stuff like top I wouldn't see any issue, plenty of free RAM. The tooling for examining this historical memory usage is pretty bad, my only option was to look at the oom killer logs and hope that eventually the culprit would show up. Thanks for the tip about vm.overcommit_ratio though, I think it's set to the default.
blibble 10mo ago
redis uses the copy-on-write property of fork() to implement saving which is elegant and completely legitimate
- ycombinatrix 10mo ago
  How does fork() work with vm.overcommit=2? A forked process would assume memory is already allocated, but I guess it would fail when writing to it as if vm.overcommit is set to 0 or 1.
  - pm215 10mo ago
    I believe (per the stuff at the bottom of https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... ) that the kernel does the accounting of how much memory the new child process needs and will fail the fork() if there isn't enough. All the COW pages should be in the "shared anonymous" category so get counted once per user (i.e. once for the parent process, once for the child), ensuring that the COW copy can't fail if the fork succeeded.
  - toast0 10mo ago
    As pm215 states, it doubles your memory commit. It's somewhat common for large programs/runtimes that may fork at runtime to spawn an intermediary process during startup to use for runtime forks to avoid the cost of CoW on memory and mapppings and etc where the CoW isn't needed or desirable; but redis has to fork the actual service process because it uses CoW to effectively snapshot memory.
    - loeg 10mo ago
      It seems like a wrong accounting to count CoWed pages twice.
      - kibwen 10mo ago
        Not if your goal is to make it such that OOM can only occur during allocation failure, and not during an arbitrary later write, as the OP purports to want.
jcalvinowens 10mo ago
There's a reason nobody does this: RAM is expensive. Disabling overcommit on your typical server workload will waste a great deal of it. TFA completely ignores this. This is one of those classic money vs idealism things. In my experience, the money always wins this particular argument: nobody is going to buy more RAM for you so you can do this.
- wmf 10mo ago
  If you have enough swap there's no waste.
  - jcalvinowens 10mo ago
    Wasted swap is still waste, and the swapping costs cycles.
    - wmf 10mo ago
      With overcommit off the swap isn't used; it's only necessary for accounting purposes. I agree that it's a waste of disk space.
- toast0 10mo ago
  Even if you disable overcommit, I don't think you will get pages assigned when you allocate. If your allocations don't trigger an allocation failure, you should get the same behavior with respect to disk cache using otherwise unused pages. The difference is that you'll fail allocations, where there's a reasonable interface for errors, rather than failing at demand paging when writing to previously unused pages where there's not a good interface. Of course, there are many software patterns where excessive allocations are made without any intent of touching most of the pages; that's fine with overcommit, but it will lead to allocation failures when you disable overcommit. Disabling overcommit does make fork in a large process tricky; I don't think the rant about redis in the article is totally on target; fork to persist is a pretty good solution, copy on write is a reasonable cost to pay while dumping the data to disk and then it returns to normal when the dump is done. But without overcommit, it doubles the memory commitment while the dump is running, and that's likely to cause issues if redis is large relative to memory and that's worth checking for and warning about. The linked jemalloc issue seems like it could be problematic too, but I only skimmed; seems like that's worth warning about as well. For the fork path, it might be nice if you could request overcommit in certain circumstances... fork but only commit X% rather than the whole memory space.
deleted 10mo ago
[deleted]
deathanatos 10mo ago
This is quite the bold statement to make with RAM prices sky high. I want to agree with the locality of errors argument, and while in simple cases, yes, it holds true, it isn't necessarily true. If we don't overcommit, the allocation that kills us is simply the one that fails. Whether this allocation is the problematic one is a different question: if we have a slow leak that, every 10k allocation allocs and leaks, we're probably (9999 / 10k, assuming spherical allocations) going to fail on one that isn't the problem. We get about as much info as the oom-killer would have, anyways: this program is allocating too much.
vin10 10mo ago
For anyone feeling brave enough to disable overcommit after reading this, be mindful that default `vm.overcommit_ratio` is 50% which means that if no swap is available, on a system with 2GB of total RAM, more than 1GB of RAM can't be allocated and requests will fail with preemptive OOMs. (e.g. postgresql servers typically disable overcommit) - https://github.com/torvalds/linux/blob/master/mm/util.c#L753 https://github.com/torvalds/linux/blob/master/mm/util.c#L753
pizlonator 10mo ago
This is such an old debate. The real answer, as with all such things, is "it depends". Two reasons why overcommit is a good idea: - It lets you reserve memory and use the dirtying of that memory to be the thing that commits it. Some algorithms and data structures rely on this strongly (i.e. you would have to use a significantly different algorithm, which is demonstrably slower or more memory intensive, if you couldn't rely on overcommit). - Many applications have no story for out-of-memory other halting. You can scream and yell at them to do better, but that won't help, because those apps that find themselves in that supposedly-bad situation ended up there for complex and well-considered reasons. My favorite: having complex OOM error handling paths is the worst kind of attack surface, since it's hard to get test coverage for it. So, it's better to just have the program killed instead, because that nixes the untested code path. For those programs, there's zero value in having the memory allocator be able to report OOM conditions other than by asserting in prod that mmap/madvise always succeed, which then means that the value of not overcommitting is much smaller. Are there server apps where the value of gracefully handling out of memory errors outweighs the perf benefits of overcommit and the attack surface mitigation of halting on OOM? Yeah! But I bet that not all server apps fall into that bucket
- PunchyHamster 10mo ago
  It's also performance, as there is no penalty for asking for more RAM than you need right now, you can reduce amount of allocation calls without sacrificing memory usage (as you would have to without overcommit)
  - pizlonator 10mo ago
    That's what I mean by my first reason
- ece 10mo ago
  Debugging what causes an OOM situation seems to be a wash with or without overcommit as an end user, I am guessing more warnings/hints would be appreciated by users and developers. I use zram exclusively as swap, and while doing memory intensive tasks in the background, it's usually something interactive like the browser that ends up being killed. I would turn on swap that's isn't zram if this happened more often. I might also turn off overcommit if apps handled OOM situations gracefully, but as you say, the complexity might not be worth it.
laurencerowe 10mo ago
Disabling overcommit on V8 servers like Deno will be incredibly inefficient. Your process might only need ~100MB of memory or so but V8's cppgc caged heap requires a 64GB allocation in order to get a 32GB aligned area in which to contain its pointers. This is a security measure to prevent any possibility of out of cage access.
- silon42 10mo ago
  Maybe it should use MAP_NORESERVE ?
  - laurencerowe 10mo ago
    I expect it does already, but I don’t think it would help here: > In mode 2 the MAP_NORESERVE flag is ignored. https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
    - silon42 10mo ago
      Hmm, we need setting 3 (more like 1.5) then.
EdiX 10mo ago
This is completely wrong. First, disabling overcommit is wasteful because of fork and because of the way thread stacks are allocated. Sorry, you don't get exact memory accounting with C, not even Windows will do exact accounting of thread stacks. Secondly, memory is a global resource so you don't get local failures when it's exhausted, whoever allocates first after memory has been exhausted will get an error they might be the application responsible for the exhaustion or they might not be. They might crash on the error or they might "handle it", keep going and render the system completely unusable. No, exact accounting is not a solution. Ulimits and configuring the OOM killer are solutions.
- otabdeveloper4 10mo ago
  > because of fork and because of the way thread stacks are allocated For modern (post-x86_64) memory allocators a common strategy is to allocate hundreds of gigabytes of virtual memory and let the kernel handle deal with actually swapping in physical memory pages upon use. This way you can partition the virtual memory space into arenas as you like. This works really well.
  - zrm 10mo ago
    Which is a major way turning off overcommit can cause problems. The expectation for disabling it is that if you request memory you're going to use it, which is frequently not true. So if you turn it off, your memory requirements go from, say, 64GB to 512GB. Obviously you don't want to have to octuple your physical memory for pages that will never be used, especially these days, so the typical way around that is to allocate a lot of swap. Then the allocations that aren't actually used can be backed by swap instead of RAM. Except then you've essentially reimplemented overcommit. Allocations report success because you have plenty of swap but if you try to really use that much the system grinds to a halt.
    - themafia 10mo ago
      > your memory requirements go from, say, 64GB to 512GB. Then your memory requirements always were potentially 512GB. It may just happen to be even with that amount of allocation you may only need 64GB of actual physical storage; however, there is clearly a path for your application to suddenly require 512GB of storage. Perhaps when it's under an attack or facing misconfigured clients. If your failure strategy is "just let the server fall over under pressure" then this might be fine for you.
charcircuit 10mo ago
>Would you rather debug a crash at the allocation site The allocation site is not necessarily what is leaking memory. What you actually want in either case is a memory dump where you can tell what is leaking or using the memory.
PunchyHamster 10mo ago
Sure if you don't like your stuff to work well. 0 is default for a reason, and "my specific workload is buggy with 0" is not a problem with it, just the reason there are other options Advertising for 2 with "but apps should handle it" is utter ignorance, and redis example shows that, the database is using the COW fork feature for basically the reason it exists, as do many, many servers and the warning is pretty much tailored for people thinking they are clever and not understanding memory subsystem
barchar 10mo ago
Fwiw you can use pressure stall information to load shed. This is superior to disabling overcommit and then praying the first allocation to fail is in the process you want to actually respond to the resource starvation. Fact is that by the time small allocations are failing you are almost no better off handling the null than you would be handling segfaults and the sigterms from the killer. Often for servers performance will fall off a cliff long before the oom killer is needed, too.
kentonv 10mo ago
When your system is out of memory, you do not want to return an error to the next process that allocates memory. That might be an important process, it might have nothing to do with the reason the system is out of memory, and it might not be able to gracefully handle allocation failure (realistically, most programs can't). Instead, you want to kill the process that's hogging all the memory. The OOM killer heuristic is not perfect, but it will generally avoid killing critical processes and is fairly good at identifying memory hogs. And if you agree that using the OOM killer is better than returning failure to a random unlucky process, then there's no reason not to use overcommit. Besides, overcommit is useful. Virtual-memory-based copy-on-write, allocate-on-write, sparse arrays, etc. are all useful and widely-used.
- coryrc 10mo ago
  OOM killer often doesn't run soon enough for me; I've even left the machine for twenty minutes and it's still swapping hard. And I do say "often" because it does sometimes work. I have set all my Firefox processes near-maximum priority to kill for the OOM killer, but it didn't help. Also don't forget about memory compression: only meaningful with overcommit.
  - rcxdude 10mo ago
    Yeah, you probably want something that activates a bit earlier. I think the issue is the OOM killer won't activate until essentially everything that can be paged out is paged out, and that includes most code pages, so the system enters a death spiral of paging code in and out and stops making progress towards a point where the OOM killer would kick in. There's userspace daemons like earlyoom that help a lot with this.
  - kentonv 10mo ago
    They say swap is highly highly recommended even if you have plenty of memory. I don't care, I disable it anyway. Have been doing so for decades. Never caused a problem.
    - coryrc 10mo ago
simscitizen 10mo ago
There's already a popular OS that disables overcommit by default (Windows). The problem with this is that disallowing overcommit (especially with software that doesn't expect that) can mean you don't get anywhere close to actually using all the RAM that's installed on your system.
- masklinn 10mo ago
  Windows also splits memory allocations between allocating the virtual space and committing real memory. So you can allocate a large VM when you need one and use piecemeal commits within that space. POSIX not so much.
- machinationu 10mo ago
  I am regularly getting close to 100% RAM usage on Windows doing data processing in Python/numpy
Skunkleton 10mo ago
Over commit is a design choice, and it is a design choice that is pretty core to Linux. Basic stuff like fork(), for example, gets wasteful when you don't over commit. Less obvious stuff like buffer caches also get less effective. There are certainly places where you would rather fail at allocation time, but that isn't everywhere and it doesn't belong as a default.
agilob 10mo ago
An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused. https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/
- self_awareness 10mo ago
  I'm wondering which overcommit strategy this example referrs to. Because If my bitcoin price checker built on electron will start allocating all memory on the machine, then (assuming no overcommitting takes place) some arbitrary process (e.g. systemd) can get malloc error. But it's not systemd's fault the memory got eaten; so why it's being punished for low memory conditions? It's like choosing a random person to be ejected from the plane.
- deleted 10mo ago
  [deleted]
- saghm 10mo ago
  I get that this is humorous, but it seems like it illustrates the point of why this strategy is useful in the first place: memory is not human life, does not feel pain, and can even be resurrected from swap (which might still take some extra time but still is way less of an issue than the corresponding problem for humans)? If the strongest objection to the system is that it can't be ethically generalized to apply to managing people instead of memory, I think I'm happy with it. I don't care about whether it's inelegant if it works well enough for me in practice when the "victims" are arbitrary values in memory.
Asmod4n 10mo ago
There are some situations where you can somewhat handle malloc returning NULL. One would be where you have frequent large mallocs which get freed fast. Another would be where you have written a garbage collected language in C/C++. When calling free, delete or letting your GC do that for you the memory isn't actually given back immediately, glibc has malloc_trim(0) for that, which tries it's best to give back as much unused memory to the OS as possible. Then you can retry your call to malloc and see if it fails and then just let your supervisor restart your service/host/whatever or not.
ris 10mo ago
This rules out some extremely useful sparse memory tricks you can pull with massive mmaps that only ever get partially accessed (in unpredictable patterns).
sevensor 10mo ago
I was just looking at an OOM situation this week. I disagree that turning overcommit off helps specifically with locality. Finding the straw that breaks the camel’s back doesn’t necessarily help you fix the problem. If you don’t understand the whole system, it’s going to be hard to debug regardless.
teeray 10mo ago
> In contrast, when overcommit is enabled, the kernel simply allocates a VMA object without guaranteeing that backing memory is available: the mapping succeeds immediately, even though it is not known whether the request can ultimately be satisfied. When overcommit is enabled, the kernel is allowed to engage in fractional reserve banking.

109 points by signa11 11mo ago

Strongly agree with this article. It highlights really well why overcommit is so harmful. Memory overcommit means that once you run out of physical memory, the OOM killer will forcefully terminate your processes with no way to handle the error. This is fundamentally incompatible with the goal of writing robust and stable software which should handle out-of-memory situations gracefully. But it feels like a lost cause these days... So much software breaks once you turn off overcommit, even in situations where you're nowhere close to running out of physical memory. What's not helping the situation is the fact that the kernel has no good page allocation API that differentiates between reserving and committing memory. Large virtual memory buffers that aren't fully committed can be very useful in certain situations. But it should be something a program has to ask for, not the default behavior.

hparadiz 10mo ago
That's a normal failure state that happens occasionally. Out of memory errors come up all the time when writing robust async job queues. There are a lot of other reasons a failure could happen but running out of memory is just one of them. Sure I can force the system to use swap but that would degrade performance for everything else so it's better to let it die and log the result and check your dead letter queue after.
201984 10mo ago
> What's not helping the situation is the fact that the kernel has no good page allocation API that differentiates between reserving and committing memory. mmap with PROT_NONE is such a reservation and doesn't count towards the commit limit. A later mmap with MAP_FIXED and PROT_READ | PROT_WRITE can commit parts of the reserved region, and mmap calls with PROT_NONE and MAP_FIXED will decommit.
barchar 10mo ago
Even besides the aforementioned fork problems not having overcommit doesn't mean you can handle oom correctly by just handling errors from malloc!
PunchyHamster 10mo ago
It's not harmful. It's necessary for modern systems that are not "an ECU in a car" > Memory overcommit means that once you run out of physical memory, the OOM killer will forcefully terminate your processes with no way to handle the error. This is fundamentally incompatible with the goal of writing robust and stable software which should handle out-of-memory situations gracefully. The big software is not written that way. In fact, writing software that way means you will have to sacrifice performance, memory usage, or both because you either * need to allocate exactly what you always need and free it when it gets smaller (if you want to keep memory footprint similar)m and that will add latency * over-allocate, and waste RAM And you'd end up with MORE memory related issues, not less. Writing app where every allocation can fail is just nightmarish waste of time for 99% of the apps that are not "onboard computer of a space ship/plane"
charcircuit 10mo ago
>terminate your processes with no way to handle the error. This is fundamentally incompatible with the goal of writing robust and stable software Having an assumption that your process will never crash is not safe. There will always be freak things like CPUs taking the wrong branch or bits randomly flipping. Parting of design a robust system is being tolerant to things like this. Another point also mentioned is this thread is that by the time you run out of memory the system already is going to be in a bad state and now you probably don't have enough memory to even get out of it. Memory should have been freed already by telling programs to lighten up on their memory usage or by killing them and reclaiming the resources.

LordGrey 11mo ago

For anyone not familiar with the meaning of '2' in this context: The Linux kernel supports the following overcommit handling modes 0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default. 1 - Always overcommit. Appropriate for some scientific applications. Classic example is code using sparse arrays and just relying on the virtual memory consisting almost entirely of zero pages. 2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate. Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.

dbdr 10mo ago
> exceed swap + a configurable amount (default is 50%) of physical RAM Naive question: why is this default 50%, and more generally why is this not the entire RAM, what happens to the rest?
- vin10 10mo ago
  it's a (then-)safe default from the age when having 1GB of RAM and 2GB of swap was the norm: https://linux-kernel.vger.kernel.narkive.com/U64kKQbW/should-calculation-of-vm-overcommit-ratio-be-changed# https://linux-kernel.vger.kernel.narkive.com/U64kKQbW/should...
  - throw0101c 10mo ago
    Probably a safe default as there's extra memory of kernel structures, file buffering, SSH sessions to allow logins to debug why your server suddenly has high load and high IOwait (swapping). If you know a a system is going to run (e.g.) a Postgres database, then tweaking the vm.* sysctl values is part of the tuning process.

wmf 11mo ago

This doesn't address the fact that forking large processes requires either overcommit or a lot of swap. That may be the source of the Redis problem.

loeg 10mo ago
Why? Most COWed pages will remain untouched. They only need to allocate when touched.
- pm215 10mo ago
  Because the point of forbidding overcommit is to ensure that the only time you can discover you're out of memory is when you make a syscall that tries (explicitly or implicitly) to allocate more memory. If you don't account the COW pages to both the parent and the child process, you have a situation where you can discover the out of memory condition when the process tries to dirty the RAM and there's no page available to do that with...
  - inkyoto 10mo ago
    The described scenario (and, consequently, a concern) is mostly a philosophical question or a real concern for a very specific workload. Memory allocation is a highly non-deterministic process which highly depends on the code path, and it is generally impossible to predict how the child will handle its own memory space – it can be little or it can be more (relatively to the parent), and it is usually somewhere in the middle. Most daemons, for example, consuming next to zero extra memory after forking. The Ruby garbage collector «mark-and-sweep» (old versions of Ruby – 1.8 and 1.9) and Python reference counting (the Instagram case) bugs are prime examples of pathological cases when a child would walk over its data pages, making each dirty and causing a system collapse, but the bugs have been fixed or workarounds have been applied. Honourable mention goes to Redis in a situation when THP (transparent huge pages) are enabled. No heuristics exist out there that would turn memory allocation into a deterministic process.
- Tuna-Fish 10mo ago
  If you have overcommit on, that happens. But if you have it off, it has to assume the worst case, otherwise there can be a failure when someone writes to a page.

jleyank 11mo ago

As I recall, this appeared in the 90’s and it was a real pain debugging then as well. Having errors deferred added a Heisenbug component to what should have been a quick, clean crash. Has malloc ever returned zero since then? Or has somebody undone this, erm, feature at times?

baq 10mo ago
This is exactly what the article’s title does

Animats 10mo ago

Setting 2 is still pretty generous. It means "Kernel does not allow allocations that exceed swap + (RAM × overcommit_ratio / 100)." It's not a "never swap or overcommit" setting. You can still get into thrashing by memory overload. We may be entering an era when everyone in computing has to get serious about resource consumption. NVidia says GPUs are going to get more expensive for the next five years. DRAM prices are way up, and Samsung says it's not getting better for the next few years. Bulk electricity prices are up due to all those AI data centers. We have to assume for planning purposes that computing gets a little more expensive each year through at least 2030. Somebody may make a breakthrough, but there's nothing in the fab pipeline likely to pay off before 2030, if then.

silon42 10mo ago
For me, on the desktop, thrashing overload is the most common way the Linux system effectively crashes... (I've left it overnight a few times, sometimes it recovered, but not always). I'm not disabling overcommit for now, but maybe I should.
- Tuna-Fish 10mo ago
  disabling overcommit does not fix trashing. Reducing the size of your swap does.
  - silon42 10mo ago
    Yes, but not fully, it may still thrash on mmaped files (especially readonly ones).
- PunchyHamster 10mo ago
  It's not gonna change anything. But you might get interested into software like earlyoom and similar, that basically tried to preempt oomkiller and kill something before it gets to sluggish state
- webstrand 10mo ago
  That thrashing is probably executable pages getting evicted, and then having to be reloaded from disk when the process resumes. Even with no swap and overcommit disabled, you'll still get thrashing before the OOM killer gets triggered. I recommend everyone to enable linux's new multi-generational LRU, that can be configured to trigger the OOM when the workingset of the last N deciseconds doesn't fit in memory. And <https://github.com/hakavlad/nohang https://github.com/hakavlad/nohang> has some more suggestions.

c0l0 10mo ago

I realize this is mostly tangential to the article, but a word of warning for those who are about to mess with overcommit for the first time: In my experience, the extreme stance of "always do [thing] with overcommit" is just not defensible, because most (yes, also "server") software is just not written under the assumption that being able to deal with allocation failures in a meaningful way is a necessity. At best, there's an "malloc() or die"-like stanza in the source, and that's that. You can and maybe even should disable overcommit this way when running postgres on the server (and only a minimum of what you would these days call sidecar processes (monitoring and backup agents, etc.) on the same host/kernel), but once you have a typical zoo of stuff using dynamic languages living there, you WILL blow someone's leg off.

kg 10mo ago
I run my development VM with overcommit disabled and the way stuff fails when it runs out of memory is really confusing and mysterious sometimes. It's useful for flushing out issues that would otherwise cause system degradation w/overcommit enabled, so I keep it that way, but yeah... doing it in production with a bunch of different applications running is probably asking for trouble.
- vin10 10mo ago
  > he way stuff fails when it runs out of memory is really confusing have you checked what your `vm.overcommit_ratio` is? If its < 100%, then you will get OOM kills even if plenty of RAM is free since the default is 50 i.e. 50% of RAM can be COMMITTED and no more. curious what kind of failures you are alluding to.
  - kg 10mo ago
    The main scenario that caused me a lot of grief is temporary RAM usage spikes, like a single process run during a build that uses ~8gb of RAM or more for a mere few seconds and then exits. In some cases the oom killer was reaping the wrong process or the build was just failing cryptically and if I examined stuff like top I wouldn't see any issue, plenty of free RAM. The tooling for examining this historical memory usage is pretty bad, my only option was to look at the oom killer logs and hope that eventually the culprit would show up. Thanks for the tip about vm.overcommit_ratio though, I think it's set to the default.

blibble 10mo ago

redis uses the copy-on-write property of fork() to implement saving which is elegant and completely legitimate

ycombinatrix 10mo ago
How does fork() work with vm.overcommit=2? A forked process would assume memory is already allocated, but I guess it would fail when writing to it as if vm.overcommit is set to 0 or 1.
- pm215 10mo ago
  I believe (per the stuff at the bottom of https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... ) that the kernel does the accounting of how much memory the new child process needs and will fail the fork() if there isn't enough. All the COW pages should be in the "shared anonymous" category so get counted once per user (i.e. once for the parent process, once for the child), ensuring that the COW copy can't fail if the fork succeeded.
- toast0 10mo ago
  As pm215 states, it doubles your memory commit. It's somewhat common for large programs/runtimes that may fork at runtime to spawn an intermediary process during startup to use for runtime forks to avoid the cost of CoW on memory and mapppings and etc where the CoW isn't needed or desirable; but redis has to fork the actual service process because it uses CoW to effectively snapshot memory.
  - loeg 10mo ago
    It seems like a wrong accounting to count CoWed pages twice.
    - kibwen 10mo ago
      Not if your goal is to make it such that OOM can only occur during allocation failure, and not during an arbitrary later write, as the OP purports to want.

jcalvinowens 10mo ago

There's a reason nobody does this: RAM is expensive. Disabling overcommit on your typical server workload will waste a great deal of it. TFA completely ignores this. This is one of those classic money vs idealism things. In my experience, the money always wins this particular argument: nobody is going to buy more RAM for you so you can do this.

wmf 10mo ago
If you have enough swap there's no waste.
- jcalvinowens 10mo ago
  Wasted swap is still waste, and the swapping costs cycles.
  - wmf 10mo ago
    With overcommit off the swap isn't used; it's only necessary for accounting purposes. I agree that it's a waste of disk space.
toast0 10mo ago
Even if you disable overcommit, I don't think you will get pages assigned when you allocate. If your allocations don't trigger an allocation failure, you should get the same behavior with respect to disk cache using otherwise unused pages. The difference is that you'll fail allocations, where there's a reasonable interface for errors, rather than failing at demand paging when writing to previously unused pages where there's not a good interface. Of course, there are many software patterns where excessive allocations are made without any intent of touching most of the pages; that's fine with overcommit, but it will lead to allocation failures when you disable overcommit. Disabling overcommit does make fork in a large process tricky; I don't think the rant about redis in the article is totally on target; fork to persist is a pretty good solution, copy on write is a reasonable cost to pay while dumping the data to disk and then it returns to normal when the dump is done. But without overcommit, it doubles the memory commitment while the dump is running, and that's likely to cause issues if redis is large relative to memory and that's worth checking for and warning about. The linked jemalloc issue seems like it could be problematic too, but I only skimmed; seems like that's worth warning about as well. For the fork path, it might be nice if you could request overcommit in certain circumstances... fork but only commit X% rather than the whole memory space.

deleted 10mo ago

[deleted]

deathanatos 10mo ago

This is quite the bold statement to make with RAM prices sky high. I want to agree with the locality of errors argument, and while in simple cases, yes, it holds true, it isn't necessarily true. If we don't overcommit, the allocation that kills us is simply the one that fails. Whether this allocation is the problematic one is a different question: if we have a slow leak that, every 10k allocation allocs and leaks, we're probably (9999 / 10k, assuming spherical allocations) going to fail on one that isn't the problem. We get about as much info as the oom-killer would have, anyways: this program is allocating too much.

vin10 10mo ago

For anyone feeling brave enough to disable overcommit after reading this, be mindful that default `vm.overcommit_ratio` is 50% which means that if no swap is available, on a system with 2GB of total RAM, more than 1GB of RAM can't be allocated and requests will fail with preemptive OOMs. (e.g. postgresql servers typically disable overcommit) - https://github.com/torvalds/linux/blob/master/mm/util.c#L753 https://github.com/torvalds/linux/blob/master/mm/util.c#L753

pizlonator 10mo ago

This is such an old debate. The real answer, as with all such things, is "it depends". Two reasons why overcommit is a good idea: - It lets you reserve memory and use the dirtying of that memory to be the thing that commits it. Some algorithms and data structures rely on this strongly (i.e. you would have to use a significantly different algorithm, which is demonstrably slower or more memory intensive, if you couldn't rely on overcommit). - Many applications have no story for out-of-memory other halting. You can scream and yell at them to do better, but that won't help, because those apps that find themselves in that supposedly-bad situation ended up there for complex and well-considered reasons. My favorite: having complex OOM error handling paths is the worst kind of attack surface, since it's hard to get test coverage for it. So, it's better to just have the program killed instead, because that nixes the untested code path. For those programs, there's zero value in having the memory allocator be able to report OOM conditions other than by asserting in prod that mmap/madvise always succeed, which then means that the value of not overcommitting is much smaller. Are there server apps where the value of gracefully handling out of memory errors outweighs the perf benefits of overcommit and the attack surface mitigation of halting on OOM? Yeah! But I bet that not all server apps fall into that bucket

PunchyHamster 10mo ago
It's also performance, as there is no penalty for asking for more RAM than you need right now, you can reduce amount of allocation calls without sacrificing memory usage (as you would have to without overcommit)
- pizlonator 10mo ago
  That's what I mean by my first reason
ece 10mo ago
Debugging what causes an OOM situation seems to be a wash with or without overcommit as an end user, I am guessing more warnings/hints would be appreciated by users and developers. I use zram exclusively as swap, and while doing memory intensive tasks in the background, it's usually something interactive like the browser that ends up being killed. I would turn on swap that's isn't zram if this happened more often. I might also turn off overcommit if apps handled OOM situations gracefully, but as you say, the complexity might not be worth it.

laurencerowe 10mo ago

Disabling overcommit on V8 servers like Deno will be incredibly inefficient. Your process might only need ~100MB of memory or so but V8's cppgc caged heap requires a 64GB allocation in order to get a 32GB aligned area in which to contain its pointers. This is a security measure to prevent any possibility of out of cage access.

silon42 10mo ago
Maybe it should use MAP_NORESERVE ?
- laurencerowe 10mo ago
  I expect it does already, but I don’t think it would help here: > In mode 2 the MAP_NORESERVE flag is ignored. https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
  - silon42 10mo ago
    Hmm, we need setting 3 (more like 1.5) then.

EdiX 10mo ago

This is completely wrong. First, disabling overcommit is wasteful because of fork and because of the way thread stacks are allocated. Sorry, you don't get exact memory accounting with C, not even Windows will do exact accounting of thread stacks. Secondly, memory is a global resource so you don't get local failures when it's exhausted, whoever allocates first after memory has been exhausted will get an error they might be the application responsible for the exhaustion or they might not be. They might crash on the error or they might "handle it", keep going and render the system completely unusable. No, exact accounting is not a solution. Ulimits and configuring the OOM killer are solutions.

otabdeveloper4 10mo ago
> because of fork and because of the way thread stacks are allocated For modern (post-x86_64) memory allocators a common strategy is to allocate hundreds of gigabytes of virtual memory and let the kernel handle deal with actually swapping in physical memory pages upon use. This way you can partition the virtual memory space into arenas as you like. This works really well.
- zrm 10mo ago
  Which is a major way turning off overcommit can cause problems. The expectation for disabling it is that if you request memory you're going to use it, which is frequently not true. So if you turn it off, your memory requirements go from, say, 64GB to 512GB. Obviously you don't want to have to octuple your physical memory for pages that will never be used, especially these days, so the typical way around that is to allocate a lot of swap. Then the allocations that aren't actually used can be backed by swap instead of RAM. Except then you've essentially reimplemented overcommit. Allocations report success because you have plenty of swap but if you try to really use that much the system grinds to a halt.
  - themafia 10mo ago
    > your memory requirements go from, say, 64GB to 512GB. Then your memory requirements always were potentially 512GB. It may just happen to be even with that amount of allocation you may only need 64GB of actual physical storage; however, there is clearly a path for your application to suddenly require 512GB of storage. Perhaps when it's under an attack or facing misconfigured clients. If your failure strategy is "just let the server fall over under pressure" then this might be fine for you.

charcircuit 10mo ago

>Would you rather debug a crash at the allocation site The allocation site is not necessarily what is leaking memory. What you actually want in either case is a memory dump where you can tell what is leaking or using the memory.

PunchyHamster 10mo ago

Sure if you don't like your stuff to work well. 0 is default for a reason, and "my specific workload is buggy with 0" is not a problem with it, just the reason there are other options Advertising for 2 with "but apps should handle it" is utter ignorance, and redis example shows that, the database is using the COW fork feature for basically the reason it exists, as do many, many servers and the warning is pretty much tailored for people thinking they are clever and not understanding memory subsystem

barchar 10mo ago

Fwiw you can use pressure stall information to load shed. This is superior to disabling overcommit and then praying the first allocation to fail is in the process you want to actually respond to the resource starvation. Fact is that by the time small allocations are failing you are almost no better off handling the null than you would be handling segfaults and the sigterms from the killer. Often for servers performance will fall off a cliff long before the oom killer is needed, too.

kentonv 10mo ago

When your system is out of memory, you do not want to return an error to the next process that allocates memory. That might be an important process, it might have nothing to do with the reason the system is out of memory, and it might not be able to gracefully handle allocation failure (realistically, most programs can't). Instead, you want to kill the process that's hogging all the memory. The OOM killer heuristic is not perfect, but it will generally avoid killing critical processes and is fairly good at identifying memory hogs. And if you agree that using the OOM killer is better than returning failure to a random unlucky process, then there's no reason not to use overcommit. Besides, overcommit is useful. Virtual-memory-based copy-on-write, allocate-on-write, sparse arrays, etc. are all useful and widely-used.

coryrc 10mo ago
OOM killer often doesn't run soon enough for me; I've even left the machine for twenty minutes and it's still swapping hard. And I do say "often" because it does sometimes work. I have set all my Firefox processes near-maximum priority to kill for the OOM killer, but it didn't help. Also don't forget about memory compression: only meaningful with overcommit.
- rcxdude 10mo ago
  Yeah, you probably want something that activates a bit earlier. I think the issue is the OOM killer won't activate until essentially everything that can be paged out is paged out, and that includes most code pages, so the system enters a death spiral of paging code in and out and stops making progress towards a point where the OOM killer would kick in. There's userspace daemons like earlyoom that help a lot with this.
- kentonv 10mo ago
  They say swap is highly highly recommended even if you have plenty of memory. I don't care, I disable it anyway. Have been doing so for decades. Never caused a problem.
  - coryrc 10mo ago

simscitizen 10mo ago

There's already a popular OS that disables overcommit by default (Windows). The problem with this is that disallowing overcommit (especially with software that doesn't expect that) can mean you don't get anywhere close to actually using all the RAM that's installed on your system.

masklinn 10mo ago
Windows also splits memory allocations between allocating the virtual space and committing real memory. So you can allocate a large VM when you need one and use piecemeal commits within that space. POSIX not so much.
machinationu 10mo ago
I am regularly getting close to 100% RAM usage on Windows doing data processing in Python/numpy

Skunkleton 10mo ago

Over commit is a design choice, and it is a design choice that is pretty core to Linux. Basic stuff like fork(), for example, gets wasteful when you don't over commit. Less obvious stuff like buffer caches also get less effective. There are certainly places where you would rather fail at allocation time, but that isn't everywhere and it doesn't belong as a default.

agilob 10mo ago

An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused. https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/

self_awareness 10mo ago
I'm wondering which overcommit strategy this example referrs to. Because If my bitcoin price checker built on electron will start allocating all memory on the machine, then (assuming no overcommitting takes place) some arbitrary process (e.g. systemd) can get malloc error. But it's not systemd's fault the memory got eaten; so why it's being punished for low memory conditions? It's like choosing a random person to be ejected from the plane.
deleted 10mo ago
[deleted]
saghm 10mo ago
I get that this is humorous, but it seems like it illustrates the point of why this strategy is useful in the first place: memory is not human life, does not feel pain, and can even be resurrected from swap (which might still take some extra time but still is way less of an issue than the corresponding problem for humans)? If the strongest objection to the system is that it can't be ethically generalized to apply to managing people instead of memory, I think I'm happy with it. I don't care about whether it's inelegant if it works well enough for me in practice when the "victims" are arbitrary values in memory.

Asmod4n 10mo ago

There are some situations where you can somewhat handle malloc returning NULL. One would be where you have frequent large mallocs which get freed fast. Another would be where you have written a garbage collected language in C/C++. When calling free, delete or letting your GC do that for you the memory isn't actually given back immediately, glibc has malloc_trim(0) for that, which tries it's best to give back as much unused memory to the OS as possible. Then you can retry your call to malloc and see if it fails and then just let your supervisor restart your service/host/whatever or not.

ris 10mo ago

This rules out some extremely useful sparse memory tricks you can pull with massive mmaps that only ever get partially accessed (in unpredictable patterns).

sevensor 10mo ago

I was just looking at an OOM situation this week. I disagree that turning overcommit off helps specifically with locality. Finding the straw that breaks the camel’s back doesn’t necessarily help you fix the problem. If you don’t understand the whole system, it’s going to be hard to debug regardless.

teeray 10mo ago

> In contrast, when overcommit is enabled, the kernel simply allocates a VMA object without guaranteeing that backing memory is available: the mapping succeeds immediately, even though it is not known whether the request can ultimately be satisfied. When overcommit is enabled, the kernel is allowed to engage in fractional reserve banking.