5 ms·

Linux Memory Overcommit (2007)

63 points by signa11 2y ago

pcwalton 2y ago
Reminder: if overcommit as a concept seems distasteful, the real ire should be directed at the Unix fork syscall, an API that will always fail on a process using over 50% of the available memory without overcommit. Ideally apps would use vfork or posix_spawn instead, but that isn't the world we live in. Overcommit is a sensible way to help apps that use a lot of memory "just work". You can always turn it off if you don't need apps using a lot of memory to be able to fork.
- senderista 2y ago
  There are also legitimate uses of overcommit for reserving huge areas of virtual memory and relying on demand paging to allocate physical memory, although strictly speaking enabling overcommit isn’t necessary if you separate the “reserve” (mmap(PROT_NONE)) and “commit” (mmap(MAP_FIXED, PROT_READ | PROT_WRITE)) stages (which is what Windows forces you to do).
- dap 2y ago
  I’m not sure how this works on Linux exactly but in principle you don’t need to support overcommit to avoid that problem. On illumos for example anonymous memory allocations come from swap, which is a virtual resource that includes both disk and memory. You do still have this problem if something is using more than half of swap, but (1) that’s much larger, and (2) you can augment it with more disk space, which won’t actually be used unless stuff actually uses all the physical memory. This has its own tradeoffs but it’s useful to keep in mind that there are other ways to do things.
  - wmf 2y ago
    If available memory means physical + swap then you're saying the same thing as pcwalton.
    - pcwalton 2y ago
      Right, available memory includes swap. (That's why I didn't say "available RAM".)
- thanatos519 2y ago
  I thought fork was copy on write nowadays.
  - layer8 2y ago
    It is, but “write” doesn’t have an error code it can return on out-of-memory. The whole crux of overcommit is that the affected process doesn’t get a chance to gracefully fall back or shutdown on OOM conditions at defined places in the code (like at invocations of malloc).
  - wmf 2y ago
    Fork is copy-on-write. The problem is that there's no way for fork() to know how much memory the child process needs. It might call exec() immediately, using very little memory. It might keep running, mostly sharing pages with the parent. It might touch every memory page, requiring a lot of memory.
- userbinator 2y ago
  That explains why Windows, which has a process model based on spawning and not forking, never had overcommit.
  - ynik 2y ago
    However Windows has stacks that commit on-demand, which is kinda similar to overcommit. It can result in programs terminating on any random function call (if that call requires allocating a new page of stack). https://learn.microsoft.com/en-us/windows/win32/procthread/thread-stack-size https://learn.microsoft.com/en-us/windows/win32/procthread/t... However this is very rare to happen in practice -- stack growth isn't very frequent in typical applications; so usually a malloc() will fail first in low-memory situations. A Windows program can avoid the risk of getting terminated on stack growth by specifying equal commit and reserve sizes for the stack. So in least in theory, it's possible to write a windows program that is reliable in low memory situations.
- jabl 2y ago
  This article explains the problems with fork pretty well: https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0...
- rcxdude 2y ago
  Also I would suggest using a Windows machine which seems to run out of memory despite hundreds of gigabytes of swap and yet still ~70% of RAM full. I don't know what's causing the bug but it affects multiple systems I use and it's a right nuisance with a nearly full disk to have >10% of it sitting there idle just in case whatever's over-allocating memory decides to actually use it (I mean, I guess I could also direct my ire at whatever broken accounting fails to actually tell me what's causing it as well, but I can be mad at two things at once).
- shoo 2y ago
  yup. i recall first being exposed to this while trying to understand why a memory-hog backend service was being oom-killed in production while attempting to spawn a resource-light subprocess via fork+exec, while there was still a good portion of free memory to use. the more you read about linux oom killer, overcommit, linux memory accounting heuristics, copy-on-write, fork+exec as a mechanism to spawn an unrelated process, the more surprising it is that anything works at all even some of the time.
__turbobrew__ 2y ago
If you want to optimize for throughout one option is to overcommit resources (CPU/MEM/IO/NET) but utilize backpressure mechanisms to reduce load during times of saturation. Kubernetes does this through node pressure eviction but it is pretty easy to hook into the pressure stall information and have the application handle this as well (for example, start returning 429 HTTP responses when PSI goes over a certain level). At the end of the day the optimal overcommit is workload dependent — good metrics and an iterative approach is needed.
- eru 2y ago
  You probably also want to do somewhat intelligent load shedding, where you basically make sure you drop the most expendable items first. That's a basic fact of life for CPU cycles even for consumers on their machines, we call that just CPU priorities or so. But it's also useful for the other resources; but there it's a bit more disruptive, because we generally need to disrupt a program to eg claim back the disk space it uses.
johnea 2y ago
Is any of this still applicable in 2024? One has to doubt...
- graymatters 2y ago
  Doubt - based on what?
  - worthless-trash 2y ago
    Knowledge, GP probably thinks that the kernel world moves like the userspace (probably js) world. Which clearly isnt true. If OS development worked like userspace/web dev we'd all be in much more pain.
nabla9 2y ago
As then as now, you can manage it. vm.overcommit_memory == 0 heuristic overcommit vm.overcommit_memory == 1 (full overcommit) allows allocating more memory than there is ram + swap. vm.overcommit_memory == 2 never overcommit. There must be enough physical ram or virtual swap to allocate the memory.
- layer8 2y ago
  Can this be set for a specific process (or process group/tree), or only globally?
  - nabla9 2y ago
    Globally For a process you can use setrlimit() (ulimit command)
    - silon42 2y ago
      I'd want the OOM killer to kill processes with more over-commit first.
      - jart 2y ago
        Why? There are many innocent applications that consume lots of virtual memory with no intention of using it. Sometimes sparse algorithms call for that sort of thing. It can be a nice crutch when you're in a pinch for time. A good example of a tool that does this is TSAN. It gobbles up virtual address space, which is practically free, thanks to the magic of modern MMUs. That's too powerful and good a thing to need to give up completely.
        kaba0 2y ago
        Another example would be the JVM’s new low-latency GC, ZGC, that does some smart addressing tricks to store some information in pointer addresses themselves, effectively tripling the actual memory usage when naively looked at through virt memory.
CAP_NET_ADMIN 2y ago
The truth is that if you're having issues with default overcommit configuration (namely overcommit_memory == 0 and overcommit_ratio == 50) your application probably sucks and you have to actually diagnose it and fix it. "We run a pretty java-heavy environment, with multiple large JVMs configured per host. The problem is that the heap sizes have been getting larger, and we were running in an overcommitted situation and did not realize it. The JVMs would all start up and malloc() their large heaps, and then at some later time once enough of the heaps were actually used, the OOM killer would kick in and more or less randomly off one of our JVMs." I know that 2007 may have been different times, but I'd argue that max heaps for all your JVMs running on a system probably shouldn't exceed around 88% of the total system memory. (percentage goes up as the total system memory goes up from 128GB -> 256GB -> 512GB)
- samus 2y ago
  HotSpot has a handy command line option for this issue: `-XX:+AlwaysPreTouch`, which forces the OS to actually assign memory and thus increases the likelihood to expose such issues.
- tredre3 2y ago
  > The truth is that if you're having issues with default overcommit configuration your application probably sucks and you have to actually diagnose it and fix it. An alternative interpretation of the issue would be that if your OS allows a single process to overcommit more than the sum of total RAM+SWAP on the system, then maybe it is it who sucks.
  - tsss 2y ago
    It is a hen and egg situation. The OS has to do this because many applications have learned to rely on the behaviour and will pre-allocate huge amounts of memory that they never use. Linux practically forces applications into this because the only way to start a new process is to fork yourself which will instantly double virtual memory usage for no reason. There's also the problem that a lack of overcommit tends to lead to underutilization. I'm eagerly waiting for the day that I no longer have to size my workloads manually on kubernetes.
crest 2y ago
The problem with overcommitting beyond the sum of RAM and swap is that userspace processes can't protect themselves against the kernel's lies. You allocate memory, handle the result, access the address space and someone dies. The Linux default behaviour is an insane over-optimisation, but without it forking large processes becomes painfully expensive. The proper solution is to spawn directly through suitable system calls instead of the traditional fork() + execve() way. Imo a good compromise to this day is to overcommit RAM with swap as fallback for correctness even if performance falls of a cliff if you use it. All of that is on top of resource limits and usage monitoring.
sylware 2y ago
I have been a linux user/dev for a while now, and it seems I have the wrong idea of what is memory overcommit: For me, for a modern large CPU implementation, it simplifies a lot the userland software stack to be able to book huge virtual address spaces in userland, which are populated on page faults. If really needed, I can "release" some memory ranges of this memory virtual address space (linux munmap has some options related to just that). I do realize I am working more and more in a 'infinite memory model' (modulo some "released" ranges from time to time). For instance, I would be more that happy to book a few 64GiB virtual address ranges on my small 8GiB workstation. I am talking about anonymous memory, not file backed mmaping which is another story.

63 points by signa11 2y ago

Reminder: if overcommit as a concept seems distasteful, the real ire should be directed at the Unix fork syscall, an API that will always fail on a process using over 50% of the available memory without overcommit. Ideally apps would use vfork or posix_spawn instead, but that isn't the world we live in. Overcommit is a sensible way to help apps that use a lot of memory "just work". You can always turn it off if you don't need apps using a lot of memory to be able to fork.

senderista 2y ago
There are also legitimate uses of overcommit for reserving huge areas of virtual memory and relying on demand paging to allocate physical memory, although strictly speaking enabling overcommit isn’t necessary if you separate the “reserve” (mmap(PROT_NONE)) and “commit” (mmap(MAP_FIXED, PROT_READ | PROT_WRITE)) stages (which is what Windows forces you to do).
dap 2y ago
I’m not sure how this works on Linux exactly but in principle you don’t need to support overcommit to avoid that problem. On illumos for example anonymous memory allocations come from swap, which is a virtual resource that includes both disk and memory. You do still have this problem if something is using more than half of swap, but (1) that’s much larger, and (2) you can augment it with more disk space, which won’t actually be used unless stuff actually uses all the physical memory. This has its own tradeoffs but it’s useful to keep in mind that there are other ways to do things.
- wmf 2y ago
  If available memory means physical + swap then you're saying the same thing as pcwalton.
  - pcwalton 2y ago
    Right, available memory includes swap. (That's why I didn't say "available RAM".)
thanatos519 2y ago
I thought fork was copy on write nowadays.
- layer8 2y ago
  It is, but “write” doesn’t have an error code it can return on out-of-memory. The whole crux of overcommit is that the affected process doesn’t get a chance to gracefully fall back or shutdown on OOM conditions at defined places in the code (like at invocations of malloc).
- wmf 2y ago
  Fork is copy-on-write. The problem is that there's no way for fork() to know how much memory the child process needs. It might call exec() immediately, using very little memory. It might keep running, mostly sharing pages with the parent. It might touch every memory page, requiring a lot of memory.
userbinator 2y ago
That explains why Windows, which has a process model based on spawning and not forking, never had overcommit.
- ynik 2y ago
  However Windows has stacks that commit on-demand, which is kinda similar to overcommit. It can result in programs terminating on any random function call (if that call requires allocating a new page of stack). https://learn.microsoft.com/en-us/windows/win32/procthread/thread-stack-size https://learn.microsoft.com/en-us/windows/win32/procthread/t... However this is very rare to happen in practice -- stack growth isn't very frequent in typical applications; so usually a malloc() will fail first in low-memory situations. A Windows program can avoid the risk of getting terminated on stack growth by specifying equal commit and reserve sizes for the stack. So in least in theory, it's possible to write a windows program that is reliable in low memory situations.
jabl 2y ago
This article explains the problems with fork pretty well: https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0...
rcxdude 2y ago
Also I would suggest using a Windows machine which seems to run out of memory despite hundreds of gigabytes of swap and yet still ~70% of RAM full. I don't know what's causing the bug but it affects multiple systems I use and it's a right nuisance with a nearly full disk to have >10% of it sitting there idle just in case whatever's over-allocating memory decides to actually use it (I mean, I guess I could also direct my ire at whatever broken accounting fails to actually tell me what's causing it as well, but I can be mad at two things at once).
shoo 2y ago
yup. i recall first being exposed to this while trying to understand why a memory-hog backend service was being oom-killed in production while attempting to spawn a resource-light subprocess via fork+exec, while there was still a good portion of free memory to use. the more you read about linux oom killer, overcommit, linux memory accounting heuristics, copy-on-write, fork+exec as a mechanism to spawn an unrelated process, the more surprising it is that anything works at all even some of the time.

__turbobrew__ 2y ago

If you want to optimize for throughout one option is to overcommit resources (CPU/MEM/IO/NET) but utilize backpressure mechanisms to reduce load during times of saturation. Kubernetes does this through node pressure eviction but it is pretty easy to hook into the pressure stall information and have the application handle this as well (for example, start returning 429 HTTP responses when PSI goes over a certain level). At the end of the day the optimal overcommit is workload dependent — good metrics and an iterative approach is needed.

eru 2y ago
You probably also want to do somewhat intelligent load shedding, where you basically make sure you drop the most expendable items first. That's a basic fact of life for CPU cycles even for consumers on their machines, we call that just CPU priorities or so. But it's also useful for the other resources; but there it's a bit more disruptive, because we generally need to disrupt a program to eg claim back the disk space it uses.

johnea 2y ago

Is any of this still applicable in 2024? One has to doubt...

graymatters 2y ago
Doubt - based on what?
- worthless-trash 2y ago
  Knowledge, GP probably thinks that the kernel world moves like the userspace (probably js) world. Which clearly isnt true. If OS development worked like userspace/web dev we'd all be in much more pain.

nabla9 2y ago

As then as now, you can manage it. vm.overcommit_memory == 0 heuristic overcommit vm.overcommit_memory == 1 (full overcommit) allows allocating more memory than there is ram + swap. vm.overcommit_memory == 2 never overcommit. There must be enough physical ram or virtual swap to allocate the memory.

layer8 2y ago
Can this be set for a specific process (or process group/tree), or only globally?
- nabla9 2y ago
  Globally For a process you can use setrlimit() (ulimit command)
  - silon42 2y ago
    I'd want the OOM killer to kill processes with more over-commit first.
    - jart 2y ago
      Why? There are many innocent applications that consume lots of virtual memory with no intention of using it. Sometimes sparse algorithms call for that sort of thing. It can be a nice crutch when you're in a pinch for time. A good example of a tool that does this is TSAN. It gobbles up virtual address space, which is practically free, thanks to the magic of modern MMUs. That's too powerful and good a thing to need to give up completely.
      - kaba0 2y ago
        Another example would be the JVM’s new low-latency GC, ZGC, that does some smart addressing tricks to store some information in pointer addresses themselves, effectively tripling the actual memory usage when naively looked at through virt memory.

CAP_NET_ADMIN 2y ago

The truth is that if you're having issues with default overcommit configuration (namely overcommit_memory == 0 and overcommit_ratio == 50) your application probably sucks and you have to actually diagnose it and fix it. "We run a pretty java-heavy environment, with multiple large JVMs configured per host. The problem is that the heap sizes have been getting larger, and we were running in an overcommitted situation and did not realize it. The JVMs would all start up and malloc() their large heaps, and then at some later time once enough of the heaps were actually used, the OOM killer would kick in and more or less randomly off one of our JVMs." I know that 2007 may have been different times, but I'd argue that max heaps for all your JVMs running on a system probably shouldn't exceed around 88% of the total system memory. (percentage goes up as the total system memory goes up from 128GB -> 256GB -> 512GB)

samus 2y ago
HotSpot has a handy command line option for this issue: `-XX:+AlwaysPreTouch`, which forces the OS to actually assign memory and thus increases the likelihood to expose such issues.
tredre3 2y ago
> The truth is that if you're having issues with default overcommit configuration your application probably sucks and you have to actually diagnose it and fix it. An alternative interpretation of the issue would be that if your OS allows a single process to overcommit more than the sum of total RAM+SWAP on the system, then maybe it is it who sucks.
- tsss 2y ago
  It is a hen and egg situation. The OS has to do this because many applications have learned to rely on the behaviour and will pre-allocate huge amounts of memory that they never use. Linux practically forces applications into this because the only way to start a new process is to fork yourself which will instantly double virtual memory usage for no reason. There's also the problem that a lack of overcommit tends to lead to underutilization. I'm eagerly waiting for the day that I no longer have to size my workloads manually on kubernetes.

crest 2y ago

The problem with overcommitting beyond the sum of RAM and swap is that userspace processes can't protect themselves against the kernel's lies. You allocate memory, handle the result, access the address space and someone dies. The Linux default behaviour is an insane over-optimisation, but without it forking large processes becomes painfully expensive. The proper solution is to spawn directly through suitable system calls instead of the traditional fork() + execve() way. Imo a good compromise to this day is to overcommit RAM with swap as fallback for correctness even if performance falls of a cliff if you use it. All of that is on top of resource limits and usage monitoring.

sylware 2y ago

I have been a linux user/dev for a while now, and it seems I have the wrong idea of what is memory overcommit: For me, for a modern large CPU implementation, it simplifies a lot the userland software stack to be able to book huge virtual address spaces in userland, which are populated on page faults. If really needed, I can "release" some memory ranges of this memory virtual address space (linux munmap has some options related to just that). I do realize I am working more and more in a 'infinite memory model' (modulo some "released" ranges from time to time). For instance, I would be more that happy to book a few 64GiB virtual address ranges on my small 8GiB workstation. I am talking about anonymous memory, not file backed mmaping which is another story.