5 ms·

Reminder: if overcommit as a concept seems distasteful, the real ire should be directed at the Unix fork syscall, an API that will always fail on a process usin

Reminder: if overcommit as a concept seems distasteful, the real ire should be directed at the Unix fork syscall, an API that will always fail on a process using over 50% of the available memory without overcommit. Ideally apps would use vfork or posix_spawn instead, but that isn't the world we live in. Overcommit is a sensible way to help apps that use a lot of memory "just work". You can always turn it off if you don't need apps using a lot of memory to be able to fork.

senderista 2y ago
There are also legitimate uses of overcommit for reserving huge areas of virtual memory and relying on demand paging to allocate physical memory, although strictly speaking enabling overcommit isn’t necessary if you separate the “reserve” (mmap(PROT_NONE)) and “commit” (mmap(MAP_FIXED, PROT_READ | PROT_WRITE)) stages (which is what Windows forces you to do).
dap 2y ago
I’m not sure how this works on Linux exactly but in principle you don’t need to support overcommit to avoid that problem. On illumos for example anonymous memory allocations come from swap, which is a virtual resource that includes both disk and memory. You do still have this problem if something is using more than half of swap, but (1) that’s much larger, and (2) you can augment it with more disk space, which won’t actually be used unless stuff actually uses all the physical memory. This has its own tradeoffs but it’s useful to keep in mind that there are other ways to do things.
- wmf 2y ago
  If available memory means physical + swap then you're saying the same thing as pcwalton.
  - pcwalton 2y ago
    Right, available memory includes swap. (That's why I didn't say "available RAM".)
thanatos519 2y ago
I thought fork was copy on write nowadays.
- layer8 2y ago
  It is, but “write” doesn’t have an error code it can return on out-of-memory. The whole crux of overcommit is that the affected process doesn’t get a chance to gracefully fall back or shutdown on OOM conditions at defined places in the code (like at invocations of malloc).
- wmf 2y ago
  Fork is copy-on-write. The problem is that there's no way for fork() to know how much memory the child process needs. It might call exec() immediately, using very little memory. It might keep running, mostly sharing pages with the parent. It might touch every memory page, requiring a lot of memory.
userbinator 2y ago
That explains why Windows, which has a process model based on spawning and not forking, never had overcommit.
- ynik 2y ago
  However Windows has stacks that commit on-demand, which is kinda similar to overcommit. It can result in programs terminating on any random function call (if that call requires allocating a new page of stack). https://learn.microsoft.com/en-us/windows/win32/procthread/thread-stack-size https://learn.microsoft.com/en-us/windows/win32/procthread/t... However this is very rare to happen in practice -- stack growth isn't very frequent in typical applications; so usually a malloc() will fail first in low-memory situations. A Windows program can avoid the risk of getting terminated on stack growth by specifying equal commit and reserve sizes for the stack. So in least in theory, it's possible to write a windows program that is reliable in low memory situations.
jabl 2y ago
This article explains the problems with fork pretty well: https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0...
rcxdude 2y ago
Also I would suggest using a Windows machine which seems to run out of memory despite hundreds of gigabytes of swap and yet still ~70% of RAM full. I don't know what's causing the bug but it affects multiple systems I use and it's a right nuisance with a nearly full disk to have >10% of it sitting there idle just in case whatever's over-allocating memory decides to actually use it (I mean, I guess I could also direct my ire at whatever broken accounting fails to actually tell me what's causing it as well, but I can be mad at two things at once).
shoo 2y ago
yup. i recall first being exposed to this while trying to understand why a memory-hog backend service was being oom-killed in production while attempting to spawn a resource-light subprocess via fork+exec, while there was still a good portion of free memory to use. the more you read about linux oom killer, overcommit, linux memory accounting heuristics, copy-on-write, fork+exec as a mechanism to spawn an unrelated process, the more surprising it is that anything works at all even some of the time.

senderista 2y ago

There are also legitimate uses of overcommit for reserving huge areas of virtual memory and relying on demand paging to allocate physical memory, although strictly speaking enabling overcommit isn’t necessary if you separate the “reserve” (mmap(PROT_NONE)) and “commit” (mmap(MAP_FIXED, PROT_READ | PROT_WRITE)) stages (which is what Windows forces you to do).

dap 2y ago

I’m not sure how this works on Linux exactly but in principle you don’t need to support overcommit to avoid that problem. On illumos for example anonymous memory allocations come from swap, which is a virtual resource that includes both disk and memory. You do still have this problem if something is using more than half of swap, but (1) that’s much larger, and (2) you can augment it with more disk space, which won’t actually be used unless stuff actually uses all the physical memory. This has its own tradeoffs but it’s useful to keep in mind that there are other ways to do things.

wmf 2y ago
If available memory means physical + swap then you're saying the same thing as pcwalton.
- pcwalton 2y ago
  Right, available memory includes swap. (That's why I didn't say "available RAM".)

thanatos519 2y ago

I thought fork was copy on write nowadays.

layer8 2y ago
It is, but “write” doesn’t have an error code it can return on out-of-memory. The whole crux of overcommit is that the affected process doesn’t get a chance to gracefully fall back or shutdown on OOM conditions at defined places in the code (like at invocations of malloc).
wmf 2y ago
Fork is copy-on-write. The problem is that there's no way for fork() to know how much memory the child process needs. It might call exec() immediately, using very little memory. It might keep running, mostly sharing pages with the parent. It might touch every memory page, requiring a lot of memory.

userbinator 2y ago

That explains why Windows, which has a process model based on spawning and not forking, never had overcommit.

ynik 2y ago
However Windows has stacks that commit on-demand, which is kinda similar to overcommit. It can result in programs terminating on any random function call (if that call requires allocating a new page of stack). https://learn.microsoft.com/en-us/windows/win32/procthread/thread-stack-size https://learn.microsoft.com/en-us/windows/win32/procthread/t... However this is very rare to happen in practice -- stack growth isn't very frequent in typical applications; so usually a malloc() will fail first in low-memory situations. A Windows program can avoid the risk of getting terminated on stack growth by specifying equal commit and reserve sizes for the stack. So in least in theory, it's possible to write a windows program that is reliable in low memory situations.

jabl 2y ago

This article explains the problems with fork pretty well: https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0...

rcxdude 2y ago

Also I would suggest using a Windows machine which seems to run out of memory despite hundreds of gigabytes of swap and yet still ~70% of RAM full. I don't know what's causing the bug but it affects multiple systems I use and it's a right nuisance with a nearly full disk to have >10% of it sitting there idle just in case whatever's over-allocating memory decides to actually use it (I mean, I guess I could also direct my ire at whatever broken accounting fails to actually tell me what's causing it as well, but I can be mad at two things at once).

shoo 2y ago

yup. i recall first being exposed to this while trying to understand why a memory-hog backend service was being oom-killed in production while attempting to spawn a resource-light subprocess via fork+exec, while there was still a good portion of free memory to use. the more you read about linux oom killer, overcommit, linux memory accounting heuristics, copy-on-write, fork+exec as a mechanism to spawn an unrelated process, the more surprising it is that anything works at all even some of the time.