5 ms·
As then as now, you can manage it. vm.overcommit_memory == 0 heuristic overcommit vm.overcommit_memory == 1 (full overcommit) allows allocating more memory
by nabla9 2y ago
As then as now, you can manage it.
vm.overcommit_memory == 0 heuristic overcommit
vm.overcommit_memory == 1 (full overcommit) allows allocating more memory than there is ram + swap.
vm.overcommit_memory == 2 never overcommit. There must be enough physical ram or virtual swap to allocate the memory.
- layer8 2y agoCan this be set for a specific process (or process group/tree), or only globally?
- nabla9 2y agoGlobally For a process you can use setrlimit() (ulimit command)
- silon42 2y agoI'd want the OOM killer to kill processes with more over-commit first.
- jart 2y agoWhy? There are many innocent applications that consume lots of virtual memory with no intention of using it. Sometimes sparse algorithms call for that sort of thing. It can be a nice crutch when you're in a pinch for time. A good example of a tool that does this is TSAN. It gobbles up virtual address space, which is practically free, thanks to the magic of modern MMUs. That's too powerful and good a thing to need to give up completely.
- kaba0 2y agoAnother example would be the JVM’s new low-latency GC, ZGC, that does some smart addressing tricks to store some information in pointer addresses themselves, effectively tripling the actual memory usage when naively looked at through virt memory.
- layer8 2y agoSuch applications could handle this explicitly, e.g. with mmap and mprotect. In principle there is no reason for such use-cases to rely on implicit overcommit behavior. Which also isn't present on Windows.
- kaba0 2y agoIt might actually be using that, I’m not intimately familiar with the topic.
- layer8 2y agoYes, I would expect that as well. Implicit overcommit is really only necessary for fork and general backwards compatibility.
- jart 2y agoThat's because Windows forces you to use special userspace libraries that account for memory and let you create "reservations" for large regions, without actually populating entries in the MMU (i.e. PML4t). Linux doesn't have that. You create reservations for large memory regions by allocating virtual memory, which of course doesn't become real (resident) until you use it. That's overcommit. But since there's no userspace libraries like WIN32 forced upon you on Linux, there's really no way for the kernel to know your intent when it comes to memory, because the abstractions for memory on Linux are so simple. Windows has a ridiculous byzantine system, which some people think is better. But people are really missing the point when they try to make an apples-to-apples comparison between the two systems of memory management. It's not a question of (a) overcommit good, vs. (b) overcommit bad. Anyone who thinks either is a fool. Linux is simply a system that requires a smart system administrator who understands your use cases and can tune sysctl accordingly when you want to push beyond the normal boundaries.
- layer8 2y agoIt seems to me that the prot parameter and/or MAP_NORESERVE would indicate the intent.
- kevingadd 2y agoI like to run my development VM with overcommit_memory == 2, but I recently learned that the way it actually fails is pretty confusing. It's not an orderly 'abort: malloc failed' or process termination like I would've expected. Instead, you get weird intermittent failures that don't look like OOMs and your system appears to have plenty of free memory at the time. Still good for flushing out scenarios where i.e. your build process needs 64GB of memory for approximately 500ms and then drops back down to reasonable levels, due to parallelism
- dataflow 2y agoThat sounds wild. How else can it possibly fail than by failing malloc? Where do the intermittent failures occur, if not in the return value? Could it be some part of your code isn't checking the return value for NULL?
- usefulcat 2y agoIIUC, the way overcommit works on Linux is the memory returned by malloc() is not actually mapped to physical memory until it is accessed. So malloc may succeed and then at some point later you may get a segfault or similar when something actually tries to access that memory.
- cyberax 2y agomalloc() will not return unmapped memory for small allocations. Your process might get killed if malloc() fails to get more backing pages.
- dent9876543 2y agoDo you mean that my process might get killed [by the OOM killer] if /another process/ fails to get more backing pages? Or are you suggesting that there is some other failure path that results in my process getting killed if my own malloc() fails to get pages? (That seems wrong. Should simply return NULL in that case, no?)