4 ms·
There is some support for this actually. Namely, when overcommit_memory is set to its default value (0), then it actually implements a heuristic for overcommit.
by burntsushi 7y ago
There is some support for this actually. Namely, when overcommit_memory is set to its default value (0), then it actually implements a heuristic for overcommit. (Where as the value `1` corresponds to "always overcommit.") Namely, only when overcommit_memory is set to 0, if you allocate memory with mmap and pass the MAP_NORESERVE option, then allocation behaves as if overcommit is always enabled and there is no check. But if you don't pass that option, then there is a heuristic check which could cause malloc to fail.
I discovered this because of differences in how jemalloc and my system allocator (from glibc) allocated memory. Namely, jemalloc would pass the MAP_NORESERVE option, which meant my program behaved as if overcommit was always enabled. Thus, an out-of-memory condition meant that it was subject to the OOM killer. But when I used my system's allocator, the MAP_NORESERVE option was not passed, which led to the heuristic detecting an out-of-memory condition and causing the malloc call to fail.
For more info, see `man 5 proc` and/or my exploration here: https://github.com/BurntSushi/ripgrep/issues/993#issuecomment-408262555 https://github.com/BurntSushi/ripgrep/issues/993#issuecommen...
(It's all still a bit wishy washy, so I think your point still stands. But I figured I'd chime in with some bits that don't appear to be widely known, and which led to some surprising differences in behavior based on which allocator one chooses.)
- dooglius 7y agoThe OOM killer can strike at any time, regardless of the mapping options you've set, even if you aren't explicitly or implicitly allocating new memory. What I'd like, and what I think twic is asking for, is a way to "opt-out" of the overcommit paradigm at a process level, so that allocations by a given process reserve memory and may fail, but in return the kernel promises not to OOM kill it.
- cheerlessbog 7y agoMost of my programming experience is in Windows. Can someone briefly explain why it doesn't seem to need an OOM killer? Is it happier to page? Or it always commits on allocation? Or something else.
- dooglius 7y agoWindows does not have fork(), which as a sibling comment indicates is one of the main issues with removing overcommit.
- detaro 7y agoAFAIK windows lets whatever application asks for memory and can't get it crash, whereas the OOM killer tries to be intelligent about what gets killed.
- cesarb 7y agoAFAIK, Windows always commits on allocation, growing the swap file if necessary.
- cheerlessbog 7y agoWhy doesn't Linux do this?
- bscphil 7y ago>so that allocations by a given process reserve memory and may fail, but in return the kernel promises not to OOM kill it The issue with this is that there are other processes on the system. If I start a program that used your idea for allocation, and it used 90% of my memory, a small growth in my memory usage might mean I run out. But because it's guaranteed not to OOM, another program gets killed rather than the one using 90% of the memory. If you want no OOM kills, you've got to disable overcommit globally.