4 ms·
I have always wondered about Linux's overcommit feature, which BTW I dislike -- Doesn't ISO C require that if malloc returns a nonzero pointer, it has to backed
by srean 9y ago
I have always wondered about Linux's overcommit feature, which BTW I dislike -- Doesn't ISO C require that if malloc returns a nonzero pointer, it has to backed physically (need not be RAM, could be swap on disk or some other thing). I have been told that the OS is allowed to overcommit but space obtained through malloc needs to be backed, if it has to comply with ISO C. Apparently AIX and ISO committee got into some argument about this when AIX's malloc implementation allowed for overcommit, anyone remembers this ?
- benou 9y agoYou can always disable it using 'sysctl -w vm.overcommit_memory=2' or have more control with vm.overcommit_* various sysctls. In the general case, I found that overcommit is usually more useful than strict accounting, but I agree that for more industrial cases it can be the opposite (but then you can spend time tuning it).
- srean 9y agoOh! of course. I was seeking comments on whether it is true that this behavior is not ISO C compliant. I understand why this behavior is default, if you _have_ to support fork() this perhaps the more pragmatic default.
- nwmcsween 9y agoThe issue with backing all malloced memory with unused memory is that a lot of programs are written with a worst case assumption.
- srean 9y agoSort of a chicken and egg thing. If the API and behavior makes it easy to abuse, people will abuse it. It might have helped if there were two different library functions (i) malloc that gave you physically backed memory and (ii) another that just reserved some address space.
- dozzie 9y agoChris Siebenmann wrote an article explaining why it's a pragmatic thing: https://utcc.utoronto.ca/~cks/space/blog/tech/OvercommittingMemoryIsSensible https://utcc.utoronto.ca/~cks/space/blog/tech/Overcommitting...
- mnarayan01 9y agoUnless you have vm.overcommit_memory set to 1, then AIUI you can implement malloc s.t. a read/write to the returned memory won't generate a SIGSEGV, e.g. using mmap sans MAP_NORESERVE. Even if vm.overcommit_memory=1, you can still (I assume) do it by simply having your malloc implementation populate each page. That said, as long as "too small to fail" (https://lwn.net/Articles/627419/ https://lwn.net/Articles/627419/) is around (which is presumably "forever"), the oom-killer won't be going away. Fork obviously compounds this massively due to copy-on-write. So ensuring your malloc'd memory is actually writable may just make you a more tempting target for the oom-killer. > Doesn't ISO C require that if malloc returns a nonzero pointer, it has to backed physically (need not be RAM, could be swap on disk or some other thing) I don't see anything like that, though admittedly I don't have a "real" copy of the standard. I'd also be at least a little surprised if it were true (though my being surprised by the C standard would...actually not be surprising). While the "spirit of the law" would probably make sense, the "letter of the law" would be difficult. Consider e.g. an otherwise side-effect free function with a statically computable result assuming a successful malloc (and corresponding free), could the compiler optimize away the malloc call?