7 ms·

The title here is just blatantly false; there are certainly some scenarios in which it won't fail, but many in which it will. The author is aware of this since

The title here is just blatantly false; there are certainly some scenarios in which it won't fail, but many in which it will. The author is aware of this since he fixes the claim toward the end: > To clarify, the surprising behaviour malloc has does not mean we should ignore its return value. We just need to be careful because malloc returning successfully does not always mean that we can use the requested memory. It's worth pointing out that if you turn overcommit off (https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou...) you will, in fact, get this guarantee

chrisseaton 7y ago
> does not always mean > if you turn overcommit off The author said ‘does not always mean’ so giving one condition in which it does doesn’t prove them wrong does it?
- dooglius 7y ago
  You're right, based on the context of the post, I was interpreting that statement to make a broader claim than it was; I've modified my comment not to contradict it.
twic 7y ago
It's a shame that turning off overcommit is done at the system level, rather than the malloc call level. Some applications, and perhaps some allocations within otherwise naive applications, might prefer to have an allocation fail early, where failure could be sensibly handled.
- mannykannot 7y ago
  I think that would ultimately have undesirable consequences. If this were an option on a malloc call, any such call would have to reserve the allocated memory at that point, reducing the usefulness of overcommit for other processes. This would set up a 'tragedy of the commons' scenario, where every application developer defensively uses this feature because other applications are using it.
  - dooglius 7y ago
    It's already quite easy to implement manually though, just do a write after allocation.
    - mannykannot 7y ago
      That's a good point, though I think one would have to write to each of the pages allocated to reserve the full extent of the memory allocated by the malloc call.
  - twic 7y ago
    You could mitigate that by treating these allocations conservatively: they only succeed if they can reserve the memory they want and there is 'enough' uncommitted memory left to cover all as-yet-uncommitted normal allocations. So, it would fail long before a normal allocation would. I don't know quite what 'enough' means, but this whole business is already a rat's nest of heuristics, so one more should fit in nicely.
- burntsushi 7y ago
  There is some support for this actually. Namely, when overcommit_memory is set to its default value (0), then it actually implements a heuristic for overcommit. (Where as the value `1` corresponds to "always overcommit.") Namely, only when overcommit_memory is set to 0, if you allocate memory with mmap and pass the MAP_NORESERVE option, then allocation behaves as if overcommit is always enabled and there is no check. But if you don't pass that option, then there is a heuristic check which could cause malloc to fail. I discovered this because of differences in how jemalloc and my system allocator (from glibc) allocated memory. Namely, jemalloc would pass the MAP_NORESERVE option, which meant my program behaved as if overcommit was always enabled. Thus, an out-of-memory condition meant that it was subject to the OOM killer. But when I used my system's allocator, the MAP_NORESERVE option was not passed, which led to the heuristic detecting an out-of-memory condition and causing the malloc call to fail. For more info, see `man 5 proc` and/or my exploration here: https://github.com/BurntSushi/ripgrep/issues/993#issuecomment-408262555 https://github.com/BurntSushi/ripgrep/issues/993#issuecommen... (It's all still a bit wishy washy, so I think your point still stands. But I figured I'd chime in with some bits that don't appear to be widely known, and which led to some surprising differences in behavior based on which allocator one chooses.)
  - dooglius 7y ago
    The OOM killer can strike at any time, regardless of the mapping options you've set, even if you aren't explicitly or implicitly allocating new memory. What I'd like, and what I think twic is asking for, is a way to "opt-out" of the overcommit paradigm at a process level, so that allocations by a given process reserve memory and may fail, but in return the kernel promises not to OOM kill it.
    - cheerlessbog 7y ago
      Most of my programming experience is in Windows. Can someone briefly explain why it doesn't seem to need an OOM killer? Is it happier to page? Or it always commits on allocation? Or something else.
      - 7y ago
joosters 7y ago
n.b. turning over-commit off comes with its own set of problems, such as causing programs to fail long before all memory has really been exhausted. For example, fork() will have to ensure that there is enough memory for a complete copy of the running process. If you have a large process, e.g. using 4GB of a 8GB machine, then fork() won't be able to run, even if you just want to fork and run a tiny program. With over-commit turned on, the fork() would work (because of copy-on-write) and the program could then happily exec() the new program. The work-around is to allocate huge amounts of swap space so that the OS can be confident that it can reserve all the potentially required memory from fork(), even if it never normally has to use all that memory.
- caf 7y ago
  Or if you're just forking to exec a tiny program, use vfork().
  - deleted 7y ago
    [deleted]
- convolvatron 7y ago
  cow is orthogonal to overcommit
- kj4ips 7y ago
  Redis is an example of this. When redis saves in the background, it will for and use COW to shapshot a point in time, while the main redis thread keeps going with the normal redis. If you disable overcommit, redis can fail to fork, when there is actually sufficient memory to save the db.
dragontamer 7y ago
> It's worth pointing out that if you turn overcommit off ... you will, in fact, get this guarantee But many Linux applications assume the overcommit behavior. Its far easier to just "go with the flow" or "When in Rome...". Overcommit is the programming culture of Linux and should be assumed when writing Linux apps.
- dooglius 7y ago
  That has not been my experience. I disable overcommit on my machines and it has been considerably nicer to have apps exit due to malloc failing than random processes getting OOM-killed.
  - bscphil 7y ago
    My experience with the oom-killer has been consistently terrible. Stuff like the X server getting killed taking down the whole user session instead of whatever was actually using the most memory. Probably happening because it's the parent process to everything else, but it's still a mind-blowingly dumb way to handle the low memory condition.
- deleted 7y ago
  [deleted]
jiveturkey 7y ago
> The title here is just blatantly false intentionally. you don't need to point out the "error". it's a rhetorical device.

chrisseaton 7y ago

> does not always mean > if you turn overcommit off The author said ‘does not always mean’ so giving one condition in which it does doesn’t prove them wrong does it?

dooglius 7y ago
You're right, based on the context of the post, I was interpreting that statement to make a broader claim than it was; I've modified my comment not to contradict it.

twic 7y ago

It's a shame that turning off overcommit is done at the system level, rather than the malloc call level. Some applications, and perhaps some allocations within otherwise naive applications, might prefer to have an allocation fail early, where failure could be sensibly handled.

mannykannot 7y ago
I think that would ultimately have undesirable consequences. If this were an option on a malloc call, any such call would have to reserve the allocated memory at that point, reducing the usefulness of overcommit for other processes. This would set up a 'tragedy of the commons' scenario, where every application developer defensively uses this feature because other applications are using it.
- dooglius 7y ago
  It's already quite easy to implement manually though, just do a write after allocation.
  - mannykannot 7y ago
    That's a good point, though I think one would have to write to each of the pages allocated to reserve the full extent of the memory allocated by the malloc call.
- twic 7y ago
  You could mitigate that by treating these allocations conservatively: they only succeed if they can reserve the memory they want and there is 'enough' uncommitted memory left to cover all as-yet-uncommitted normal allocations. So, it would fail long before a normal allocation would. I don't know quite what 'enough' means, but this whole business is already a rat's nest of heuristics, so one more should fit in nicely.
burntsushi 7y ago
There is some support for this actually. Namely, when overcommit_memory is set to its default value (0), then it actually implements a heuristic for overcommit. (Where as the value `1` corresponds to "always overcommit.") Namely, only when overcommit_memory is set to 0, if you allocate memory with mmap and pass the MAP_NORESERVE option, then allocation behaves as if overcommit is always enabled and there is no check. But if you don't pass that option, then there is a heuristic check which could cause malloc to fail. I discovered this because of differences in how jemalloc and my system allocator (from glibc) allocated memory. Namely, jemalloc would pass the MAP_NORESERVE option, which meant my program behaved as if overcommit was always enabled. Thus, an out-of-memory condition meant that it was subject to the OOM killer. But when I used my system's allocator, the MAP_NORESERVE option was not passed, which led to the heuristic detecting an out-of-memory condition and causing the malloc call to fail. For more info, see `man 5 proc` and/or my exploration here: https://github.com/BurntSushi/ripgrep/issues/993#issuecomment-408262555 https://github.com/BurntSushi/ripgrep/issues/993#issuecommen... (It's all still a bit wishy washy, so I think your point still stands. But I figured I'd chime in with some bits that don't appear to be widely known, and which led to some surprising differences in behavior based on which allocator one chooses.)
- dooglius 7y ago
  The OOM killer can strike at any time, regardless of the mapping options you've set, even if you aren't explicitly or implicitly allocating new memory. What I'd like, and what I think twic is asking for, is a way to "opt-out" of the overcommit paradigm at a process level, so that allocations by a given process reserve memory and may fail, but in return the kernel promises not to OOM kill it.
  - cheerlessbog 7y ago
    Most of my programming experience is in Windows. Can someone briefly explain why it doesn't seem to need an OOM killer? Is it happier to page? Or it always commits on allocation? Or something else.
    - 7y ago

joosters 7y ago

n.b. turning over-commit off comes with its own set of problems, such as causing programs to fail long before all memory has really been exhausted. For example, fork() will have to ensure that there is enough memory for a complete copy of the running process. If you have a large process, e.g. using 4GB of a 8GB machine, then fork() won't be able to run, even if you just want to fork and run a tiny program. With over-commit turned on, the fork() would work (because of copy-on-write) and the program could then happily exec() the new program. The work-around is to allocate huge amounts of swap space so that the OS can be confident that it can reserve all the potentially required memory from fork(), even if it never normally has to use all that memory.

caf 7y ago
Or if you're just forking to exec a tiny program, use vfork().
- deleted 7y ago
  [deleted]
convolvatron 7y ago
cow is orthogonal to overcommit
kj4ips 7y ago
Redis is an example of this. When redis saves in the background, it will for and use COW to shapshot a point in time, while the main redis thread keeps going with the normal redis. If you disable overcommit, redis can fail to fork, when there is actually sufficient memory to save the db.

dragontamer 7y ago

> It's worth pointing out that if you turn overcommit off ... you will, in fact, get this guarantee But many Linux applications assume the overcommit behavior. Its far easier to just "go with the flow" or "When in Rome...". Overcommit is the programming culture of Linux and should be assumed when writing Linux apps.

dooglius 7y ago
That has not been my experience. I disable overcommit on my machines and it has been considerably nicer to have apps exit due to malloc failing than random processes getting OOM-killed.
- bscphil 7y ago
  My experience with the oom-killer has been consistently terrible. Stuff like the X server getting killed taking down the whole user session instead of whatever was actually using the most memory. Probably happening because it's the parent process to everything else, but it's still a mind-blowingly dumb way to handle the low memory condition.
deleted 7y ago
[deleted]

jiveturkey 7y ago

> The title here is just blatantly false intentionally. you don't need to point out the "error". it's a rhetorical device.