3 ms·

IIUC, the way overcommit works on Linux is the memory returned by malloc() is not actually mapped to physical memory until it is accessed. So malloc may succeed

IIUC, the way overcommit works on Linux is the memory returned by malloc() is not actually mapped to physical memory until it is accessed. So malloc may succeed and then at some point later you may get a segfault or similar when something actually tries to access that memory.

cyberax 2y ago
malloc() will not return unmapped memory for small allocations. Your process might get killed if malloc() fails to get more backing pages.
- dent9876543 2y ago
  Do you mean that my process might get killed [by the OOM killer] if /another process/ fails to get more backing pages? Or are you suggesting that there is some other failure path that results in my process getting killed if my own malloc() fails to get pages? (That seems wrong. Should simply return NULL in that case, no?)
  - cyberax 2y ago
    > Do you mean that my process might get killed [by the OOM killer] if /another process/ fails to get more backing pages? It can be your process or some other process, depending on the OOM killer weights and heuristics. > Or are you suggesting that there is some other failure path that results in my process getting killed if my own malloc() fails to get pages? (That seems wrong. Should simply return NULL in that case, no?) No, malloc() in Linux will never return NULL for small allocations. What happens is that glibc will try to get more pages for its heap by calling mmap(). The call to mmap() will result in the OOM killer waking up and freeing some memory. It then can kill some other process, and your process mmap() will succeed, or it can kill your process, and in this case mmap() never returns.
dataflow 2y ago
> IIUC, the way overcommit works on Linux Doesn't vm.overcommit_memory == 2 disable overcommit? That's the situation we're talking about...
- usefulcat 2y ago
  You're correct; for some reason I didn't interpret the question in that context. I was thinking about when vm.overcommit_memmory is 0 or 1.

cyberax 2y ago

malloc() will not return unmapped memory for small allocations. Your process might get killed if malloc() fails to get more backing pages.

dent9876543 2y ago
Do you mean that my process might get killed [by the OOM killer] if /another process/ fails to get more backing pages? Or are you suggesting that there is some other failure path that results in my process getting killed if my own malloc() fails to get pages? (That seems wrong. Should simply return NULL in that case, no?)
- cyberax 2y ago
  > Do you mean that my process might get killed [by the OOM killer] if /another process/ fails to get more backing pages? It can be your process or some other process, depending on the OOM killer weights and heuristics. > Or are you suggesting that there is some other failure path that results in my process getting killed if my own malloc() fails to get pages? (That seems wrong. Should simply return NULL in that case, no?) No, malloc() in Linux will never return NULL for small allocations. What happens is that glibc will try to get more pages for its heap by calling mmap(). The call to mmap() will result in the OOM killer waking up and freeing some memory. It then can kill some other process, and your process mmap() will succeed, or it can kill your process, and in this case mmap() never returns.

dataflow 2y ago

> IIUC, the way overcommit works on Linux Doesn't vm.overcommit_memory == 2 disable overcommit? That's the situation we're talking about...

usefulcat 2y ago
You're correct; for some reason I didn't interpret the question in that context. I was thinking about when vm.overcommit_memmory is 0 or 1.