4 ms·

> malloc() return isn't checked small digression, but I thought that at least on Linux, malloc() never returns an error because the actual allocation happens l

> malloc() return isn't checked small digression, but I thought that at least on Linux, malloc() never returns an error because the actual allocation happens lazily, when the memory is first used?

formerly_proven 6y ago
On 64 bits malloc basically always succeeds, because you are very likely not running out of virtual memory. You can totally go ahead and malloc terabytes of RAM. On 32 bits malloc is going to fail once your virtual memory space is full (~3 GB). Error checking is somewhat redundant for most applications, since you're going to abort anyway, and if there are no pages available on access - well you're getting SIGSEGV anyway, just like you would accessing a null pointer (except on embedded devices). But beware of the exceptions. In C especially it's common to use null pointers as a flag... one of the reasons why new/delete in C++ is preferable; it throws and thus aborts when it fails, which you don't care to handle, so that's about perfect for a C++ exception.
fulafel 6y ago
Cases where memory allocations fail on linux include: hitting memory resource limits (ulimit), overcommit behaviour is set to stricter than the default via sysctl, kernel heuristics with the default overcommit settings end up failing allocation, container / cgroups limits, 32-bit virtual address space is exhausted - i'm sure there are more. The default overcommit-within-reason algorithm is designed to deny allocations that are obviously unrealistic I think. Using semi conservative ulimit settings is pretty common in interactive use to catch runaway / swapped to death situations.
simias 6y ago
It's sloppy coding and not portable. If you really don't want to check for malloc() return value because you consider that it shouldn't fail (or that you won't be able to recover from it anyway) just implement an xmalloc() that just aborts on failure and do that. At least it'll crash "cleanly" this way.
- mcpcpc 6y ago
  will try to address the malloc concerns asap
nyc_pizzadev 6y ago
Take a look at vm.max_map_count. A lot of systems have that set artificially low and that will force a memory allocator to fail and return NULL. I can confirm this is the case for jemalloc. Also, on Linux, only 48 of 64 bits are available for VM addressing. That leaves you with about 250TB of address space. Seems like a lot until you start using VM for disk mmaps. I have actually seen this limit hit.
- moonchild 6y ago
  > on Linux, only 48 of 64 bits are available for VM That is on amd64, not linux. And arm does the same thing. (It's also 256tb, not 250tb.) Newer intel cpus use 5-level paging[1], which gives you 128pb VM. 1. https://en.wikipedia.org/wiki/Intel_5-level_paging https://en.wikipedia.org/wiki/Intel_5-level_paging
  - fulafel 6y ago
    Also it's 48-bit just on current amd64 chips, it's not an architectural amd64 property - your binary is compatible with >48bit VA space of the future (unless you yourself misguidedly bake the 48-bit assumption into it). But I guess the point was to point out lower bounds.
    - moonchild 6y ago
      Right; hence the mention of intel 5-level paging with 56-bit virtual address space.
knorker 6y ago
In addition to what others have said, which basically boils down to "it'll never return an error… unless it does", there's also the aspect where if you start assuming things about the kernel environment you're not coding in "POSIX C99" like the title says, but to "Debian GNU/Linux circa 2020". And that's part of the "and do it well". A thousand years from now, if the C99&POSIX spec survives, will support correctly written code. Dereferencing a null pointer can also be a security issue (mostly for kernels, though) so one should assume a malicious general environment.

> malloc() return isn't checked small digression, but I thought that at least on Linux, malloc() never returns an error because the actual allocation happens lazily, when the memory is first used?

formerly_proven 6y ago

On 64 bits malloc basically always succeeds, because you are very likely not running out of virtual memory. You can totally go ahead and malloc terabytes of RAM. On 32 bits malloc is going to fail once your virtual memory space is full (~3 GB). Error checking is somewhat redundant for most applications, since you're going to abort anyway, and if there are no pages available on access - well you're getting SIGSEGV anyway, just like you would accessing a null pointer (except on embedded devices). But beware of the exceptions. In C especially it's common to use null pointers as a flag... one of the reasons why new/delete in C++ is preferable; it throws and thus aborts when it fails, which you don't care to handle, so that's about perfect for a C++ exception.

fulafel 6y ago

Cases where memory allocations fail on linux include: hitting memory resource limits (ulimit), overcommit behaviour is set to stricter than the default via sysctl, kernel heuristics with the default overcommit settings end up failing allocation, container / cgroups limits, 32-bit virtual address space is exhausted - i'm sure there are more. The default overcommit-within-reason algorithm is designed to deny allocations that are obviously unrealistic I think. Using semi conservative ulimit settings is pretty common in interactive use to catch runaway / swapped to death situations.

simias 6y ago

It's sloppy coding and not portable. If you really don't want to check for malloc() return value because you consider that it shouldn't fail (or that you won't be able to recover from it anyway) just implement an xmalloc() that just aborts on failure and do that. At least it'll crash "cleanly" this way.

mcpcpc 6y ago
will try to address the malloc concerns asap

nyc_pizzadev 6y ago

Take a look at vm.max_map_count. A lot of systems have that set artificially low and that will force a memory allocator to fail and return NULL. I can confirm this is the case for jemalloc. Also, on Linux, only 48 of 64 bits are available for VM addressing. That leaves you with about 250TB of address space. Seems like a lot until you start using VM for disk mmaps. I have actually seen this limit hit.

moonchild 6y ago
> on Linux, only 48 of 64 bits are available for VM That is on amd64, not linux. And arm does the same thing. (It's also 256tb, not 250tb.) Newer intel cpus use 5-level paging[1], which gives you 128pb VM. 1. https://en.wikipedia.org/wiki/Intel_5-level_paging https://en.wikipedia.org/wiki/Intel_5-level_paging
- fulafel 6y ago
  Also it's 48-bit just on current amd64 chips, it's not an architectural amd64 property - your binary is compatible with >48bit VA space of the future (unless you yourself misguidedly bake the 48-bit assumption into it). But I guess the point was to point out lower bounds.
  - moonchild 6y ago
    Right; hence the mention of intel 5-level paging with 56-bit virtual address space.

knorker 6y ago

In addition to what others have said, which basically boils down to "it'll never return an error… unless it does", there's also the aspect where if you start assuming things about the kernel environment you're not coding in "POSIX C99" like the title says, but to "Debian GNU/Linux circa 2020". And that's part of the "and do it well". A thousand years from now, if the C99&POSIX spec survives, will support correctly written code. Dereferencing a null pointer can also be a security issue (mostly for kernels, though) so one should assume a malicious general environment.