4 ms·
The project I work on, Mesa3D, routinely ignores malloc failures for a couple of reasons: no one knows how to reasonably test the additional code, and often no
by mattst88 11y ago
The project I work on, Mesa3D, routinely ignores malloc failures for a couple of reasons: no one knows how to reasonably test the additional code, and often no one has any idea what sort of graceful failure is even possible when malloc fails.
We've had some cargo culted patches to call _mesa_error_no_memory() when malloc fails, but it was recently noted that this can't possibly work because it internally calls fprintf, and there's no guarantee that fprintf doesn't call malloc! In fact, glibc's implementation does.
Suggestions for best practices welcome.
- Sanddancer 11y agoIf you can't fail gracefully, then crash as soon as possible. Trying to sputter along is just asking for worse problems.
- webkike 11y agoWell, considering malloc gives a NULL pointer on failure, a crash at some point is pretty clearly inevitable.
- Sanddancer 11y agoYes, but it becomes harder to tell where in the code that happened. If you run abort() or the like, you've got more information as to the circumstances of the crash.
- richm44 11y agoSadly it doesn't on linux due to overcommit. You can actually get the failure much later when you try to use the memory. The exception is when you try to allocate absolutely huge buffers. So malloc returning a valid pointer doesn't indicate success (though it returning NULL does indicate failure).
- striking 11y ago(these kinds of) Crashes can be manipulated, though. If you can't restore a stable state, you have created instability. Failing gracefully in this context means failing without creating instability or insecurity.
- teacup50 11y agoAn abort()/__builtin_trap/etc shouldn't generally be manipulable.
- AnthonyMouse 11y agoThe failure mode is the one where you terminate in the middle of an operation that was expected to be atomic or ordered and then the program restarts and does step two without finishing step one.
- webkike 11y agoIt is quite possible to write to files, stdout, etc without using any heap memory. It can be achieved with the bare bones read/write syscalls. I'm not sure I would consider this a "best practice", but it's worth noting. Boehm GC actively uses read when it needs to read files without allocating any heap memory.
- justincormack 11y agoIn general, for a library, this is not good practise. For an application, it may work. You have to be very careful to not use any formatted output.
- cesarb 11y agoThe proper way to deal with a memory allocation failure, when a graceful return isn't possible, is to call abort() or similar, instead of trying to continue with a potentially inconsistent state. Of course, it would be better to avoid getting into a situation where either a graceful return isn't possible, or poorly-tested recovery code has to be run to deal with the failure. I read somewhere that seL4's strategy is to divide the work in two phases. The first phase allocates all the resources it will need and checks all preconditions, but changes no state; the second phase has no allocations, does all the work, and will never fail. That way, any error recovery is a simple release of the resources it had already allocated.
- greglindahl 11y agoCheck out the concept of fault injection -- you can wrap all of the malloc() calls and the systematically cause each one to fail. You might want to combine that with valgrind to make sure you don't end up doing undefined things after a malloc failure. You can also use that wrapping technique to test _mesa_error_no_memory(), by setting the state to "all mallocs fail" and then calling it. Hint: Don't call fprintf, don't use stdio, call write() directly. Fault injection can't cover all of the combinations of all inputs and all places to fail, but it's a lot better than nothing.
- CJefferson 11y agoIf you don't have anything better to do, just abort(). One (naughty) option is to malloc a decent sized buffer at startup (50MB say), and then when malloc fails free it, and immediately display a warning box. One most modern 64-bit systems, malloc will never fail anyway, you'll get killed due to memory overcommit.
- caf 11y agoThat's not so hotso for a library though, which Mesa3D is.
- ori_b 11y agoThe alternative, if you have no good way of testing the codepaths, or if you're running on a system with overcommitted VM, is basically segfaulting. And you're running on a system with overcommitted VM. An abort with a reason is a far better choice.
- caf 11y agoOvercommit is not actually relevant. An overcommit OOM-kill is a manifestation of the administrator killing the process for being a memory hog (it's just that the administrator has delegated this decision to some less-than-scrutable kernel code). Library code should in general punt out-of-memory conditions back to the caller.
- ori_b 11y ago> Overcommit is not actually relevant. It is if you ever want to exercise the code paths. Overcommit means you never get an out of memory error. Your process just dies. Kind of like it would if you just aborted.
- caf 11y agoWhen you're running tests in development you can easily set up a test environment with and without overcommit enabled (and in fact there are other ways to limit allocated memory on a per-process basis that work even with overcommit, such as with rlimits). ...but that doesn't even really matter, because in order to fully exercise those code paths you need precise control over exactly which call to malloc() returns NULL, which in practice means you need to interpose malloc() with a debugging framework to force a NULL return.
- signa11 11y ago> ... no one knows how to reasonably test the additional code ... wouldn't a simple 'ulimit -m' set appropriate limits on memory for the process ? and then use that to simulate failures, and hopefully see what breaks, fix it, and rinse-lather-repeat ? edit-1: this is ofcourse predicated on the fact that you are using a unix like system.
- Arnavion 11y agoA Unix-like system that's not Linux, mind, since ulimit -m hasn't worked there in a long time.