3 ms·

This is one of those "let's ignore malloc failing" bugs. This is pretty common in C code and is, IMHO, a common "worse is better"[1] anti-pattern, which is pr

This is one of those "let's ignore malloc failing" bugs. This is pretty common in C code and is, IMHO, a common "worse is better"[1] anti-pattern, which is predicated on the "worse is better" principle that correctness can be sacrified if it makes things too complicated. Many programmers, unfortunately, don't give a crap about things that far off the happy path, and when the language implements checked exceptions or Option types they get upset that it forces them to think about those distracting corner cases. Such is the state of software engineering. 1. https://en.m.wikipedia.org/wiki/Worse_is_better https://en.m.wikipedia.org/wiki/Worse_is_better

mattst88 11y ago
The project I work on, Mesa3D, routinely ignores malloc failures for a couple of reasons: no one knows how to reasonably test the additional code, and often no one has any idea what sort of graceful failure is even possible when malloc fails. We've had some cargo culted patches to call _mesa_error_no_memory() when malloc fails, but it was recently noted that this can't possibly work because it internally calls fprintf, and there's no guarantee that fprintf doesn't call malloc! In fact, glibc's implementation does. Suggestions for best practices welcome.
- Sanddancer 11y ago
  If you can't fail gracefully, then crash as soon as possible. Trying to sputter along is just asking for worse problems.
  - webkike 11y ago
    Well, considering malloc gives a NULL pointer on failure, a crash at some point is pretty clearly inevitable.
    - Sanddancer 11y ago
      Yes, but it becomes harder to tell where in the code that happened. If you run abort() or the like, you've got more information as to the circumstances of the crash.
    - richm44 11y ago
      Sadly it doesn't on linux due to overcommit. You can actually get the failure much later when you try to use the memory. The exception is when you try to allocate absolutely huge buffers. So malloc returning a valid pointer doesn't indicate success (though it returning NULL does indicate failure).
  - striking 11y ago
    (these kinds of) Crashes can be manipulated, though. If you can't restore a stable state, you have created instability. Failing gracefully in this context means failing without creating instability or insecurity.
    - teacup50 11y ago
      An abort()/__builtin_trap/etc shouldn't generally be manipulable.
      - AnthonyMouse 11y ago
        The failure mode is the one where you terminate in the middle of an operation that was expected to be atomic or ordered and then the program restarts and does step two without finishing step one.
        cesarb 11y ago
        You have to protect against that anyway, otherwise you're vulnerable to power cuts.
        Johnny555 11y ago
        And operating system crashes, and being killed by the OOM handler (which is probably likely if you're out of heap), and all of the other things that can kill a process without warning.
- webkike 11y ago
  It is quite possible to write to files, stdout, etc without using any heap memory. It can be achieved with the bare bones read/write syscalls. I'm not sure I would consider this a "best practice", but it's worth noting. Boehm GC actively uses read when it needs to read files without allocating any heap memory.
  - justincormack 11y ago
    In general, for a library, this is not good practise. For an application, it may work. You have to be very careful to not use any formatted output.
- cesarb 11y ago
  The proper way to deal with a memory allocation failure, when a graceful return isn't possible, is to call abort() or similar, instead of trying to continue with a potentially inconsistent state. Of course, it would be better to avoid getting into a situation where either a graceful return isn't possible, or poorly-tested recovery code has to be run to deal with the failure. I read somewhere that seL4's strategy is to divide the work in two phases. The first phase allocates all the resources it will need and checks all preconditions, but changes no state; the second phase has no allocations, does all the work, and will never fail. That way, any error recovery is a simple release of the resources it had already allocated.
- greglindahl 11y ago
  Check out the concept of fault injection -- you can wrap all of the malloc() calls and the systematically cause each one to fail. You might want to combine that with valgrind to make sure you don't end up doing undefined things after a malloc failure. You can also use that wrapping technique to test _mesa_error_no_memory(), by setting the state to "all mallocs fail" and then calling it. Hint: Don't call fprintf, don't use stdio, call write() directly. Fault injection can't cover all of the combinations of all inputs and all places to fail, but it's a lot better than nothing.
- CJefferson 11y ago
  If you don't have anything better to do, just abort(). One (naughty) option is to malloc a decent sized buffer at startup (50MB say), and then when malloc fails free it, and immediately display a warning box. One most modern 64-bit systems, malloc will never fail anyway, you'll get killed due to memory overcommit.
  - caf 11y ago
    That's not so hotso for a library though, which Mesa3D is.
    - ori_b 11y ago
      The alternative, if you have no good way of testing the codepaths, or if you're running on a system with overcommitted VM, is basically segfaulting. And you're running on a system with overcommitted VM. An abort with a reason is a far better choice.
      - caf 11y ago
        Overcommit is not actually relevant. An overcommit OOM-kill is a manifestation of the administrator killing the process for being a memory hog (it's just that the administrator has delegated this decision to some less-than-scrutable kernel code). Library code should in general punt out-of-memory conditions back to the caller.
        ori_b 11y ago
        > Overcommit is not actually relevant. It is if you ever want to exercise the code paths. Overcommit means you never get an out of memory error. Your process just dies. Kind of like it would if you just aborted.
        caf 11y ago
        When you're running tests in development you can easily set up a test environment with and without overcommit enabled (and in fact there are other ways to limit allocated memory on a per-process basis that work even with overcommit, such as with rlimits). ...but that doesn't even really matter, because in order to fully exercise those code paths you need precise control over exactly which call to malloc() returns NULL, which in practice means you need to interpose malloc() with a debugging framework to force a NULL return.
- signa11 11y ago
  > ... no one knows how to reasonably test the additional code ... wouldn't a simple 'ulimit -m' set appropriate limits on memory for the process ? and then use that to simulate failures, and hopefully see what breaks, fix it, and rinse-lather-repeat ? edit-1: this is ofcourse predicated on the fact that you are using a unix like system.
  - Arnavion 11y ago
    A Unix-like system that's not Linux, mind, since ulimit -m hasn't worked there in a long time.
pjmlp 11y ago
> Such is the state of software engineering. Yes, as Hoare stated in 1981, regarding Algol design: "The first principle was security: The principle that every syntactically incorrect program should be rejected by the compiler and that every syntactically correct program should give a result or an error message that was predictable and comprehensible in terms of the source language program itself. Thus no core dumps should ever be necessary. It was logically impossible for any source language program to cause the computer to run wild, either at compile time or at run time. A consequence of this principle is that every occurrence of every subscript of every subscripted variable was on every occasion checked at run time against both the upper and the lower declared bounds of the array. Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to - they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."
- tosseraccount 11y ago
  How did the market for Algol turn out?
  - nickpsecurity 11y ago
    Ada, Eiffel, Java, C#, and Go are doing fine. All making money. :P Three even used in operating systems with two in embedded systems. So, can be fairly efficient, too. Note: Not to mention Unisys still makes tons of money off their ALGOL machines from Burroughs. IBM & Fujitsu still service PL/1. So, old ones too.
  - pjmlp 11y ago
    Except for C, Objective-C(++) and C++, all other programming languages, even systems programming ones do bounds checking. And at least in C++, thanks to the STL or similar library there is the option to always enable bounds checking. So even if the market didn't got hold of Algol, except for C and languages that offer copy-paste compatibility with it, all other ones do offer sane defaults in terms of security.
icebraining 11y ago
Actually, I'd say that if you're trying to protect against lazy programmers, checked exceptions are worse than unchecked, since they'll just add an empty try/catch around the call, whereas crashing the process would usually be a safer option.
- PhantomGremlin 11y ago
  they'll just add an empty try/catch around the call In an ideal world, someone who did that should be taken out back and shot. In this real world, someone who did that should be immediately walked out the door. It's astonishing to me just how lazy programmers can be.
jlebar 11y ago
> the "worse is better" principle that correctness can be sacrified if it makes things too complicated To be clear, in systems designed with the "worse is better" approach, responsibility for correctness can be pushed up the stack, from e.g. kernel programmers to user-space programmers, if doing the Right Thing lower in the stack is hard. The example given in the paper is a syscall being interrupted -- in unix, it just fails with EINTR, and it's the application author's responsibility to retry. (I agree that malloc returning null is similar.) But worse is better isn't saying that programs shouldn't be correct if it's complicated; it's just a question of where the responsibility for correctly handling API edge cases should lie. Moreover, if one reads the "worse is better" paper carefully, you'll notice he doesn't say that "worse is better" actually is better, overall. The argument is only that it has better "survival characteristics". Meaning, as I read it, that they can gain and maintain momentum against competitors because they can move faster. Which, tangentially, is all very interesting to me, because the three modern systems with the best survival characteristics are unix, C, and the web, the latter of which is definitely not worse is better. (However bad you may think the web is as an API, "worse is better" is a design philosophy about giving users simple, low-level APIs, whereas the web is all about high-level APIs where the browser implements lots of logic to prevent you from shooting yourself in the foot.) So maybe we need to reconsider the conclusions of the worse is better paper in light of the giant counterexample that is the web.
- pcwalton 11y ago
  I'd also add Java, C#, Python, ARM (especially ARM64) and Windows NT as examples of success of better-is-better systems. (Windows NT is somewhat less better-is-better though due to Win32 compatibility.)
  - nickpsecurity 11y ago
    Maybe Python and ARM. Java was a was Worse is Better compromise trying to bring in features from Right Thing languages. Took lots of marketing to make it. C# similarly done with adoption mainly because Microsoft pushed it w/ tooling. They're both Worse is Better candidates. Just not as bad as some others. Modula-3 or maybe Eiffel would be closer to Right Thing examples in that category.
tptacek 11y ago
You say that, but then, there are examples of terrible security bugs resulting from broken attempts at handling malloc failures, too. The right answer is, I think: * Don't add checking code at individual malloc call-sites. * If you have an allocation regime where you need to do something better than abort in response to failure, don't use malloc directly for those allocations. * Run your program with malloc rigged to blow up if it fails.
deleted 11y ago
[deleted]

mattst88 11y ago

The project I work on, Mesa3D, routinely ignores malloc failures for a couple of reasons: no one knows how to reasonably test the additional code, and often no one has any idea what sort of graceful failure is even possible when malloc fails. We've had some cargo culted patches to call _mesa_error_no_memory() when malloc fails, but it was recently noted that this can't possibly work because it internally calls fprintf, and there's no guarantee that fprintf doesn't call malloc! In fact, glibc's implementation does. Suggestions for best practices welcome.

Sanddancer 11y ago
If you can't fail gracefully, then crash as soon as possible. Trying to sputter along is just asking for worse problems.
- webkike 11y ago
  Well, considering malloc gives a NULL pointer on failure, a crash at some point is pretty clearly inevitable.
  - Sanddancer 11y ago
    Yes, but it becomes harder to tell where in the code that happened. If you run abort() or the like, you've got more information as to the circumstances of the crash.
  - richm44 11y ago
    Sadly it doesn't on linux due to overcommit. You can actually get the failure much later when you try to use the memory. The exception is when you try to allocate absolutely huge buffers. So malloc returning a valid pointer doesn't indicate success (though it returning NULL does indicate failure).
- striking 11y ago
  (these kinds of) Crashes can be manipulated, though. If you can't restore a stable state, you have created instability. Failing gracefully in this context means failing without creating instability or insecurity.
  - teacup50 11y ago
    An abort()/__builtin_trap/etc shouldn't generally be manipulable.
    - AnthonyMouse 11y ago
      The failure mode is the one where you terminate in the middle of an operation that was expected to be atomic or ordered and then the program restarts and does step two without finishing step one.
      - cesarb 11y ago
        You have to protect against that anyway, otherwise you're vulnerable to power cuts.
        Johnny555 11y ago
        And operating system crashes, and being killed by the OOM handler (which is probably likely if you're out of heap), and all of the other things that can kill a process without warning.
webkike 11y ago
It is quite possible to write to files, stdout, etc without using any heap memory. It can be achieved with the bare bones read/write syscalls. I'm not sure I would consider this a "best practice", but it's worth noting. Boehm GC actively uses read when it needs to read files without allocating any heap memory.
- justincormack 11y ago
  In general, for a library, this is not good practise. For an application, it may work. You have to be very careful to not use any formatted output.
cesarb 11y ago
The proper way to deal with a memory allocation failure, when a graceful return isn't possible, is to call abort() or similar, instead of trying to continue with a potentially inconsistent state. Of course, it would be better to avoid getting into a situation where either a graceful return isn't possible, or poorly-tested recovery code has to be run to deal with the failure. I read somewhere that seL4's strategy is to divide the work in two phases. The first phase allocates all the resources it will need and checks all preconditions, but changes no state; the second phase has no allocations, does all the work, and will never fail. That way, any error recovery is a simple release of the resources it had already allocated.
greglindahl 11y ago
Check out the concept of fault injection -- you can wrap all of the malloc() calls and the systematically cause each one to fail. You might want to combine that with valgrind to make sure you don't end up doing undefined things after a malloc failure. You can also use that wrapping technique to test _mesa_error_no_memory(), by setting the state to "all mallocs fail" and then calling it. Hint: Don't call fprintf, don't use stdio, call write() directly. Fault injection can't cover all of the combinations of all inputs and all places to fail, but it's a lot better than nothing.
CJefferson 11y ago
If you don't have anything better to do, just abort(). One (naughty) option is to malloc a decent sized buffer at startup (50MB say), and then when malloc fails free it, and immediately display a warning box. One most modern 64-bit systems, malloc will never fail anyway, you'll get killed due to memory overcommit.
- caf 11y ago
  That's not so hotso for a library though, which Mesa3D is.
  - ori_b 11y ago
    The alternative, if you have no good way of testing the codepaths, or if you're running on a system with overcommitted VM, is basically segfaulting. And you're running on a system with overcommitted VM. An abort with a reason is a far better choice.
    - caf 11y ago
      Overcommit is not actually relevant. An overcommit OOM-kill is a manifestation of the administrator killing the process for being a memory hog (it's just that the administrator has delegated this decision to some less-than-scrutable kernel code). Library code should in general punt out-of-memory conditions back to the caller.
      - ori_b 11y ago
        > Overcommit is not actually relevant. It is if you ever want to exercise the code paths. Overcommit means you never get an out of memory error. Your process just dies. Kind of like it would if you just aborted.
        caf 11y ago
        When you're running tests in development you can easily set up a test environment with and without overcommit enabled (and in fact there are other ways to limit allocated memory on a per-process basis that work even with overcommit, such as with rlimits). ...but that doesn't even really matter, because in order to fully exercise those code paths you need precise control over exactly which call to malloc() returns NULL, which in practice means you need to interpose malloc() with a debugging framework to force a NULL return.
signa11 11y ago
> ... no one knows how to reasonably test the additional code ... wouldn't a simple 'ulimit -m' set appropriate limits on memory for the process ? and then use that to simulate failures, and hopefully see what breaks, fix it, and rinse-lather-repeat ? edit-1: this is ofcourse predicated on the fact that you are using a unix like system.
- Arnavion 11y ago
  A Unix-like system that's not Linux, mind, since ulimit -m hasn't worked there in a long time.

pjmlp 11y ago

> Such is the state of software engineering. Yes, as Hoare stated in 1981, regarding Algol design: "The first principle was security: The principle that every syntactically incorrect program should be rejected by the compiler and that every syntactically correct program should give a result or an error message that was predictable and comprehensible in terms of the source language program itself. Thus no core dumps should ever be necessary. It was logically impossible for any source language program to cause the computer to run wild, either at compile time or at run time. A consequence of this principle is that every occurrence of every subscript of every subscripted variable was on every occasion checked at run time against both the upper and the lower declared bounds of the array. Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to - they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."

tosseraccount 11y ago
How did the market for Algol turn out?
- nickpsecurity 11y ago
  Ada, Eiffel, Java, C#, and Go are doing fine. All making money. :P Three even used in operating systems with two in embedded systems. So, can be fairly efficient, too. Note: Not to mention Unisys still makes tons of money off their ALGOL machines from Burroughs. IBM & Fujitsu still service PL/1. So, old ones too.
- pjmlp 11y ago
  Except for C, Objective-C(++) and C++, all other programming languages, even systems programming ones do bounds checking. And at least in C++, thanks to the STL or similar library there is the option to always enable bounds checking. So even if the market didn't got hold of Algol, except for C and languages that offer copy-paste compatibility with it, all other ones do offer sane defaults in terms of security.

icebraining 11y ago

Actually, I'd say that if you're trying to protect against lazy programmers, checked exceptions are worse than unchecked, since they'll just add an empty try/catch around the call, whereas crashing the process would usually be a safer option.

PhantomGremlin 11y ago
they'll just add an empty try/catch around the call In an ideal world, someone who did that should be taken out back and shot. In this real world, someone who did that should be immediately walked out the door. It's astonishing to me just how lazy programmers can be.

jlebar 11y ago

> the "worse is better" principle that correctness can be sacrified if it makes things too complicated To be clear, in systems designed with the "worse is better" approach, responsibility for correctness can be pushed up the stack, from e.g. kernel programmers to user-space programmers, if doing the Right Thing lower in the stack is hard. The example given in the paper is a syscall being interrupted -- in unix, it just fails with EINTR, and it's the application author's responsibility to retry. (I agree that malloc returning null is similar.) But worse is better isn't saying that programs shouldn't be correct if it's complicated; it's just a question of where the responsibility for correctly handling API edge cases should lie. Moreover, if one reads the "worse is better" paper carefully, you'll notice he doesn't say that "worse is better" actually is better, overall. The argument is only that it has better "survival characteristics". Meaning, as I read it, that they can gain and maintain momentum against competitors because they can move faster. Which, tangentially, is all very interesting to me, because the three modern systems with the best survival characteristics are unix, C, and the web, the latter of which is definitely not worse is better. (However bad you may think the web is as an API, "worse is better" is a design philosophy about giving users simple, low-level APIs, whereas the web is all about high-level APIs where the browser implements lots of logic to prevent you from shooting yourself in the foot.) So maybe we need to reconsider the conclusions of the worse is better paper in light of the giant counterexample that is the web.

pcwalton 11y ago
I'd also add Java, C#, Python, ARM (especially ARM64) and Windows NT as examples of success of better-is-better systems. (Windows NT is somewhat less better-is-better though due to Win32 compatibility.)
- nickpsecurity 11y ago
  Maybe Python and ARM. Java was a was Worse is Better compromise trying to bring in features from Right Thing languages. Took lots of marketing to make it. C# similarly done with adoption mainly because Microsoft pushed it w/ tooling. They're both Worse is Better candidates. Just not as bad as some others. Modula-3 or maybe Eiffel would be closer to Right Thing examples in that category.

tptacek 11y ago

You say that, but then, there are examples of terrible security bugs resulting from broken attempts at handling malloc failures, too. The right answer is, I think: * Don't add checking code at individual malloc call-sites. * If you have an allocation regime where you need to do something better than abort in response to failure, don't use malloc directly for those allocations. * Run your program with malloc rigged to blow up if it fails.

deleted 11y ago

[deleted]