4 ms·

> In short, if the OS over-commits RAM, it is impossible for a program to safely handle out of memory situations. That's a non-sequitur. If you don't check fo

> In short, if the OS over-commits RAM, it is impossible for a program to safely handle out of memory situations. That's a non-sequitur. If you don't check for NULL return, you have undefined behaviour--that is: your program might end up doing just about anything. If the OOM killer kills you, all visible effects of your program will still be perfectly consistent with the semantics of your program. And you have to be able to safely deal with that scenario anyhow, as much more fundamental resources such as electricity might run out at any time as well.

joosters 11y ago
You're splitting hairs. Imagine your program is processing a folder of emails. If the program is halted part-way through (or if you yank the power cable out) then the mail folder will be left in an inconsistent state. If the program runs in an OS that does not overcommit memory, it is possible to write code that checks every malloc() and, if it hits a memory limit, you could shut down gracefully, fixing the mail folder so that its state is correct. If the OS overcommits, then even if you checked every malloc() call, your program might die at any time because of a SIGSEGV or the OOM killer nuking it. There's no way to tidy up an incomplete run. It's nothing to do with undefined behaviour, or dereferencing NULL pointers.
- jimrandomh 11y ago
  I think the point is that it's easier to use techniques that keep the mail folder always in a consistent or recoverable state, such as journaling or copy-modify-move, than it is to fully handle out-of-memory conditions. Since the former technique is both easier and can handle problems that the latter can't, it's always preferable.
- zAy0LfpBZLC8mAC 11y ago
  If you leave something in a corrupted state when execution is aborted at some random point, that means that your code (and possibly your on-disk data structure) is defective. It is perfectly possible to handle that case safely (using transactions, journaling, rollback, atomic replace, ...). However, if you don't check for allocation failures, you get undefined behaviour, which you indeed cannot handle safely ... other than by avoiding the undefined behaviour in the first place by checking for allocation failures. Also, if you check for allocation failures, you won't get a SIGSEGV. SIGSEGV is for invalid virtual addresses, not for lack of resources.
  - nightski 11y ago
    You might be technically right, but I guarantee you that by your definition the vast majority of software out there is defective. The solutions you mention are non-trivial. It is really easy to make a mistake. While it may not be the operating systems fault at the end of the day, designing a system like this is setting up the overall experience for failure. A well designed system should make it easy to get right and not the other way around.
    - zAy0LfpBZLC8mAC 11y ago
      > You might be technically right, but I guarantee you that by your definition the vast majority of software out there is defective. Your point being? That it's untechnically correct? > The solutions you mention are non-trivial. It is really easy to make a mistake. While it may not be the operating systems fault at the end of the day, designing a system like this is setting up the overall experience for failure. A well designed system should make it easy to get right and not the other way around. Which is all kindof true, but doesn't change that those APIs are the way they are, for historical reasons. Just because some API is broken, doesn't mean your code will work correctly when you pretend it's not broken. edit: just to be sure: yes, the special-casing of zero-sized allocations is somewhat of a bug, which we still have to live with. Other than that, the API is actuall perfectly fine--if you don't check for allocation failures and you can't handle it if your program gets interrupted at any point, that's just a bug in your code, there is no useful general way to abstract those problems away.
      - deleted 11y ago
        [deleted]
  - joosters 11y ago
    You can get memory errors outside of malloc() on an OS that overcommits RAM. Whether that is a SIGSEGV or some other signal, or just the heavy boots of the OOM killer nuking your process, doesn't really matter. The key point is that your program cannot handle them. It is impossible to handle every case. Allocation failures can and do occur outside of malloc() when overcommit is the memory policy. Malloc might return memory that it believes is valid, but when you try to write to it later on, the OS discovers that it has no free space in the swap and no spare pages to evict. Result: process death (SIGSEGV? SIGBUS? not sure, but it doesn't matter)

joosters 11y ago

You're splitting hairs. Imagine your program is processing a folder of emails. If the program is halted part-way through (or if you yank the power cable out) then the mail folder will be left in an inconsistent state. If the program runs in an OS that does not overcommit memory, it is possible to write code that checks every malloc() and, if it hits a memory limit, you could shut down gracefully, fixing the mail folder so that its state is correct. If the OS overcommits, then even if you checked every malloc() call, your program might die at any time because of a SIGSEGV or the OOM killer nuking it. There's no way to tidy up an incomplete run. It's nothing to do with undefined behaviour, or dereferencing NULL pointers.

jimrandomh 11y ago
I think the point is that it's easier to use techniques that keep the mail folder always in a consistent or recoverable state, such as journaling or copy-modify-move, than it is to fully handle out-of-memory conditions. Since the former technique is both easier and can handle problems that the latter can't, it's always preferable.
zAy0LfpBZLC8mAC 11y ago
If you leave something in a corrupted state when execution is aborted at some random point, that means that your code (and possibly your on-disk data structure) is defective. It is perfectly possible to handle that case safely (using transactions, journaling, rollback, atomic replace, ...). However, if you don't check for allocation failures, you get undefined behaviour, which you indeed cannot handle safely ... other than by avoiding the undefined behaviour in the first place by checking for allocation failures. Also, if you check for allocation failures, you won't get a SIGSEGV. SIGSEGV is for invalid virtual addresses, not for lack of resources.
- nightski 11y ago
  You might be technically right, but I guarantee you that by your definition the vast majority of software out there is defective. The solutions you mention are non-trivial. It is really easy to make a mistake. While it may not be the operating systems fault at the end of the day, designing a system like this is setting up the overall experience for failure. A well designed system should make it easy to get right and not the other way around.
  - zAy0LfpBZLC8mAC 11y ago
    > You might be technically right, but I guarantee you that by your definition the vast majority of software out there is defective. Your point being? That it's untechnically correct? > The solutions you mention are non-trivial. It is really easy to make a mistake. While it may not be the operating systems fault at the end of the day, designing a system like this is setting up the overall experience for failure. A well designed system should make it easy to get right and not the other way around. Which is all kindof true, but doesn't change that those APIs are the way they are, for historical reasons. Just because some API is broken, doesn't mean your code will work correctly when you pretend it's not broken. edit: just to be sure: yes, the special-casing of zero-sized allocations is somewhat of a bug, which we still have to live with. Other than that, the API is actuall perfectly fine--if you don't check for allocation failures and you can't handle it if your program gets interrupted at any point, that's just a bug in your code, there is no useful general way to abstract those problems away.
    - deleted 11y ago
      [deleted]
- joosters 11y ago
  You can get memory errors outside of malloc() on an OS that overcommits RAM. Whether that is a SIGSEGV or some other signal, or just the heavy boots of the OOM killer nuking your process, doesn't really matter. The key point is that your program cannot handle them. It is impossible to handle every case. Allocation failures can and do occur outside of malloc() when overcommit is the memory policy. Malloc might return memory that it believes is valid, but when you try to write to it later on, the OS discovers that it has no free space in the swap and no spare pages to evict. Result: process death (SIGSEGV? SIGBUS? not sure, but it doesn't matter)