17 ms·

What about the assumption (fatally flawed decision?) that malloc never fails when rust asks? Sounds like something that could affect concurrency

anon4 11y ago
Malloc never fails, but you might die if you touch the memory. In general, modern OSes don't have a good story about exhausting available memory beyond "let's kill a bunch of processes to free up memory".
- masklinn 11y ago
  > Malloc never fails malloc can fail, even on default linux (overcommit enabled), if you go above the process's vmem limit for instance (because 32b or rlimited). And of course not all OS overcommit, Windows famously does not.
Manishearth 11y ago
A failed malloc aborts the process. If this is important to you, don't use the heap abstractions in the stdlib then. This is no different from the situation in C++. (You can also plug in a custom allocator which behaves differently)
- qznc 11y ago
  On Linux malloc never fails actually. Instead, the kernel kills processes if it runs out of memory.
  - pm215 11y ago
    Overcommit doesn't mean that malloc never fails. malloc will fail if you ask for an allocation that won't fit in the virtual address space (for instance because of your rlimit settings, or if you asked for a 3G chunk on a 32-bit system).
  - cyphar 11y ago
    Overcommit will fail if you go over the overcommit ratio. In addition, the process will die if you start using the fake pages the kernel gave you. But yes, malloc can fail on Linux.
  - Manishearth 11y ago
    A lot of folks are pointing out that malloc can fail, which is true, but the important part is that there are situations where your application will just abort randomly in the middle of nowhere (i.e. not during malloc) and there's nothing you can do about it. There are also situations where malloc fails and returns a null, but given the existence of situations with no errors on a malloc, handling the error in these cases isn't close to a solution. No language or stdlib can solve this problem 100%. At a baremetal level, it helps having this, but you're probably better off not using the stdlib anyway.
    - quotemstr 11y ago
      Bullshit. Every part of the C++ standard library (which is _much_ bigger than Rust's) can gracefully propagate resource exhaustion errors, including memory allocation failure, to callers. That Rust can't is a design flaw. > there are situations where your application will just abort randomly That's sure as hell not the case in any environment I choose to use.
      - Manishearth 11y ago
        > can gracefully propagate resource exhaustion errors, including memory allocation failure, to callers only on malloc. If the kernel overcommits, your process will abort when you try to use the memory, possibly way after the malloc and there's nothing you can do about it. That's the point being made here. > That Rust can't is a design flaw. (This is false, see Steve's reply above about this)
        quotemstr 11y ago
        The world is not Linux. I happen to believe that believe that overcommit in the Linux kernel is a disgrace. It is, however, at least possible to disable it. It's not possible to retroactively add real exceptions to Rust, or to change the signature of all memory-allocation function to return Result. Rust is supposed to be a general-purpose systems programming language, not a Linux programming language. Windows does not overcommit. A correctly configured Linux system does not overcommit. Lost of embedded systems don't (and can't) overcommit. Are you saying all of these people should avoid Rust's standard library? > > That Rust can't is a design flaw. > (This is false, see Steve's reply above about this) It's clear that my opinion differs from that of many Rust developers and users. I still think I'm correct, that these developers and users are misguided, and that as Rust attempts to fill more niches, experience will show that my position is the correct one. All I can say is that I personally will not use any language that bakes cornucopian assumptions about memory baked into its core library. I know that you say that it's possible to just avoid stdlib --- but the temptation to use it will be irresistible, and once somebody succumbs to the temptation, the entire program is now capable of aborting irrecoverably. I will stick with languages . Modern C++ is safe and expressive enough, and it correctly reacts to resource exhaustion.
  - bzbarsky 11y ago
    > On Linux malloc never fails actually. Not true. For example, it can fail if there is no big enough chunk of virtual address space available (i.e. your heap is fragmented enough and your attempted allocation is big enough). I've even seen 64-bit processes manage to do this, my mmapping lots of multi-GB things at once and then trying to do large allocations.
  - mtanski 11y ago
    You're correct in the way that it's default behavior, but the behavior can be turned off. Also, if you're using cgroups (or anything that leverages that like containers) you can put a limit on the memory resource and then malloc will fail. Which is common enough, and only becoming more common as people are collocating a lot of disparate workloads on nodes (using the kurbenets scheduler).
- quotemstr 11y ago
  No custom allocator can make a task that fails to allocate gracefully report an error. Rust's error handling design is just terrible, and mostly a consequence of eschewing exceptions. Had Rust opted for exceptions, it'd be a much better, and actually usable, language. Rust's terribly error-handling strategy is the chief reason not to use it.
  - steveklabnik 11y ago
    > No custom allocator can make the task gracefully report failure > instead of panicing. So, first of all, "custom allocators" means two things: * overloading the allocator that's used by liballoc, and the crates that depend on it, like libstd * other allocators entirely The first is described here: https://doc.rust-lang.org/book/custom-allocators.html https://doc.rust-lang.org/book/custom-allocators.html And the second is still in RFCs: https://github.com/rust-lang/rfcs/pull/1398 https://github.com/rust-lang/rfcs/pull/1398 Both of these things are not yet stable. The second does, in fact, give you the ability to return an error code, by returning a Result. However, on top of that, I don't see how > mostly a consequence of eschewing exceptions. and > No custom allocator can make the task gracefully report failure > instead of panicing. Work together. Or rather, why is panic-ing bad, but an exception good?
    - deleted 11y ago
      [deleted]
    - quotemstr 11y ago
      > why is panic-ing bad, but an exception good? Because the Rust people don't believe in making "catch" a first-class primitive in the language, and in fact, fully support a runtime option to turn all panics into aborts. Even if abort-on-panic were to be killed as a legal mode of operation, and even if the stigma were to be removed from std::panic::recover, we'd still be left with a language with two error handling strategies and endless programmer confusion over which to use. Rust's designers have done permanent damage to the language by not making exceptions the primary error reporting mechanism available to programmers, and it's not a mistake they can undo now.
losvedir 11y ago
That's not a property of rust the language, but the standard library. I believe you could use other allocators that behave differently. In any case, the standard library panics on OOM, and panics are described at the bottom of the linked page.
- kzrdude 11y ago
  calls abort(), not panic!(). This is important since unwinding does not happen with the former.

anon4 11y ago

Malloc never fails, but you might die if you touch the memory. In general, modern OSes don't have a good story about exhausting available memory beyond "let's kill a bunch of processes to free up memory".

masklinn 11y ago
> Malloc never fails malloc can fail, even on default linux (overcommit enabled), if you go above the process's vmem limit for instance (because 32b or rlimited). And of course not all OS overcommit, Windows famously does not.

Manishearth 11y ago

A failed malloc aborts the process. If this is important to you, don't use the heap abstractions in the stdlib then. This is no different from the situation in C++. (You can also plug in a custom allocator which behaves differently)

qznc 11y ago
On Linux malloc never fails actually. Instead, the kernel kills processes if it runs out of memory.
- pm215 11y ago
  Overcommit doesn't mean that malloc never fails. malloc will fail if you ask for an allocation that won't fit in the virtual address space (for instance because of your rlimit settings, or if you asked for a 3G chunk on a 32-bit system).
- cyphar 11y ago
  Overcommit will fail if you go over the overcommit ratio. In addition, the process will die if you start using the fake pages the kernel gave you. But yes, malloc can fail on Linux.
- Manishearth 11y ago
  A lot of folks are pointing out that malloc can fail, which is true, but the important part is that there are situations where your application will just abort randomly in the middle of nowhere (i.e. not during malloc) and there's nothing you can do about it. There are also situations where malloc fails and returns a null, but given the existence of situations with no errors on a malloc, handling the error in these cases isn't close to a solution. No language or stdlib can solve this problem 100%. At a baremetal level, it helps having this, but you're probably better off not using the stdlib anyway.
  - quotemstr 11y ago
    Bullshit. Every part of the C++ standard library (which is _much_ bigger than Rust's) can gracefully propagate resource exhaustion errors, including memory allocation failure, to callers. That Rust can't is a design flaw. > there are situations where your application will just abort randomly That's sure as hell not the case in any environment I choose to use.
    - Manishearth 11y ago
      > can gracefully propagate resource exhaustion errors, including memory allocation failure, to callers only on malloc. If the kernel overcommits, your process will abort when you try to use the memory, possibly way after the malloc and there's nothing you can do about it. That's the point being made here. > That Rust can't is a design flaw. (This is false, see Steve's reply above about this)
      - quotemstr 11y ago
        The world is not Linux. I happen to believe that believe that overcommit in the Linux kernel is a disgrace. It is, however, at least possible to disable it. It's not possible to retroactively add real exceptions to Rust, or to change the signature of all memory-allocation function to return Result. Rust is supposed to be a general-purpose systems programming language, not a Linux programming language. Windows does not overcommit. A correctly configured Linux system does not overcommit. Lost of embedded systems don't (and can't) overcommit. Are you saying all of these people should avoid Rust's standard library? > > That Rust can't is a design flaw. > (This is false, see Steve's reply above about this) It's clear that my opinion differs from that of many Rust developers and users. I still think I'm correct, that these developers and users are misguided, and that as Rust attempts to fill more niches, experience will show that my position is the correct one. All I can say is that I personally will not use any language that bakes cornucopian assumptions about memory baked into its core library. I know that you say that it's possible to just avoid stdlib --- but the temptation to use it will be irresistible, and once somebody succumbs to the temptation, the entire program is now capable of aborting irrecoverably. I will stick with languages . Modern C++ is safe and expressive enough, and it correctly reacts to resource exhaustion.
- bzbarsky 11y ago
  > On Linux malloc never fails actually. Not true. For example, it can fail if there is no big enough chunk of virtual address space available (i.e. your heap is fragmented enough and your attempted allocation is big enough). I've even seen 64-bit processes manage to do this, my mmapping lots of multi-GB things at once and then trying to do large allocations.
- mtanski 11y ago
  You're correct in the way that it's default behavior, but the behavior can be turned off. Also, if you're using cgroups (or anything that leverages that like containers) you can put a limit on the memory resource and then malloc will fail. Which is common enough, and only becoming more common as people are collocating a lot of disparate workloads on nodes (using the kurbenets scheduler).
quotemstr 11y ago
No custom allocator can make a task that fails to allocate gracefully report an error. Rust's error handling design is just terrible, and mostly a consequence of eschewing exceptions. Had Rust opted for exceptions, it'd be a much better, and actually usable, language. Rust's terribly error-handling strategy is the chief reason not to use it.
- steveklabnik 11y ago
  > No custom allocator can make the task gracefully report failure > instead of panicing. So, first of all, "custom allocators" means two things: * overloading the allocator that's used by liballoc, and the crates that depend on it, like libstd * other allocators entirely The first is described here: https://doc.rust-lang.org/book/custom-allocators.html https://doc.rust-lang.org/book/custom-allocators.html And the second is still in RFCs: https://github.com/rust-lang/rfcs/pull/1398 https://github.com/rust-lang/rfcs/pull/1398 Both of these things are not yet stable. The second does, in fact, give you the ability to return an error code, by returning a Result. However, on top of that, I don't see how > mostly a consequence of eschewing exceptions. and > No custom allocator can make the task gracefully report failure > instead of panicing. Work together. Or rather, why is panic-ing bad, but an exception good?
  - deleted 11y ago
    [deleted]
  - quotemstr 11y ago
    > why is panic-ing bad, but an exception good? Because the Rust people don't believe in making "catch" a first-class primitive in the language, and in fact, fully support a runtime option to turn all panics into aborts. Even if abort-on-panic were to be killed as a legal mode of operation, and even if the stigma were to be removed from std::panic::recover, we'd still be left with a language with two error handling strategies and endless programmer confusion over which to use. Rust's designers have done permanent damage to the language by not making exceptions the primary error reporting mechanism available to programmers, and it's not a mistake they can undo now.

losvedir 11y ago

That's not a property of rust the language, but the standard library. I believe you could use other allocators that behave differently. In any case, the standard library panics on OOM, and panics are described at the bottom of the linked page.

kzrdude 11y ago
calls abort(), not panic!(). This is important since unwinding does not happen with the former.