3 ms·

The overcommit is not the only argument. The second argument is that complex programs that need to handle OOM properly will be full of complex untested error p

The overcommit is not the only argument. The second argument is that complex programs that need to handle OOM properly will be full of complex untested error paths. How should one handle a OOM error If one can't allocate some temporary node during an algorithm and how to recover without leaving a corrupted state? That's not a question I want to answer for almost every function call. That's why the rust standard library data structures don't handle OOM. To make it easier to use for the most common case. And for the cases where the OOM handling is required, one can simply use data structures for it, which is what Linux will do (problem solved) Also compare that to the case of mutex poisoning which can return an error on each unlock, and which is now considered a mistake because in practice, nobody want to handle a poisoned mutex error.

pornel 6y ago
I argue it's a rare-enough problem: • Statistically, likelihood of failure is proportional to the allocation size. You'll most often fail to allocate a very large Vec, and then only need to allocate a few bytes for a string that says "error!". I'm using https://lib.rs/fallible_collections https://lib.rs/fallible_collections and just a few strategically placed try_reserve has brought down my coredumps by 99%. • Panic on OOM would unwind, and unwinding tends to free memory rather than allocate anything. The standard library should reserve/preallocate enough memory to be able to start unwinding. • Rust is good at avoiding temporary allocations and relies on the stack a lot. You don't have to use fancy error-handling libraries that collect backtrace on every error. Error handling with enums doesn't touch the heap at all. • Even if everything fails, and you get OOM during OOM handling, that's fair, and still better than abort every time. Note that Rust currently doesn't handle OOM by panicking. It unconditionally aborts the whole process. Libstd could switch to panics without API changes, and that would make OOM handling possible via catch_unwind.
- volta83 6y ago
  > Panic on OOM would unwind, OR abort the process. Panics are for fatal irrecoverable errors, not for recoverable errors.
varajelle 6y ago
For example of the problem that the API would have if it wanted to handle OOM: Box, String or Vec, the most basic types in the rust alloc crate, couldn't implement Clone, because clone() can't fail, and you need to allocate memory to clone these data structures. As a result, lots of generic algorithms that relies on on their type to be clonable wouldn't work, and most user types could not #[derive(Clone)] anymore.
- volta83 6y ago
  Hows that' a problem? Provide `TryClone`, there, problem solved.
- loeg 6y ago
  I agree, it doesn't seem like these types can actually implement Clone if allocation can fail.
- astrobe_ 6y ago
  So, if I get this right, this means that the "just don't use the alloc crate" argument given somewhere else here is not practical ?
  - pixel_fcker 6y ago
    No not at all.
    - astrobe_ 6y ago
      Because ...? I mean, from the perspective of an outsider reading that you "can't use strings" if there's a risk of allocation failure is a bit worrying.
      - pixel_fcker 6y ago
        Because you just don’t use the types in the alloc crate, or you use the (only partially implemented) APIs that return a Result<> instead of panicking on allocation failure. For that kind of low-level work I’d probably do the former and implement my own data structures, which is what you do with no_std currently. As others have said there’s nothing in Rust the language which assumes allocations always succeed. The standard library made the choice to panic on allocation failure, which was the right choice in order to make an ergonomic API that’s suitable for higher-level work. If you’re not using the standard library (which currently includes people on eg embedded targets) then you can handle allocation any way you like. And if you’re writing drivers for the kernel you’d assume you’re ok handling that. GP was pointing out that the Clone trait doesn’t allow for returning a Result, so you can’t use that trait to clone anything that allocates. But that’s not really a big deal since you can just make your own TryClone that does.

pornel 6y ago

I argue it's a rare-enough problem: • Statistically, likelihood of failure is proportional to the allocation size. You'll most often fail to allocate a very large Vec, and then only need to allocate a few bytes for a string that says "error!". I'm using https://lib.rs/fallible_collections https://lib.rs/fallible_collections and just a few strategically placed try_reserve has brought down my coredumps by 99%. • Panic on OOM would unwind, and unwinding tends to free memory rather than allocate anything. The standard library should reserve/preallocate enough memory to be able to start unwinding. • Rust is good at avoiding temporary allocations and relies on the stack a lot. You don't have to use fancy error-handling libraries that collect backtrace on every error. Error handling with enums doesn't touch the heap at all. • Even if everything fails, and you get OOM during OOM handling, that's fair, and still better than abort every time. Note that Rust currently doesn't handle OOM by panicking. It unconditionally aborts the whole process. Libstd could switch to panics without API changes, and that would make OOM handling possible via catch_unwind.

volta83 6y ago
> Panic on OOM would unwind, OR abort the process. Panics are for fatal irrecoverable errors, not for recoverable errors.

varajelle 6y ago

For example of the problem that the API would have if it wanted to handle OOM: Box, String or Vec, the most basic types in the rust alloc crate, couldn't implement Clone, because clone() can't fail, and you need to allocate memory to clone these data structures. As a result, lots of generic algorithms that relies on on their type to be clonable wouldn't work, and most user types could not #[derive(Clone)] anymore.

volta83 6y ago
Hows that' a problem? Provide `TryClone`, there, problem solved.
loeg 6y ago
I agree, it doesn't seem like these types can actually implement Clone if allocation can fail.
astrobe_ 6y ago
So, if I get this right, this means that the "just don't use the alloc crate" argument given somewhere else here is not practical ?
- pixel_fcker 6y ago
  No not at all.
  - astrobe_ 6y ago
    Because ...? I mean, from the perspective of an outsider reading that you "can't use strings" if there's a risk of allocation failure is a bit worrying.
    - pixel_fcker 6y ago
      Because you just don’t use the types in the alloc crate, or you use the (only partially implemented) APIs that return a Result<> instead of panicking on allocation failure. For that kind of low-level work I’d probably do the former and implement my own data structures, which is what you do with no_std currently. As others have said there’s nothing in Rust the language which assumes allocations always succeed. The standard library made the choice to panic on allocation failure, which was the right choice in order to make an ergonomic API that’s suitable for higher-level work. If you’re not using the standard library (which currently includes people on eg embedded targets) then you can handle allocation any way you like. And if you’re writing drivers for the kernel you’d assume you’re ok handling that. GP was pointing out that the Clone trait doesn’t allow for returning a Result, so you can’t use that trait to clone anything that allocates. But that’s not really a big deal since you can just make your own TryClone that does.