6 ms·

The "but the OOM killer!" argument has been a disaster for Rust, and keeps derailing all design discussions. • Linux is not the only OS in the world. • Even o

by pornel 6y ago

The "but the OOM killer!" argument has been a disaster for Rust, and keeps derailing all design discussions. • Linux is not the only OS in the world. • Even on Linux you have containers/cgroups that can impose hard limits. • Platforms without virtual memory are also an important target for Rust. • On 32-bit platforms you can run out of address space before you run out of RAM. • Regardless of what the OS does, the application may still want to impose its own internal limit (e.g. https://lib.rs/cap https://lib.rs/cap) to avoid being OOM-killed or swap death. People keep telling me how Linux never runs out of memory, while I'm currently firefighting a torrent of coredumps caused by Rust's self-own on OOM that actually happens. I generally love Rust, but its OOM handling is awful, and the "but the OOM killer" nonsense is to blame for stalling the absolutely critical fixes it urgently needs.

teknopaul 6y ago
word
varajelle 6y ago
The overcommit is not the only argument. The second argument is that complex programs that need to handle OOM properly will be full of complex untested error paths. How should one handle a OOM error If one can't allocate some temporary node during an algorithm and how to recover without leaving a corrupted state? That's not a question I want to answer for almost every function call. That's why the rust standard library data structures don't handle OOM. To make it easier to use for the most common case. And for the cases where the OOM handling is required, one can simply use data structures for it, which is what Linux will do (problem solved) Also compare that to the case of mutex poisoning which can return an error on each unlock, and which is now considered a mistake because in practice, nobody want to handle a poisoned mutex error.
- pornel 6y ago
  I argue it's a rare-enough problem: • Statistically, likelihood of failure is proportional to the allocation size. You'll most often fail to allocate a very large Vec, and then only need to allocate a few bytes for a string that says "error!". I'm using https://lib.rs/fallible_collections https://lib.rs/fallible_collections and just a few strategically placed try_reserve has brought down my coredumps by 99%. • Panic on OOM would unwind, and unwinding tends to free memory rather than allocate anything. The standard library should reserve/preallocate enough memory to be able to start unwinding. • Rust is good at avoiding temporary allocations and relies on the stack a lot. You don't have to use fancy error-handling libraries that collect backtrace on every error. Error handling with enums doesn't touch the heap at all. • Even if everything fails, and you get OOM during OOM handling, that's fair, and still better than abort every time. Note that Rust currently doesn't handle OOM by panicking. It unconditionally aborts the whole process. Libstd could switch to panics without API changes, and that would make OOM handling possible via catch_unwind.
  - volta83 6y ago
    > Panic on OOM would unwind, OR abort the process. Panics are for fatal irrecoverable errors, not for recoverable errors.
- varajelle 6y ago
  For example of the problem that the API would have if it wanted to handle OOM: Box, String or Vec, the most basic types in the rust alloc crate, couldn't implement Clone, because clone() can't fail, and you need to allocate memory to clone these data structures. As a result, lots of generic algorithms that relies on on their type to be clonable wouldn't work, and most user types could not #[derive(Clone)] anymore.
  - volta83 6y ago
    Hows that' a problem? Provide `TryClone`, there, problem solved.
  - loeg 6y ago
    I agree, it doesn't seem like these types can actually implement Clone if allocation can fail.
  - astrobe_ 6y ago
    So, if I get this right, this means that the "just don't use the alloc crate" argument given somewhere else here is not practical ?
    - pixel_fcker 6y ago
      No not at all.
      - astrobe_ 6y ago
        Because ...? I mean, from the perspective of an outsider reading that you "can't use strings" if there's a risk of allocation failure is a bit worrying.
        pixel_fcker 6y ago
        Because you just don’t use the types in the alloc crate, or you use the (only partially implemented) APIs that return a Result<> instead of panicking on allocation failure. For that kind of low-level work I’d probably do the former and implement my own data structures, which is what you do with no_std currently. As others have said there’s nothing in Rust the language which assumes allocations always succeed. The standard library made the choice to panic on allocation failure, which was the right choice in order to make an ergonomic API that’s suitable for higher-level work. If you’re not using the standard library (which currently includes people on eg embedded targets) then you can handle allocation any way you like. And if you’re writing drivers for the kernel you’d assume you’re ok handling that. GP was pointing out that the Clone trait doesn’t allow for returning a Result, so you can’t use that trait to clone anything that allocates. But that’s not really a big deal since you can just make your own TryClone that does.
dathinab 6y ago
> but the OOM killer!" [..] disaster for Rust [...] Only on HN discussions (the disaster part) ;=) (Through it's is a broken discussion, OOM on Linux is broken, but it becomes increasingly "un-"broken and like you said other platform exists, too). Anyway there is no disaster as: Rust does support handling memory allocations gracefully, the allocator API defaults to this!! Just various types defined in [lib]std (or precise [lib]alloc) default in there default methods to calling the (replaceable!) allocation error hook. This is due to ergonomics, there is no sane/ergonomic way for "common non embedded/kernel programming or non special purpose use-cases" to make literally every method which potentially allocates return a result. And outside of such special purpose cases you can always recover from panics (i.e. outside of special purpose cases panic=abort is a anti-pattern). Still, even in std, methods like `try_reserve` exists and are used, e.g. by serde deserializers to make sure to gracefully fail if the user tries to allocate a 10GiB array (because compressed data formats ;-) ). BUT for the kernel all this doesn't matter as it is very unlikely it will use [lib]std or [lib]alloc. So they can default to just always implement their types in a way which doesn't panic on memory allocation. And wrt. to panic in other (basically guaranteed to be a bug cases) panic=BUG() (ther kernel BUG() macro) can be done. I'm more worried about the float/128bit integer thing he mentioned, as I don't know anything about it. I assume it's not just a miss understanding of overflow checks and other debug assertions the kernel likely will disable for non-debug builds?
- littlestymaar 6y ago
  > I'm more worried about the float/128bit integer thing he mentioned, as I don't know anything about it. I assume it's not just a miss understanding of overflow checks and other debug assertions the kernel likely will disable for non-debug builds? Idk about 128bits integer, but the floating point question has been discussed on Rust's subreddit[1]: > Normally, the kernel leaves the floating-point state of the CPU in whatever state the userspace process left it in, so that you can do a quick system call without having to save and restore all that state. It's possible to use floating point in the kernel with a great deal of care, but you have to notify the kernel that you're doing so, so that it can save the userspace floating-point state. > So, it'd be helpful if by default Rust didn't allow use of floating-point, and then you could opt in to allowing it for specific code. And from Linus[2] > In other words: it's still very much a special case, and if the question was "can I just use FP in the kernel" then the answer is still a resounding NO, since other architectures may not support it AT ALL. [1]: https://www.reddit.com/r/rust/comments/mqxr1a/rfc_rust_support_for_linux_kernel/gujty1u/?context=3 https://www.reddit.com/r/rust/comments/mqxr1a/rfc_rust_suppo... [2]: (from the same reddit thread) https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47ir1bZGAK/comp/linux/kernel_fp.html https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47i...
simias 6y ago
I agree that the OOM killer is frankly irrelevant in this discussion (and a bad idea in the first place IMO, cue the "airplane company deciding which passenger to throw out the plane" argument). But on the other hand in my experience with C and C++, languages which do allow explicit OOM handling, most applications either just crash when that happen (the `xmalloc` route) or attempt to recover but often really can't do much. It goes even beyond programming languages. Very few environments deal nicely with near-OOM conditions. My Linux desktop becomes effectively unusable if I start aggressively swapping. And technically at this point as long as I have swap I'm not literally out of memory. My general point is not that I don't think Rust would benefit from allowing the developer to explicitly handle OOM conditions, it's more that the vast majority of applications are not written to gracefully degrade in these conditions anyway, and if the only handling you do is effectively `alert("out of memory!"); exit(1);` it's really not worth bothering with it. On the other hand for the minority of applications that want to do something more meaningful on OOM you probably want to take the time to design some ergonomic system that won't litter the code with boilerplate, i.e. you probably don't want every single Vec and String and basically every single facet of the standard API that can allocate behind the scenes to return a `Result<_>`. I'm sure there's something to do, but I think the Rust devs are right not to rush it given that there are so many other features left to implement.
- ff317 6y ago
  > if the only handling you do is effectively `alert("out of memory!"); exit(1);` it's really not worth bothering with it. I'd argue that even this is worth it, for most software. At least then you get a clean exit and an obvious problem. The alternative is to not detect it and let the code walk off into undefined behaviors and potentially subtle bugs that can harm data and/or break security.
  - loeg 6y ago
    Yeah, explicit OOM exit beats random NULL deref any day. The former can be analyzed by an L2 tech out of a logfile. The latter requires spinning up GDB to figure out where NULL came from.
silon42 6y ago
It would be really helpful, if the OOM killer should be adjustable to kill non-overcommitting programs last (kill any with overcommit first). Then programs could be slowly fixed/improved.

by pornel 6y ago

teknopaul 6y ago

word

varajelle 6y ago

The overcommit is not the only argument. The second argument is that complex programs that need to handle OOM properly will be full of complex untested error paths. How should one handle a OOM error If one can't allocate some temporary node during an algorithm and how to recover without leaving a corrupted state? That's not a question I want to answer for almost every function call. That's why the rust standard library data structures don't handle OOM. To make it easier to use for the most common case. And for the cases where the OOM handling is required, one can simply use data structures for it, which is what Linux will do (problem solved) Also compare that to the case of mutex poisoning which can return an error on each unlock, and which is now considered a mistake because in practice, nobody want to handle a poisoned mutex error.

pornel 6y ago
I argue it's a rare-enough problem: • Statistically, likelihood of failure is proportional to the allocation size. You'll most often fail to allocate a very large Vec, and then only need to allocate a few bytes for a string that says "error!". I'm using https://lib.rs/fallible_collections https://lib.rs/fallible_collections and just a few strategically placed try_reserve has brought down my coredumps by 99%. • Panic on OOM would unwind, and unwinding tends to free memory rather than allocate anything. The standard library should reserve/preallocate enough memory to be able to start unwinding. • Rust is good at avoiding temporary allocations and relies on the stack a lot. You don't have to use fancy error-handling libraries that collect backtrace on every error. Error handling with enums doesn't touch the heap at all. • Even if everything fails, and you get OOM during OOM handling, that's fair, and still better than abort every time. Note that Rust currently doesn't handle OOM by panicking. It unconditionally aborts the whole process. Libstd could switch to panics without API changes, and that would make OOM handling possible via catch_unwind.
- volta83 6y ago
  > Panic on OOM would unwind, OR abort the process. Panics are for fatal irrecoverable errors, not for recoverable errors.
varajelle 6y ago
For example of the problem that the API would have if it wanted to handle OOM: Box, String or Vec, the most basic types in the rust alloc crate, couldn't implement Clone, because clone() can't fail, and you need to allocate memory to clone these data structures. As a result, lots of generic algorithms that relies on on their type to be clonable wouldn't work, and most user types could not #[derive(Clone)] anymore.
- volta83 6y ago
  Hows that' a problem? Provide `TryClone`, there, problem solved.
- loeg 6y ago
  I agree, it doesn't seem like these types can actually implement Clone if allocation can fail.
- astrobe_ 6y ago
  So, if I get this right, this means that the "just don't use the alloc crate" argument given somewhere else here is not practical ?
  - pixel_fcker 6y ago
    No not at all.
    - astrobe_ 6y ago
      Because ...? I mean, from the perspective of an outsider reading that you "can't use strings" if there's a risk of allocation failure is a bit worrying.
      - pixel_fcker 6y ago
        Because you just don’t use the types in the alloc crate, or you use the (only partially implemented) APIs that return a Result<> instead of panicking on allocation failure. For that kind of low-level work I’d probably do the former and implement my own data structures, which is what you do with no_std currently. As others have said there’s nothing in Rust the language which assumes allocations always succeed. The standard library made the choice to panic on allocation failure, which was the right choice in order to make an ergonomic API that’s suitable for higher-level work. If you’re not using the standard library (which currently includes people on eg embedded targets) then you can handle allocation any way you like. And if you’re writing drivers for the kernel you’d assume you’re ok handling that. GP was pointing out that the Clone trait doesn’t allow for returning a Result, so you can’t use that trait to clone anything that allocates. But that’s not really a big deal since you can just make your own TryClone that does.

dathinab 6y ago

> but the OOM killer!" [..] disaster for Rust [...] Only on HN discussions (the disaster part) ;=) (Through it's is a broken discussion, OOM on Linux is broken, but it becomes increasingly "un-"broken and like you said other platform exists, too). Anyway there is no disaster as: Rust does support handling memory allocations gracefully, the allocator API defaults to this!! Just various types defined in [lib]std (or precise [lib]alloc) default in there default methods to calling the (replaceable!) allocation error hook. This is due to ergonomics, there is no sane/ergonomic way for "common non embedded/kernel programming or non special purpose use-cases" to make literally every method which potentially allocates return a result. And outside of such special purpose cases you can always recover from panics (i.e. outside of special purpose cases panic=abort is a anti-pattern). Still, even in std, methods like `try_reserve` exists and are used, e.g. by serde deserializers to make sure to gracefully fail if the user tries to allocate a 10GiB array (because compressed data formats ;-) ). BUT for the kernel all this doesn't matter as it is very unlikely it will use [lib]std or [lib]alloc. So they can default to just always implement their types in a way which doesn't panic on memory allocation. And wrt. to panic in other (basically guaranteed to be a bug cases) panic=BUG() (ther kernel BUG() macro) can be done. I'm more worried about the float/128bit integer thing he mentioned, as I don't know anything about it. I assume it's not just a miss understanding of overflow checks and other debug assertions the kernel likely will disable for non-debug builds?

littlestymaar 6y ago
> I'm more worried about the float/128bit integer thing he mentioned, as I don't know anything about it. I assume it's not just a miss understanding of overflow checks and other debug assertions the kernel likely will disable for non-debug builds? Idk about 128bits integer, but the floating point question has been discussed on Rust's subreddit[1]: > Normally, the kernel leaves the floating-point state of the CPU in whatever state the userspace process left it in, so that you can do a quick system call without having to save and restore all that state. It's possible to use floating point in the kernel with a great deal of care, but you have to notify the kernel that you're doing so, so that it can save the userspace floating-point state. > So, it'd be helpful if by default Rust didn't allow use of floating-point, and then you could opt in to allowing it for specific code. And from Linus[2] > In other words: it's still very much a special case, and if the question was "can I just use FP in the kernel" then the answer is still a resounding NO, since other architectures may not support it AT ALL. [1]: https://www.reddit.com/r/rust/comments/mqxr1a/rfc_rust_support_for_linux_kernel/gujty1u/?context=3 https://www.reddit.com/r/rust/comments/mqxr1a/rfc_rust_suppo... [2]: (from the same reddit thread) https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47ir1bZGAK/comp/linux/kernel_fp.html https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47i...

simias 6y ago

I agree that the OOM killer is frankly irrelevant in this discussion (and a bad idea in the first place IMO, cue the "airplane company deciding which passenger to throw out the plane" argument). But on the other hand in my experience with C and C++, languages which do allow explicit OOM handling, most applications either just crash when that happen (the `xmalloc` route) or attempt to recover but often really can't do much. It goes even beyond programming languages. Very few environments deal nicely with near-OOM conditions. My Linux desktop becomes effectively unusable if I start aggressively swapping. And technically at this point as long as I have swap I'm not literally out of memory. My general point is not that I don't think Rust would benefit from allowing the developer to explicitly handle OOM conditions, it's more that the vast majority of applications are not written to gracefully degrade in these conditions anyway, and if the only handling you do is effectively `alert("out of memory!"); exit(1);` it's really not worth bothering with it. On the other hand for the minority of applications that want to do something more meaningful on OOM you probably want to take the time to design some ergonomic system that won't litter the code with boilerplate, i.e. you probably don't want every single Vec and String and basically every single facet of the standard API that can allocate behind the scenes to return a `Result<_>`. I'm sure there's something to do, but I think the Rust devs are right not to rush it given that there are so many other features left to implement.

ff317 6y ago
> if the only handling you do is effectively `alert("out of memory!"); exit(1);` it's really not worth bothering with it. I'd argue that even this is worth it, for most software. At least then you get a clean exit and an obvious problem. The alternative is to not detect it and let the code walk off into undefined behaviors and potentially subtle bugs that can harm data and/or break security.
- loeg 6y ago
  Yeah, explicit OOM exit beats random NULL deref any day. The former can be analyzed by an L2 tech out of a logfile. The latter requires spinning up GDB to figure out where NULL came from.

silon42 6y ago

It would be really helpful, if the OOM killer should be adjustable to kill non-overcommitting programs last (kill any with overcommit first). Then programs could be slowly fixed/improved.