18 ms·

As usual HN comments react to the headline, without reading the content. A lot of modern userspace code, including Rust code in the standard library, thinks th

As usual HN comments react to the headline, without reading the content. A lot of modern userspace code, including Rust code in the standard library, thinks that invariant failures (AKA "programmer errors") should cause some sort of assertion failure or crash (Rust or Go `panic`, C/C++ `assert`, etc). In the kernel, claims Linus, failing loudly is worse than trying to keep going because failing would also kill the failure reporting mechanisms. He advocates for a sort of soft-failure, where the code tells you you're entering unknown territory and then goes ahead and does whatever. Maybe it crashes later, maybe it returns the wrong answer, who knows, the only thing it won't do is halt the kernel at the point the error was detected. Think of the following Rust API for an array, which needs to be able to handle the case of a user reading an index outside its bounds: struct Array<T> { ... } impl<T> Array<T> { fn len(&self) -> usize; // if idx >= len, panic fn get_or_panic(&self, idx: usize) -> T; // if idx >= len, return None fn get_or_none(&self, idx: usize) -> Option<T>; // if idx >= len, print a stack trace and return // who knows what unsafe fn get_or_undefined(&self, idx: usize) -> T; } The first two are safe by the Rust definition, because they can't cause memory-unsafe behavior. The second two are safe by the Linus/Linux definition, because they won't cause a kernel panic. If you have to choose between #1 and #3, Linus is putting his foot down and saying that the kernel's answer is #3.

ChrisSD 4y ago
That's all very well but the Rust-in-Linux advocates are advocating for #2 and fully agree with Linus on #1. So attacking #1 is attacking straw.
layer8 4y ago
Please correct me if I’m wrong, but Rust also has no built-in mechanism to statically determine “this code won’t ever panic”, and thus with regards to Linux kernel requirements isn’t safer in that aspect than C. To the contrary, Rust is arguably less safe in that aspect than C, due to the general Rust practice of panicking upon unexpected conditions.
- pornel 4y ago
  Lack of a non-hacky no-panic guarantee is a pain. That would be like a no-segfault guarantee in C. But Rust's situation is still safer, because Rust can typically prevent more errors from ever becoming a run-time issue, e.g. you may not even need to use array indexing at all if you use iterators. You have a guarantee that references are never NULL, so you don't risk nullptr crash, etc. Rust panics are safer, because they reliably happen instead of an actually unsafe operation. Mitigations in C are usually best-effort and you may be lucky/unlucky to silently corrupt memory. Panics are a problem for uptime, but not for safety (in the sense they're not exploitable for more than DoS). In the long term crashing loud and clear may be better for reliability. You shake out all the bugs instead of having latent issues that corrupt your data.
  - layer8 4y ago
    One has to be careful about words. When Rust (or Linux) is used in (say) a vehicle or in a nuclear power plant, panicking certainly has immediate safety implications.
    - pornel 4y ago
      But it can still be safer - e.g. a panic can trigger an emergency stop instead of silently overwriting the "go full throttle" variable.
      - mike_hock 4y ago
        Yes, or jumping to the "emergency stop" routine can instead trigger "go full throttle" because the jump address has been corrupted. Or in an actual vehicle, the "emergency stop" (if that means just stomping on the brakes) can flip the car and kill its passengers.
        stjohnswarts 4y ago
        It's about the odds here. Nothing is 100% safe. Independent systems almost always provide backup safety incase the OS/embedded system fails. Thing like overcurrent detector, brown out detectors, speed governors, etc in case code does something as a result of running corrupted (or something similarly awful)
lucasyvas 4y ago
Then Linus is wrong because the unsafe keyword has nothing to do with no-panic guarantees? Unsafe correlates with memory safety / UB, so using it in a different way in the kernel would be flat out wrong. The language determines the definition of its constructs, not the software being written with it. Edit: It's worth mentioning that while I think he is wrong, I think it's symptomatic of there not being a keyword/designation in Rust to express what Linus is trying to say. I would completely oppose misusing the unsafe keyword since it has negative downstream effects on all future dependency crates, where it's not clear what characteristics "unsafe" refers to which causes a split. So maybe they need to just discuss a different way to label these for now and agree to improve it later.
titzer 4y ago
The way to handle this is split up kernel work into fail-able tasks [1]. When a safety check (like array OOB) occurs, it unwinds the stack up to the start of the task, and the task fails. Linus sounds so ignorant in this comment. As if no one else thought of writing safety-critical systems in a language that had dynamic errors, and that dynamic errors are going to bring the whole system down or turn it into a brick. No way! Errors don't have to be full-blown exceptions with all that rigamarole, but silently continuing with corruption is utter madness and in 2022 Linus should feel embarrassed for advocating such a backwards view. [1] This works for Erlang. Not everything needs to be a shared-nothing actor, to be sure, but failing a whole task is about the right granularity to allow reasoning about the system. E.g. a few dozen to a few hundred types of tasks or processes seems about right.
- jmillikin 4y ago
  I think Linus's response would be that those failable tasks are called "processes", and the low-level supervisor that starts + monitors them is the kernel. If you have code that might fail and restart, it belongs in userspace. If you want to run an Erlang-style distributed system in the kernel then that's an interesting research project, but it isn't where Linux is today. You'd be better off starting with SeL4 or Fuchsia.
  - titzer 4y ago
    40 years of microkernels, of which I know Linus is aware of, beg to differ. Maybe Linus's extreme opposition to microkernels, ostensibly because they have historically a little lower performance--I dunno--but my comment should not be read as "yes, you must have a microkernel". There are cheaper fault isolation mechanisms than full-blown separate processes. Just having basic stack unwinding and failing a task would be a start.
    - pca006132 4y ago
      I think it might be a bit more complicated than that, considering you can have static data and unwinding the stack will not reset those states. I guess you still need some sort of task level abstraction and reset all the data for that task when unwinding from it. Btw, do we need stack unwinding or can we just do a sjlj?
echelon 4y ago
Use #2 everywhere you're not doing C FFI. Both #1 and #3 are gross and wrong.
EdSchouten 4y ago
The policy of ‘oopsing’ and limping on is, in my opinion, literally one of Linux’s worst features. It has bitten me in various cases: - Remember when Linux had that caused the kernel to partially crash and eat 100% CPU due to some bug in the leap second application code? That caused a >1MW spike in power usage at Hetzner at the time. That must have been >1GW globally. Many people didn’t notice it immediately, so it must have taken weeks before everyone rebooted. - I’ve personally run into issues where not crashing caused Linux to go on and eat my file system. On any Linux server I maintain, I always toggle those sysctls that cause the kernel to panic on oops, and reboot on panic.
- xyzzy_plugh 4y ago
  Right, the fact that you can toggle those sysctls is sort of the point. There are plenty of environments where running at all is better than not, hardware watchdogs can solve for unresponsiveness. TFA is about making it possible for the kernel to decide what to do, rather than exploding on the spot, which is terrible.
- deleted 4y ago
  [deleted]
- mike_hock 4y ago
  So instead of a power spike, we'd have had a major internet outage across the world, across the entire industry and beyond, probably, if everyone had panicked on oops. The blame really lies with people not monitoring their systems. As you said, you have the option to reboot on panic, but Linus is absolutely not wrong that this size does not fit all. What about a medical procedure that WILL kill the patient if interrupted? What about life support in space? Hitting an assert in those kinds of systems is a very bad place to be, but an automatic halt is worse than at least giving the people involved a CHANCE to try and get to a state where it's safe to take the system offline and restart it.
  - Tomte 4y ago
    You won't see Linux in those fields. I'm aware of a project by OSADL to qualify Linux for SIL 2, but your examples are way beyond that.
whatshisface 4y ago
#3 could cause a security problem. I don't think you'd find it in the Linux kernel - it would let attackers read arbitrary kernel memory. That's one of safe Rust's strongest features: it will not compile a direct memory access vulnerability.
hedgehog 4y ago
As I read it the issue is a little bit deeper, start with the context here and read down the thread: https://lkml.org/lkml/2022/9/19/640 https://lkml.org/lkml/2022/9/19/640 Get at least down to here: https://lkml.org/lkml/2022/9/20/1342 https://lkml.org/lkml/2022/9/20/1342 What Linus seems to be getting at is that there are many varying contextual restrictions on what code can do in different parts of the kernel, that Filho etc appear to be attempting to hide that complexity using language features, and that in his opinion it is not workable to fit kernel APIs into Rust's common definition of a single kind of "safe" code. All of this makes sense, in user land you don't normally have to care about things like whether different functional units are turned on or off, how interrupts are set up, etc, but in kernel you do. I'm not sure if Rust's macros and type system will allow solving the problem as Linus frames it but it seems like a worthy target & will be interesting to watch.
3a2d29 4y ago
I might be missing something here. So I understand panic! will essentially crash the kernel, that makes sense to me as a problem. But wouldn't reading outside an array bounds also possibly do that? It coudl seg fault which is essentially the same thing. Is it that reading out of bounds on an array isn't guaranteed to crash everything while a panic always will?
- fdr 4y ago
  What segment? You don't have mapped memory to overrun in the kernel, so that array would have to be in a very special spot to cause a bus error in such a situation. Also, even in user space, overrunning an array is far from guaranteed in trying to address an unmapped page...in fact, it often doesn't, since mapping memory with gaps for each array is prohibitively expensive (though, debugging aides like Electric Fence rely on exactly this mechanism)
oconnor663 4y ago
I don't have any experience with this project, but I know a lot of panics in my Rust code look like this (you probably know this already, just setting up a question): fn foo<T>() -> Option<T> { // Oops, something went wrong and we don't have a T. None } fn bar<T>() -> T { if let Some(t) = foo() { t } else { // This could've been an `unwrap`; just being explicit here panic!("oh no!"); } } A panic in this case is exactly like an exception in that the function that's failing doesn't need to come up with a return value. Unwinding happens instead of returning anything. But if I was writing `bar` and I was trying to follow a policy like "never unwind, always return something", I'd be in a real pickle, because the way the underlying `foo` function is designed, there aren't any T's sitting around for me to return. Should I conjure one out of thin air / uninitialized memory? What does the kernel do in situations like this? I guess the ideal solution is making `bar` return `Option<T>` instead of `T`, but I don't imagine that's always possible?
- strictfp 4y ago
  The harsh truth is that you need to think about every single case of failure, and decide what to do when things go south. If you look at how POSIX does it, pretty much every single function has error codes, signaling everything from lost connections, to running out of memory, entropy or whatnot. Failures are hard to abstract away. Unless you have some real viable fallback to use, you're going to have to tell the user that something went wrong and leave it up to them to decide what the application can best do in this case. So in your case, I would return Result<T>, and encode the errors in that. Simply expose the problem to the caller.
- jmillikin 4y ago
  Couple options: 1. Have a constraint on T that lets you return some sort of placeholder. For example, if you've got an array of u8, maybe every read past the end of the array returns 0. fn bar<T: Default>() -> T { if let Some(t) = foo() { t } else { eprintln!("programmer error, foo() returned None!"); Default::default() } } 2. Return a `Option<T>` from bar, as you describe. 3. Return a `Result<T, BarError>`, where `BarError` is a struct or enum describing possible error conditions. #[non_exhaustive] enum BarError { FooIsNone, } fn bar<T>() -> Result<T, BarError> { if let Some(t) = foo() { Ok(t) } else { eprintln!("programmer error, foo() returned None!"); Err(BarError::FooIsNone) } }
zozbot234 4y ago
The problem with #3 is not really about C vs. Rust, it's about modern optimizing compilers (including GCC). A compiler is allowed to assume that UB simply won't happen, so it makes no guarantees whatsoever about what happens if UB is ever hit. It's not "return a random value", it really is "all bets are off". There is no guarantee that you manage to "limp along" in any reasonable sense, let alone report the failure. That's what "panic and recover" mechanisms are for, and yes even the ? operator in Rust can be seen as such a mechanism.
scoutt 4y ago
I think it goes beyond "panics". Linus is saying that Rust cannot guarantee safety under certain circumstances and that safety still depends on the order of fuctions called by the kernel module developer. Because some checks on some compilations will be disabled. "If you want to allocate memory, and you don't want to care about what context you are in, or whether you are holding spinlocks etc, then you damn well shouldn't be doing kernel programming. Not in C, and not in Rust. It really is that simple. Contexts like this ("I am in a critical region, I must not do memory allocation or use sleeping locks") is fundamental to kernel programming. It has nothing to do with the language, and everything to do with the problem space." https://lkml.org/lkml/2022/9/19/840 https://lkml.org/lkml/2022/9/19/840

ChrisSD 4y ago

That's all very well but the Rust-in-Linux advocates are advocating for #2 and fully agree with Linus on #1. So attacking #1 is attacking straw.

layer8 4y ago

Please correct me if I’m wrong, but Rust also has no built-in mechanism to statically determine “this code won’t ever panic”, and thus with regards to Linux kernel requirements isn’t safer in that aspect than C. To the contrary, Rust is arguably less safe in that aspect than C, due to the general Rust practice of panicking upon unexpected conditions.

pornel 4y ago
Lack of a non-hacky no-panic guarantee is a pain. That would be like a no-segfault guarantee in C. But Rust's situation is still safer, because Rust can typically prevent more errors from ever becoming a run-time issue, e.g. you may not even need to use array indexing at all if you use iterators. You have a guarantee that references are never NULL, so you don't risk nullptr crash, etc. Rust panics are safer, because they reliably happen instead of an actually unsafe operation. Mitigations in C are usually best-effort and you may be lucky/unlucky to silently corrupt memory. Panics are a problem for uptime, but not for safety (in the sense they're not exploitable for more than DoS). In the long term crashing loud and clear may be better for reliability. You shake out all the bugs instead of having latent issues that corrupt your data.
- layer8 4y ago
  One has to be careful about words. When Rust (or Linux) is used in (say) a vehicle or in a nuclear power plant, panicking certainly has immediate safety implications.
  - pornel 4y ago
    But it can still be safer - e.g. a panic can trigger an emergency stop instead of silently overwriting the "go full throttle" variable.
    - mike_hock 4y ago
      Yes, or jumping to the "emergency stop" routine can instead trigger "go full throttle" because the jump address has been corrupted. Or in an actual vehicle, the "emergency stop" (if that means just stomping on the brakes) can flip the car and kill its passengers.
      - stjohnswarts 4y ago
        It's about the odds here. Nothing is 100% safe. Independent systems almost always provide backup safety incase the OS/embedded system fails. Thing like overcurrent detector, brown out detectors, speed governors, etc in case code does something as a result of running corrupted (or something similarly awful)

lucasyvas 4y ago

Then Linus is wrong because the unsafe keyword has nothing to do with no-panic guarantees? Unsafe correlates with memory safety / UB, so using it in a different way in the kernel would be flat out wrong. The language determines the definition of its constructs, not the software being written with it. Edit: It's worth mentioning that while I think he is wrong, I think it's symptomatic of there not being a keyword/designation in Rust to express what Linus is trying to say. I would completely oppose misusing the unsafe keyword since it has negative downstream effects on all future dependency crates, where it's not clear what characteristics "unsafe" refers to which causes a split. So maybe they need to just discuss a different way to label these for now and agree to improve it later.

titzer 4y ago

The way to handle this is split up kernel work into fail-able tasks [1]. When a safety check (like array OOB) occurs, it unwinds the stack up to the start of the task, and the task fails. Linus sounds so ignorant in this comment. As if no one else thought of writing safety-critical systems in a language that had dynamic errors, and that dynamic errors are going to bring the whole system down or turn it into a brick. No way! Errors don't have to be full-blown exceptions with all that rigamarole, but silently continuing with corruption is utter madness and in 2022 Linus should feel embarrassed for advocating such a backwards view. [1] This works for Erlang. Not everything needs to be a shared-nothing actor, to be sure, but failing a whole task is about the right granularity to allow reasoning about the system. E.g. a few dozen to a few hundred types of tasks or processes seems about right.

jmillikin 4y ago
I think Linus's response would be that those failable tasks are called "processes", and the low-level supervisor that starts + monitors them is the kernel. If you have code that might fail and restart, it belongs in userspace. If you want to run an Erlang-style distributed system in the kernel then that's an interesting research project, but it isn't where Linux is today. You'd be better off starting with SeL4 or Fuchsia.
- titzer 4y ago
  40 years of microkernels, of which I know Linus is aware of, beg to differ. Maybe Linus's extreme opposition to microkernels, ostensibly because they have historically a little lower performance--I dunno--but my comment should not be read as "yes, you must have a microkernel". There are cheaper fault isolation mechanisms than full-blown separate processes. Just having basic stack unwinding and failing a task would be a start.
  - pca006132 4y ago
    I think it might be a bit more complicated than that, considering you can have static data and unwinding the stack will not reset those states. I guess you still need some sort of task level abstraction and reset all the data for that task when unwinding from it. Btw, do we need stack unwinding or can we just do a sjlj?

echelon 4y ago

Use #2 everywhere you're not doing C FFI. Both #1 and #3 are gross and wrong.

EdSchouten 4y ago

The policy of ‘oopsing’ and limping on is, in my opinion, literally one of Linux’s worst features. It has bitten me in various cases: - Remember when Linux had that caused the kernel to partially crash and eat 100% CPU due to some bug in the leap second application code? That caused a >1MW spike in power usage at Hetzner at the time. That must have been >1GW globally. Many people didn’t notice it immediately, so it must have taken weeks before everyone rebooted. - I’ve personally run into issues where not crashing caused Linux to go on and eat my file system. On any Linux server I maintain, I always toggle those sysctls that cause the kernel to panic on oops, and reboot on panic.

xyzzy_plugh 4y ago
Right, the fact that you can toggle those sysctls is sort of the point. There are plenty of environments where running at all is better than not, hardware watchdogs can solve for unresponsiveness. TFA is about making it possible for the kernel to decide what to do, rather than exploding on the spot, which is terrible.
deleted 4y ago
[deleted]
mike_hock 4y ago
So instead of a power spike, we'd have had a major internet outage across the world, across the entire industry and beyond, probably, if everyone had panicked on oops. The blame really lies with people not monitoring their systems. As you said, you have the option to reboot on panic, but Linus is absolutely not wrong that this size does not fit all. What about a medical procedure that WILL kill the patient if interrupted? What about life support in space? Hitting an assert in those kinds of systems is a very bad place to be, but an automatic halt is worse than at least giving the people involved a CHANCE to try and get to a state where it's safe to take the system offline and restart it.
- Tomte 4y ago
  You won't see Linux in those fields. I'm aware of a project by OSADL to qualify Linux for SIL 2, but your examples are way beyond that.

whatshisface 4y ago

#3 could cause a security problem. I don't think you'd find it in the Linux kernel - it would let attackers read arbitrary kernel memory. That's one of safe Rust's strongest features: it will not compile a direct memory access vulnerability.

hedgehog 4y ago

As I read it the issue is a little bit deeper, start with the context here and read down the thread: https://lkml.org/lkml/2022/9/19/640 https://lkml.org/lkml/2022/9/19/640 Get at least down to here: https://lkml.org/lkml/2022/9/20/1342 https://lkml.org/lkml/2022/9/20/1342 What Linus seems to be getting at is that there are many varying contextual restrictions on what code can do in different parts of the kernel, that Filho etc appear to be attempting to hide that complexity using language features, and that in his opinion it is not workable to fit kernel APIs into Rust's common definition of a single kind of "safe" code. All of this makes sense, in user land you don't normally have to care about things like whether different functional units are turned on or off, how interrupts are set up, etc, but in kernel you do. I'm not sure if Rust's macros and type system will allow solving the problem as Linus frames it but it seems like a worthy target & will be interesting to watch.

3a2d29 4y ago

I might be missing something here. So I understand panic! will essentially crash the kernel, that makes sense to me as a problem. But wouldn't reading outside an array bounds also possibly do that? It coudl seg fault which is essentially the same thing. Is it that reading out of bounds on an array isn't guaranteed to crash everything while a panic always will?

fdr 4y ago
What segment? You don't have mapped memory to overrun in the kernel, so that array would have to be in a very special spot to cause a bus error in such a situation. Also, even in user space, overrunning an array is far from guaranteed in trying to address an unmapped page...in fact, it often doesn't, since mapping memory with gaps for each array is prohibitively expensive (though, debugging aides like Electric Fence rely on exactly this mechanism)

oconnor663 4y ago

I don't have any experience with this project, but I know a lot of panics in my Rust code look like this (you probably know this already, just setting up a question): fn foo<T>() -> Option<T> { // Oops, something went wrong and we don't have a T. None } fn bar<T>() -> T { if let Some(t) = foo() { t } else { // This could've been an `unwrap`; just being explicit here panic!("oh no!"); } } A panic in this case is exactly like an exception in that the function that's failing doesn't need to come up with a return value. Unwinding happens instead of returning anything. But if I was writing `bar` and I was trying to follow a policy like "never unwind, always return something", I'd be in a real pickle, because the way the underlying `foo` function is designed, there aren't any T's sitting around for me to return. Should I conjure one out of thin air / uninitialized memory? What does the kernel do in situations like this? I guess the ideal solution is making `bar` return `Option<T>` instead of `T`, but I don't imagine that's always possible?

strictfp 4y ago
The harsh truth is that you need to think about every single case of failure, and decide what to do when things go south. If you look at how POSIX does it, pretty much every single function has error codes, signaling everything from lost connections, to running out of memory, entropy or whatnot. Failures are hard to abstract away. Unless you have some real viable fallback to use, you're going to have to tell the user that something went wrong and leave it up to them to decide what the application can best do in this case. So in your case, I would return Result<T>, and encode the errors in that. Simply expose the problem to the caller.
jmillikin 4y ago
Couple options: 1. Have a constraint on T that lets you return some sort of placeholder. For example, if you've got an array of u8, maybe every read past the end of the array returns 0. fn bar<T: Default>() -> T { if let Some(t) = foo() { t } else { eprintln!("programmer error, foo() returned None!"); Default::default() } } 2. Return a `Option<T>` from bar, as you describe. 3. Return a `Result<T, BarError>`, where `BarError` is a struct or enum describing possible error conditions. #[non_exhaustive] enum BarError { FooIsNone, } fn bar<T>() -> Result<T, BarError> { if let Some(t) = foo() { Ok(t) } else { eprintln!("programmer error, foo() returned None!"); Err(BarError::FooIsNone) } }

zozbot234 4y ago

The problem with #3 is not really about C vs. Rust, it's about modern optimizing compilers (including GCC). A compiler is allowed to assume that UB simply won't happen, so it makes no guarantees whatsoever about what happens if UB is ever hit. It's not "return a random value", it really is "all bets are off". There is no guarantee that you manage to "limp along" in any reasonable sense, let alone report the failure. That's what "panic and recover" mechanisms are for, and yes even the ? operator in Rust can be seen as such a mechanism.

scoutt 4y ago

I think it goes beyond "panics". Linus is saying that Rust cannot guarantee safety under certain circumstances and that safety still depends on the order of fuctions called by the kernel module developer. Because some checks on some compilations will be disabled. "If you want to allocate memory, and you don't want to care about what context you are in, or whether you are holding spinlocks etc, then you damn well shouldn't be doing kernel programming. Not in C, and not in Rust. It really is that simple. Contexts like this ("I am in a critical region, I must not do memory allocation or use sleeping locks") is fundamental to kernel programming. It has nothing to do with the language, and everything to do with the problem space." https://lkml.org/lkml/2022/9/19/840 https://lkml.org/lkml/2022/9/19/840