4 ms·

> In fact, for many applications malfunctioning is better than crashing — particulary in the embedded world where Rust wants to be present. Not a fact. Partic

by deng 11mo ago

> In fact, for many applications malfunctioning is better than crashing — particulary in the embedded world where Rust wants to be present. Not a fact. Particularly in the embedded world, crashing is preferable to malfunctioning, as many embedded devices control things that might hurt people, directly or indirectly. > If a pacemaker stops — telling a victim “but the memory was not corrupted in the crash” is a weak consolation. If a pacemaker suddenly starts firing at 200Hz, telling a victim "but at least it didn't crash" is a weak consolation. A stopping pacemaker is almost always preferable to a malfunctioning one, as most people with pacemakers still have sufficient natural rhythm to survive this for long enough to get help. > We actually had a recent Cloudflare outage caused by a crash on unwrap() function Please read the whole article. If the unwrap hadn't caused an exit, the process would've run out of memory, leading to a much less deterministic behavior which is much harder to diagnose and fix. I always prefer an early exit with a clear error instead of getting killed by the OOM reaper.

jmaker 11mo ago
Yeah, the blog post is a very confused write-up. I saw lots of similar posts on LinkedIn recently, with quite a lot of likes and echo chamber comments. It’s just hilarious how a narrative emerges that reinforces biases due to ignorance. There must be a name for that sort of fallacy. I love to write in Rust precisely because I can express failures more explicitly, it’s the transparency that wins here. I’d frame the issue Cloudflare had rather in the PR review and QA corner, maybe as some AI complacency. But it’s not a problem with Rust.
- dev_l1x_be 11mo ago
  I love the ub Go forces you to have explicit error handling ones the most.
- ameixaseca 11mo ago
  > There must be a name for that sort of fallacy. Motivated reasoning
tialaramex 11mo ago
Also stopping is a state which definitely might happen anyway and must be planned for. The pacemaker's wires can be dislodged or damaged, power sources fail, these things will stop the pacemaker regardless of how much you love the "just keep going" approach to software engineering. Which means medics have already thought about what they're going to do when this happens to a patient, and "The software failed" just goes on the same list as "10 year battery only last 8 years" in terms of undesirable but hardly impossible scenarios which constitute a medical emergency.
phaylon 11mo ago
Something else I usually don't see: A system hitting a fail-safe is a lot easier to detect and handle from the outside than one that just enters an unknown invalid state. Like, if the rule were "Always-Keep-Running" then hospital equipment power supplies wouldn't have circuit breakers that cut the power when something is wrong. But cutting power seems lot easier to detect for the backup power supply so it can fully take over.
- mlsu 11mo ago
  Engineer: “It crashed on an assert…” Pointy haired boss: “… well, what are you waiting for!? remove all the asserts so it doesn’t crash any more!”
  - jbn 11mo ago
    it's funny because I have seen the opposite. Engineer: "it crashed because it dereferenced a null pointer" boss: "add null pointer checks everywhere!" ... and because it used "if" instead of "assert", it made the null pointer arg a valid argument, making it a tolerable state of the running software, which displaced the locus of crashes far from the source of the issue. Moral of the story, use "assert" to make it crash as early as possible and debug THAT. You want to restrict the representable states in the software, not expand them by adding null checks everywhere.
imtringued 11mo ago
It's honestly mind boggling how people react to this. Rust turns unknown failures in C and C++ into known failures and suddenly the C/C++ people start caring about the failures, but attribute the failure to the new language, even though the same failures are secretly lurking in their C/C++ code bases. It's kind of like trying to silence a whistleblower. >Please read the whole article. If the unwrap hadn't caused an exit, the process would've run out of memory, leading to a much less deterministic behavior which is much harder to diagnose and fix. I always prefer an early exit with a clear error instead of getting killed by the OOM reaper. I am running into an undiagnosable CUDA "illegal memory access" problem in vLLM, a code base that is a mix of python and CUDA (via pytorch). At a certain load something appears to either overflow or corrupt the memory and vLLM restarts, which takes a minute, because it has to reload several dozens of GBs into memory and then rerun the CUDA graph optimizations. The pacemaker argument is complete nonsense, because the pacemaker must keep working even if it crashes. You can forcibly induce crashes into the pacemaker during testing and engineer it to restart fast enough that it hits its timing deadline anyway. Meanwhile a silent memory corruption could cause the pacemaker to enter an unknown state where the code that runs the pacemaker algorithm is overwritten and it simply stops working altogether. Having a known failure state is a thousand times more preferrable to an unknown number of unknown failure states. Critical sections (mutexes) and unsafe code has to be panic free (or at least panic safe) in Rust, so the concept of writing code without panics isn't exactly a niche concept in Rust. For every panic based feature, there is usually a panic free equivalent.
- byko3y 11mo ago
  >Rust turns unknown failures in C and C++ into known failures and suddenly the C/C++ people start caring about the failures I'm actually the one who promotes paranoidal assert-s everywhere. I do agree the original statement from the article is ambiguous, probably should have written something like "memory safety in Rust does not increase reliability". >The pacemaker argument is complete nonsense, because the pacemaker must keep working even if it crashes. You can forcibly induce crashes into the pacemaker during testing and engineer it to restart fast enough that it hits its timing deadline anyway. I'm not sure whether there is a deadlock-free modification of Rust — deadlock is not considered an undefined behavior in Rust.
QuiEgo 11mo ago
> Not a fact. Particularly in the embedded world, crashing is preferable to malfunctioning, as many embedded devices control things that might hurt people, directly or indirectly. Strong agree on this one, usually embedded systems are designed to "fail safe" on crash - the watchdog will trip, and hardware reset will put everything in a deterministic, known safe state. What you want, above all else, is not to fall into undefined behavior. That's the beauty of `unsafe`, it bounds UB into small boxes you know to test the hell out of.
byko3y 11mo ago
>Not a fact. Particularly in the embedded world, crashing is preferable to malfunctioning, as many embedded devices control things that might hurt people, directly or indirectly. It really depends on how deeply Turing you mechanism is. By being "Turing" I mean "the behavior is totally dependant on every single bit of previous information". For a reliable system turing-completeness is unacceptable for separate functions i.e. it should produce a correct result in a finite amount of time no matter what hapened in the past. Particulary, that's why modern real-time systems cannot be fit into Turing machine, because Turing machine has no interrupts. >If a pacemaker suddenly starts firing at 200Hz, telling a victim "but at least it didn't crash" is a weak consolation. A stopping pacemaker is almost always preferable to a malfunctioning one You almost make an excuse for general unreliability of programs. Mainstream C is unreliable, C++ is unreliable, Rust is unreliable. I can agree that Rust is not less reliable than C/C++, but it is definitely less reliable than some other language e.g. BEAM-based ones. I mean in Rust standard library some time ago I actually read "in these and these conditions the following code will deadlock. But deadlock is not an undefined behavior, so it's ok". The designers of Rust did not really try to support any kind of "recover and continue" way of functioning. Yes, you can catch the panic, but it will irreversibly poison some data.
yencabulator 11mo ago
A pacemaker crashing can restart, log a diagnostic, continue operating, and report the diagnostic remotely. (yes, really! Bluetooth to phone or to dedicated relay gadget.) A pacemaker in an unknown state goes forever undiagnosed.
oconnor663 10mo ago
> If the unwrap hadn't caused an exit, the process would've run out of memory It was trying to push an element into a full ArrayVec. The options are: - Blindly write off the end of the array. Obviously no one wants this, despite the decades of tradition... - Panic and unwind, as the program actually did in this case. - Return an error. Some folks assume that returning an error instead of unwinding would've been better. But my assumption is that the outcome would've been the same. I think the issue came up when loading configs, which isn't usually a recoverable situation. If you have an "invalid config error", you're probably just going to return that all the way up, which is effectively the same outcome as unwinding: your process exits with an error code. There are cases where the difference matters a lot, but I don't think this was one of them. The real gap seems to be why it took hours for folks to notice that this service was crash looping. That should normally be really prominent in alerts and dashboards. (Probably part of the story is that alerts were firing all over the place. Tough day at the office.)

by deng 11mo ago

jmaker 11mo ago

Yeah, the blog post is a very confused write-up. I saw lots of similar posts on LinkedIn recently, with quite a lot of likes and echo chamber comments. It’s just hilarious how a narrative emerges that reinforces biases due to ignorance. There must be a name for that sort of fallacy. I love to write in Rust precisely because I can express failures more explicitly, it’s the transparency that wins here. I’d frame the issue Cloudflare had rather in the PR review and QA corner, maybe as some AI complacency. But it’s not a problem with Rust.

dev_l1x_be 11mo ago
I love the ub Go forces you to have explicit error handling ones the most.
ameixaseca 11mo ago
> There must be a name for that sort of fallacy. Motivated reasoning

tialaramex 11mo ago

Also stopping is a state which definitely might happen anyway and must be planned for. The pacemaker's wires can be dislodged or damaged, power sources fail, these things will stop the pacemaker regardless of how much you love the "just keep going" approach to software engineering. Which means medics have already thought about what they're going to do when this happens to a patient, and "The software failed" just goes on the same list as "10 year battery only last 8 years" in terms of undesirable but hardly impossible scenarios which constitute a medical emergency.

phaylon 11mo ago

Something else I usually don't see: A system hitting a fail-safe is a lot easier to detect and handle from the outside than one that just enters an unknown invalid state. Like, if the rule were "Always-Keep-Running" then hospital equipment power supplies wouldn't have circuit breakers that cut the power when something is wrong. But cutting power seems lot easier to detect for the backup power supply so it can fully take over.

mlsu 11mo ago
Engineer: “It crashed on an assert…” Pointy haired boss: “… well, what are you waiting for!? remove all the asserts so it doesn’t crash any more!”
- jbn 11mo ago
  it's funny because I have seen the opposite. Engineer: "it crashed because it dereferenced a null pointer" boss: "add null pointer checks everywhere!" ... and because it used "if" instead of "assert", it made the null pointer arg a valid argument, making it a tolerable state of the running software, which displaced the locus of crashes far from the source of the issue. Moral of the story, use "assert" to make it crash as early as possible and debug THAT. You want to restrict the representable states in the software, not expand them by adding null checks everywhere.

imtringued 11mo ago

It's honestly mind boggling how people react to this. Rust turns unknown failures in C and C++ into known failures and suddenly the C/C++ people start caring about the failures, but attribute the failure to the new language, even though the same failures are secretly lurking in their C/C++ code bases. It's kind of like trying to silence a whistleblower. >Please read the whole article. If the unwrap hadn't caused an exit, the process would've run out of memory, leading to a much less deterministic behavior which is much harder to diagnose and fix. I always prefer an early exit with a clear error instead of getting killed by the OOM reaper. I am running into an undiagnosable CUDA "illegal memory access" problem in vLLM, a code base that is a mix of python and CUDA (via pytorch). At a certain load something appears to either overflow or corrupt the memory and vLLM restarts, which takes a minute, because it has to reload several dozens of GBs into memory and then rerun the CUDA graph optimizations. The pacemaker argument is complete nonsense, because the pacemaker must keep working even if it crashes. You can forcibly induce crashes into the pacemaker during testing and engineer it to restart fast enough that it hits its timing deadline anyway. Meanwhile a silent memory corruption could cause the pacemaker to enter an unknown state where the code that runs the pacemaker algorithm is overwritten and it simply stops working altogether. Having a known failure state is a thousand times more preferrable to an unknown number of unknown failure states. Critical sections (mutexes) and unsafe code has to be panic free (or at least panic safe) in Rust, so the concept of writing code without panics isn't exactly a niche concept in Rust. For every panic based feature, there is usually a panic free equivalent.

byko3y 11mo ago
>Rust turns unknown failures in C and C++ into known failures and suddenly the C/C++ people start caring about the failures I'm actually the one who promotes paranoidal assert-s everywhere. I do agree the original statement from the article is ambiguous, probably should have written something like "memory safety in Rust does not increase reliability". >The pacemaker argument is complete nonsense, because the pacemaker must keep working even if it crashes. You can forcibly induce crashes into the pacemaker during testing and engineer it to restart fast enough that it hits its timing deadline anyway. I'm not sure whether there is a deadlock-free modification of Rust — deadlock is not considered an undefined behavior in Rust.

QuiEgo 11mo ago

> Not a fact. Particularly in the embedded world, crashing is preferable to malfunctioning, as many embedded devices control things that might hurt people, directly or indirectly. Strong agree on this one, usually embedded systems are designed to "fail safe" on crash - the watchdog will trip, and hardware reset will put everything in a deterministic, known safe state. What you want, above all else, is not to fall into undefined behavior. That's the beauty of `unsafe`, it bounds UB into small boxes you know to test the hell out of.

byko3y 11mo ago

>Not a fact. Particularly in the embedded world, crashing is preferable to malfunctioning, as many embedded devices control things that might hurt people, directly or indirectly. It really depends on how deeply Turing you mechanism is. By being "Turing" I mean "the behavior is totally dependant on every single bit of previous information". For a reliable system turing-completeness is unacceptable for separate functions i.e. it should produce a correct result in a finite amount of time no matter what hapened in the past. Particulary, that's why modern real-time systems cannot be fit into Turing machine, because Turing machine has no interrupts. >If a pacemaker suddenly starts firing at 200Hz, telling a victim "but at least it didn't crash" is a weak consolation. A stopping pacemaker is almost always preferable to a malfunctioning one You almost make an excuse for general unreliability of programs. Mainstream C is unreliable, C++ is unreliable, Rust is unreliable. I can agree that Rust is not less reliable than C/C++, but it is definitely less reliable than some other language e.g. BEAM-based ones. I mean in Rust standard library some time ago I actually read "in these and these conditions the following code will deadlock. But deadlock is not an undefined behavior, so it's ok". The designers of Rust did not really try to support any kind of "recover and continue" way of functioning. Yes, you can catch the panic, but it will irreversibly poison some data.

yencabulator 11mo ago

A pacemaker crashing can restart, log a diagnostic, continue operating, and report the diagnostic remotely. (yes, really! Bluetooth to phone or to dedicated relay gadget.) A pacemaker in an unknown state goes forever undiagnosed.

oconnor663 10mo ago

> If the unwrap hadn't caused an exit, the process would've run out of memory It was trying to push an element into a full ArrayVec. The options are: - Blindly write off the end of the array. Obviously no one wants this, despite the decades of tradition... - Panic and unwind, as the program actually did in this case. - Return an error. Some folks assume that returning an error instead of unwinding would've been better. But my assumption is that the outcome would've been the same. I think the issue came up when loading configs, which isn't usually a recoverable situation. If you have an "invalid config error", you're probably just going to return that all the way up, which is effectively the same outcome as unwinding: your process exits with an error code. There are cases where the difference matters a lot, but I don't think this was one of them. The real gap seems to be why it took hours for folks to notice that this service was crash looping. That should normally be really prominent in alerts and dashboards. (Probably part of the story is that alerts were firing all over the place. Tough day at the office.)