That's good advice, to always be vigilant about how a system would recover from a possible error/failure, at every layer. I imagine this is a discipline that a

That's good advice, to always be vigilant about how a system would recover from a possible error/failure, at every layer. I imagine this is a discipline that a software developer or engineer learns from experience, by running into edge cases, weird bugs, corrupted state, crashes, etc. It's quite a responsibility on the human side of software. Handling unexpected errors is a pretty easy thing to miss, for programmers who are inexperienced or have become too comfortable (not paying attention to every detail). I can't help but think that the language or compiler (or tests, I suppose) could enforce this better, so that it forces the programmer to explicitly handle recovery from failures - at certain layers/thresholds - so that the error doesn't bubble up to the top and bring the whole thing down. As you pointed out, an error in the UI layer shouldn't have been allowed to trigger a system crash loop. But, that's kind of the nature of bugs like these, they creep in through unexpected logic paths. In which case, the development environment needs to better enforce the (explicit) behavior of all possible paths that an error could take. Well, I'm sure that's constantly improving, as a perennial challenge of reliable software, and there will always be a need for discipline and attentiveness by the humans involved.