4 ms·

Uncaught exception and die. For a banking application, yes, die, hard, as loudly as possible. Downtime is worth it. Do not continue, do nothing until it is und

by harry8 7y ago

Uncaught exception and die. For a banking application, yes, die, hard, as loudly as possible. Downtime is worth it. Do not continue, do nothing until it is understood and fixed. When you're flying. Hmmm. Not so much. Dying is a really bad idea. Ok you should have it tested thoroughly so it isn't going to happen but if it does and "anything could happen" well "anything" is better than killing all the passengers, crew and anyone you might crash on, right? Literally anything is better than that. Idealy, you log it, you refuse to take off again, all the fleet is grounded etc etc but deciding to fall out of the sky because something unexpected happened is probably not the right response. Is graceful shutdown possible? No? Ok, so take your chances and log noisily is a less bad option.

MayeulC 7y ago
Well, for an unmanned rocket, have it automatically self-destruct is likely the saner choice, as you don't want it to fall back somewhere inhabited, and there is usually little use for something on the wrong orbit.
Mikhail_Edoshin 7y ago
The default out-of-box runtime can only handle one general scenario. The runtime designers choose to print the error message and terminate, and this seems to be the only sound option for default behavior. It's up to the developer to replace this default handler when the scenario differs that much.
- mumblemumble 7y ago
  > this seems to be the only sound option for default behavior I think that is the crux of this debate. In my background, yeah, you fail hard when anything unexpected happens. It's the most straightforward way to fail safe. But my background maps better to the financial case; I've never worked on avionics or anything like that. I can see the point, though, that, in that kind of situation, failing hard doesn't fail safe at all. It's conceivable to me that different problem domains require different default behavior. (And perhaps, by extension, different programming languages.)
  - Const-me 7y ago
    That particular set of requirements is not unique to avionics. It’s pretty close in game development. A glitch is way better than a game which crashes in the middle of gameplay. Especially because many glitches only last a single frame i.e. barely visible. > It's conceivable to me that different problem domains require different default behavior. (And perhaps, by extension, different programming languages) Not sure about problem domains, e.g. 3D graphics is the main problem domain in videogames people play for fun, and CT scanners which save lives. Different projects indeed have different requirements. Every time you see a programming-related holy war on the internets (exceptions/status codes, static/dynamic typing, unit testing or not, FP/OOP, etc.), you can bet the participants have different types of projects on their background, and these projects needed different tradeoffs. More often than not, what’s good for a web startup aint’t good for Nasa or MS, and vice versa.
wycy 7y ago
Whether die-hard or steamroll-thru is the better option depends more on what exactly goes wrong rather than what the application is. For a flight control computer, for example, if the ordered pitch suddenly becomes 3.402823466e+38 degrees it might be better to die hard and restart rather than try to execute the order.
- ff317 7y ago
  I imagine (just thinking out loud randomly, I'm sure there's other issues to consider!) one way to mitigate this would be to have a second set of much simpler software (written in a different language, perhaps running on a different platform) whose only runtime job is to sit in the communications path from the primary software to the avionics hardware and monitor the outputs of the primary software and ensure the values and/or rate-of-change in certain values are within some physics-based sanity limits before they hit the hardware. It could be responsible for rebooting the primary software and holding the last-known-good output values (for the consuming avionics) while the reboot happens (hopefully quickly!). Of course then who watches the watcher, and you've added more things in the critical path which can have their own failure modes...
  - xaedes 7y ago
    "whose only runtime job is to sit in the communications path from the primary software to the avionics hardware and monitor the outputs of the primary software and ensure the values and/or rate-of-change in certain values are within some physics-based sanity limits before they hit the hardware" Brings to my mind: Erlang
    - deathtrader666 7y ago
      Was waiting for someone to mention Erlang.
- K0balt 7y ago
  The answer for this is sanity checking and some kind of layered "reflexes". I design for mission critical things on a regular basis, and one of the error modes I must accommodate is random bit flipping by cosmic rays, emf, or other failures. Sometimes, it is possible to push things to a "safe" failure state and reboot (which often takes only 100 ms or so) Sometimes, though, the error must be caught and corrected to a last known good value or something like that. Everything critical is sanity checked : overall boundaries, rate of change, state related boundaries, etc. Layers of "reflexes" are more robust than a single programmed behavior, because an error in one will be resisted by the others. So much the better when there are segregated systems to check each other. Often, I'll have a "brilliant" system that performs to a very high standard and is complex and brittle. If the brilliant system fails, it just shuts down or reboots. Underneath the brilliant system is another layer or two. A "competent" system that is simpler and more robust, but less efficient and with soft failure modes, and a "brainstem" system that takes nothing for granted, even checking basic functions by bit flipping to negate stuck ram bits or broken ALU logic, but only tries to do a pasable job, reducing algorythmic complexity to its bare acceptable minimum. Typically the system will generate a basic range of acceptable parameters at the lowest level (and take corrective action if needed), then refine the range at each subsequent level..rinse and repeat. That way, each lower level checks the ones above. Or, you just fail downwards if errors are suspected. Either way. Designing failure tolerant systems is not impossible, but it requires a different mindset.

by harry8 7y ago

MayeulC 7y ago

Well, for an unmanned rocket, have it automatically self-destruct is likely the saner choice, as you don't want it to fall back somewhere inhabited, and there is usually little use for something on the wrong orbit.

Mikhail_Edoshin 7y ago

The default out-of-box runtime can only handle one general scenario. The runtime designers choose to print the error message and terminate, and this seems to be the only sound option for default behavior. It's up to the developer to replace this default handler when the scenario differs that much.

mumblemumble 7y ago
> this seems to be the only sound option for default behavior I think that is the crux of this debate. In my background, yeah, you fail hard when anything unexpected happens. It's the most straightforward way to fail safe. But my background maps better to the financial case; I've never worked on avionics or anything like that. I can see the point, though, that, in that kind of situation, failing hard doesn't fail safe at all. It's conceivable to me that different problem domains require different default behavior. (And perhaps, by extension, different programming languages.)
- Const-me 7y ago
  That particular set of requirements is not unique to avionics. It’s pretty close in game development. A glitch is way better than a game which crashes in the middle of gameplay. Especially because many glitches only last a single frame i.e. barely visible. > It's conceivable to me that different problem domains require different default behavior. (And perhaps, by extension, different programming languages) Not sure about problem domains, e.g. 3D graphics is the main problem domain in videogames people play for fun, and CT scanners which save lives. Different projects indeed have different requirements. Every time you see a programming-related holy war on the internets (exceptions/status codes, static/dynamic typing, unit testing or not, FP/OOP, etc.), you can bet the participants have different types of projects on their background, and these projects needed different tradeoffs. More often than not, what’s good for a web startup aint’t good for Nasa or MS, and vice versa.

wycy 7y ago

Whether die-hard or steamroll-thru is the better option depends more on what exactly goes wrong rather than what the application is. For a flight control computer, for example, if the ordered pitch suddenly becomes 3.402823466e+38 degrees it might be better to die hard and restart rather than try to execute the order.

ff317 7y ago
I imagine (just thinking out loud randomly, I'm sure there's other issues to consider!) one way to mitigate this would be to have a second set of much simpler software (written in a different language, perhaps running on a different platform) whose only runtime job is to sit in the communications path from the primary software to the avionics hardware and monitor the outputs of the primary software and ensure the values and/or rate-of-change in certain values are within some physics-based sanity limits before they hit the hardware. It could be responsible for rebooting the primary software and holding the last-known-good output values (for the consuming avionics) while the reboot happens (hopefully quickly!). Of course then who watches the watcher, and you've added more things in the critical path which can have their own failure modes...
- xaedes 7y ago
  "whose only runtime job is to sit in the communications path from the primary software to the avionics hardware and monitor the outputs of the primary software and ensure the values and/or rate-of-change in certain values are within some physics-based sanity limits before they hit the hardware" Brings to my mind: Erlang
  - deathtrader666 7y ago
    Was waiting for someone to mention Erlang.
K0balt 7y ago
The answer for this is sanity checking and some kind of layered "reflexes". I design for mission critical things on a regular basis, and one of the error modes I must accommodate is random bit flipping by cosmic rays, emf, or other failures. Sometimes, it is possible to push things to a "safe" failure state and reboot (which often takes only 100 ms or so) Sometimes, though, the error must be caught and corrected to a last known good value or something like that. Everything critical is sanity checked : overall boundaries, rate of change, state related boundaries, etc. Layers of "reflexes" are more robust than a single programmed behavior, because an error in one will be resisted by the others. So much the better when there are segregated systems to check each other. Often, I'll have a "brilliant" system that performs to a very high standard and is complex and brittle. If the brilliant system fails, it just shuts down or reboots. Underneath the brilliant system is another layer or two. A "competent" system that is simpler and more robust, but less efficient and with soft failure modes, and a "brainstem" system that takes nothing for granted, even checking basic functions by bit flipping to negate stuck ram bits or broken ALU logic, but only tries to do a pasable job, reducing algorythmic complexity to its bare acceptable minimum. Typically the system will generate a basic range of acceptable parameters at the lowest level (and take corrective action if needed), then refine the range at each subsequent level..rinse and repeat. That way, each lower level checks the ones above. Or, you just fail downwards if errors are suspected. Either way. Designing failure tolerant systems is not impossible, but it requires a different mindset.