5 ms·
I totally disagree. Sure, usually the most graceful thing to do is exit and hope a human fixes it. But that's usual because the usual condition is that sudden
by wpietri 11y ago
I totally disagree.
Sure, usually the most graceful thing to do is exit and hope a human fixes it. But that's usual because the usual condition is that sudden failure is NBD and a human is right there to screw with it.
That's becoming less common, though. When software was mostly something running on a PC doing some boring office task, reliability didn't matter. But as software is running our airplanes, our cars, our medical devices, and even, as with implanted pacemakers and insulin pumps, our bodies, then reliability gos from NBD to BFD.
We see the way forward with things Chaos Monkey [1] and crash-only software [2] and the sort of design for failure you see in things like Agent supervisor hierarchies [3], where the way to reliability is through designing for failure recovery from the beginning and testing thoroughly to make sure it really happens.
[1] https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
[2] https://en.wikipedia.org/wiki/Crash-only_software https://en.wikipedia.org/wiki/Crash-only_software
[3] http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance.html http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance....
- sillysaurus3 11y agoIt wasn't a rhetorical question. What could this program possibly do to recover? If the CPU fails, no one would say the program was unreliable. In this case, the disk failed, because the antivirus unplugged it. Was the program unreliable?
- Avernar 11y agoThe disk did not fail. An I/O operation failed. The first is a permanent condition, the second can be permanent or transient. Big difference. In Linux a signal can cause an I/O to fail. In Windows it's antivirus and other background tasks can cause I/O to fail. What can it do to recover? Retry the I/O operation! It should keep trying until the operator tells it to stop.
- sillysaurus3 11y agoIn that scenario, your surgeon would see the program suddenly freeze. The program likely looks like this: data acquisition -> transformation -> display transformation on monitor. If the transformation step fails, the monitor will end up displaying (a) nothing, (b) random data, or (c) the most recent image. None of these help the surgeon continue surgery. It's the same as a crash. If your environment fails, there's nothing you can do to recover. Planes aren't designed to survive the loss of a wing. Why is this case any different?
- Avernar 11y ago> In that scenario, your surgeon would see the program suddenly freeze. Only if the programmer or his management were incompetent. The display routine should be running on a separate thread than the processing code. No whole program freeze should occur. As for displaying random data, why would the programmer want to do this? Either display nothing or the last readings WITH a message that it's not real time. It's not the same as a crash! A crash requires 5 minutes minimum guaranteed. Restarting instantly after telemetry returns can happen under a second in the best case which can be the difference between a live and dead patient. > If your environment fails, there's nothing you can do to recover. Planes aren't designed to survive the loss of a wing. Why is this case any different? There are different kinds of failure. Permanent and transient. Following the permanent procedure for a transient case can be fatal. Take your airplane example. Loss of a wing is permanent. That would be like the CPU failing or an external cable being cut. But your engines shutting down can be permanent or transient. Just like disk I/O failing. You'd use the transient procedure in this case. Keep trying to restart the engines. If they restart, great! You've just saved the plane. Same with the disk I/O. The programmer should keep trying to restart the I/O. If it comes back, great! You've just saved the patient.
- wpietri 11y agoDefinitely. Each component should do its best to keep on keeping on. The display program should keep displaying something, even it's just the most recent data with a big "connection lost" warning. The device should ring-buffer the data and upon reconnection the screen should show as much as possible. The OS should have a strong opinion that the surgery app is very important, and that should the app fail, it should be restarted instantly. Moreover, this is the kind of thing that should come up in robustness testing. Things should get bumped and wiggled. They should get unplugged and turned off. If the software is really going to run on random Windows boxes, then it should be tested on random Windows boxes. (At which point somebody will hopefully say, "Wow, this sucks, let's make it an appliance.") No matter what happens, it shouldn't result in a "mysterious crash right in the middle of a heart procedure when the screen went black and doctors had to reboot their computer".
- PhasmaFelis 11y agoNormal software does not crash the entire OS because the antivirus was looking at a file it wanted. We expect better of text editors, for God's sake. There is absolutely no excuse for a life-critical system to fail to meet that rock-bottom standard.