5 ms·

Software shouldn't necessarily try to account for errors in that manner. Usually, the most graceful thing to do is to exit cleanly. For example, if there is a

Software shouldn't necessarily try to account for errors in that manner. Usually, the most graceful thing to do is to exit cleanly. For example, if there is a massive amount of data, it has to be stored on disk. It's too large to keep in memory. And if the point of the program is to transform that data in real time, then it has to have access to the disk. The antivirus basically unplugged the disk. What can it do to recover? There's nothing to be done. It should be able to survive that situation, of course. When the disk is plugged back in, it should be able to restart without any problems. But I think that's a different kind of resiliency than what you're referring to. In this case, the only way to recover would be to copy the frozen data to a new area of the hard drive, assuming it retained read access. But such complexities result in brittle implementations, prone to acquiring bugs. What if the disk space runs out? So you check beforehand whether there's enough space. But what if some other program starts consuming disk space in the middle of your copy operation? And so on. It's an endless spiral of design complexity. The situation in the article seems closer to hardware failure than a design oversight.

wpietri 11y ago
I totally disagree. Sure, usually the most graceful thing to do is exit and hope a human fixes it. But that's usual because the usual condition is that sudden failure is NBD and a human is right there to screw with it. That's becoming less common, though. When software was mostly something running on a PC doing some boring office task, reliability didn't matter. But as software is running our airplanes, our cars, our medical devices, and even, as with implanted pacemakers and insulin pumps, our bodies, then reliability gos from NBD to BFD. We see the way forward with things Chaos Monkey [1] and crash-only software [2] and the sort of design for failure you see in things like Agent supervisor hierarchies [3], where the way to reliability is through designing for failure recovery from the beginning and testing thoroughly to make sure it really happens. [1] https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey [2] https://en.wikipedia.org/wiki/Crash-only_software https://en.wikipedia.org/wiki/Crash-only_software [3] http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance.html http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance....
- sillysaurus3 11y ago
  It wasn't a rhetorical question. What could this program possibly do to recover? If the CPU fails, no one would say the program was unreliable. In this case, the disk failed, because the antivirus unplugged it. Was the program unreliable?
  - Avernar 11y ago
    The disk did not fail. An I/O operation failed. The first is a permanent condition, the second can be permanent or transient. Big difference. In Linux a signal can cause an I/O to fail. In Windows it's antivirus and other background tasks can cause I/O to fail. What can it do to recover? Retry the I/O operation! It should keep trying until the operator tells it to stop.
    - sillysaurus3 11y ago
      In that scenario, your surgeon would see the program suddenly freeze. The program likely looks like this: data acquisition -> transformation -> display transformation on monitor. If the transformation step fails, the monitor will end up displaying (a) nothing, (b) random data, or (c) the most recent image. None of these help the surgeon continue surgery. It's the same as a crash. If your environment fails, there's nothing you can do to recover. Planes aren't designed to survive the loss of a wing. Why is this case any different?
      - Avernar 11y ago
        > In that scenario, your surgeon would see the program suddenly freeze. Only if the programmer or his management were incompetent. The display routine should be running on a separate thread than the processing code. No whole program freeze should occur. As for displaying random data, why would the programmer want to do this? Either display nothing or the last readings WITH a message that it's not real time. It's not the same as a crash! A crash requires 5 minutes minimum guaranteed. Restarting instantly after telemetry returns can happen under a second in the best case which can be the difference between a live and dead patient. > If your environment fails, there's nothing you can do to recover. Planes aren't designed to survive the loss of a wing. Why is this case any different? There are different kinds of failure. Permanent and transient. Following the permanent procedure for a transient case can be fatal. Take your airplane example. Loss of a wing is permanent. That would be like the CPU failing or an external cable being cut. But your engines shutting down can be permanent or transient. Just like disk I/O failing. You'd use the transient procedure in this case. Keep trying to restart the engines. If they restart, great! You've just saved the plane. Same with the disk I/O. The programmer should keep trying to restart the I/O. If it comes back, great! You've just saved the patient.
Avernar 11y ago
> When the disk is plugged back in, it should be able to restart without any problems. But I think that's a different kind of resiliency than what you're referring to. Yes and no. I was referring to restarting internally when the error condition went away but restarting the app and waiting for telemetry to return can be a valid solution. Think of your torrent software. If you crank your firewall to block it while it's running it will not crash. If your disk fills up it won't crash. When the network comes back or more drive space if freed it will restart it's internal mechanisms. You wouldn't want it to restart in these conditions. If it runs out of memory however choosing to exit might be the best recovery mechanism. I think a life critical medical application can at least strive for internal restart and do an external restart if all else failed. The article stated they had to reboot the machine to get it back. Now that's way worse. > The situation in the article seems closer to hardware failure than a design oversight. Hardware failure is almost always a permanent condition. This was a "my I/O stopped briefly and would have came back if my code could handle it".
- sillysaurus3 11y ago
  During a surgery, the program doesn't have the luxury of showing a screen that says "No telemetry available." Such a program would be considered equally unreliable. Worse, it would lead to confusion: "Why is the telemetry unavailable? What does 'Error Code 2931' mean?" A spectacular crash immediately led to pinpointing the problem: The antivirus. If the program's sole purpose is to transform a massive amount of data in real time, it must have disk access by definition. It can't not have disk access. What would you suggest it do?
  - Avernar 11y ago
    Yes it does! Showing "no telemetry available" is exactly what it should do. Crashing = unreliable. Reporting an error condition = reliable. Immediately? Took them 5 minutes to reboot the computer. The scan of the folder would take seconds let alone minutes. Pinpointing the problem is secondary. Not killing the patient is primary. > If the program's sole purpose is to transform a massive amount of data in real time, it must have disk access by definition. It can't not have disk access. And that is the mind set the programmers of the software had. You have to take care of error conditions. The processing can't have no disk access but no disk access can occur temporarily or permanently. What can you do? Pause the processing part of your program. Or make the processing part treat "no data" as valid input and display something else. Imagine taking that viewpoint with an ECG machine: This machine displays a heart rate waveform. So it must have a heart rate input. If there is no heart rate we'll just crash requiring a 5 minute reboot. Hell no! Draw a straight line and set off a buzzer!
stevetrewick 11y ago
Software should absolutely try to account for these kinds of errors and routinely does. When was the last time you saw a word processor or spreadsheet bork a machine so hard it needed to be restarted just because an AV scan kicked in? ReadFile and ReadFileEx both hand you a specific error if some part of the file you are trying to read is locked by another process because it's hardly rare. 'Halt and catch fire' is not generally considered a proportionate response. I've no idea where you're getting this 'unplugged the disk' thing from, AV software does not work this way.

wpietri 11y ago

I totally disagree. Sure, usually the most graceful thing to do is exit and hope a human fixes it. But that's usual because the usual condition is that sudden failure is NBD and a human is right there to screw with it. That's becoming less common, though. When software was mostly something running on a PC doing some boring office task, reliability didn't matter. But as software is running our airplanes, our cars, our medical devices, and even, as with implanted pacemakers and insulin pumps, our bodies, then reliability gos from NBD to BFD. We see the way forward with things Chaos Monkey [1] and crash-only software [2] and the sort of design for failure you see in things like Agent supervisor hierarchies [3], where the way to reliability is through designing for failure recovery from the beginning and testing thoroughly to make sure it really happens. [1] https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey [2] https://en.wikipedia.org/wiki/Crash-only_software https://en.wikipedia.org/wiki/Crash-only_software [3] http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance.html http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance....

sillysaurus3 11y ago
It wasn't a rhetorical question. What could this program possibly do to recover? If the CPU fails, no one would say the program was unreliable. In this case, the disk failed, because the antivirus unplugged it. Was the program unreliable?
- Avernar 11y ago
  The disk did not fail. An I/O operation failed. The first is a permanent condition, the second can be permanent or transient. Big difference. In Linux a signal can cause an I/O to fail. In Windows it's antivirus and other background tasks can cause I/O to fail. What can it do to recover? Retry the I/O operation! It should keep trying until the operator tells it to stop.
  - sillysaurus3 11y ago
    In that scenario, your surgeon would see the program suddenly freeze. The program likely looks like this: data acquisition -> transformation -> display transformation on monitor. If the transformation step fails, the monitor will end up displaying (a) nothing, (b) random data, or (c) the most recent image. None of these help the surgeon continue surgery. It's the same as a crash. If your environment fails, there's nothing you can do to recover. Planes aren't designed to survive the loss of a wing. Why is this case any different?
    - Avernar 11y ago
      > In that scenario, your surgeon would see the program suddenly freeze. Only if the programmer or his management were incompetent. The display routine should be running on a separate thread than the processing code. No whole program freeze should occur. As for displaying random data, why would the programmer want to do this? Either display nothing or the last readings WITH a message that it's not real time. It's not the same as a crash! A crash requires 5 minutes minimum guaranteed. Restarting instantly after telemetry returns can happen under a second in the best case which can be the difference between a live and dead patient. > If your environment fails, there's nothing you can do to recover. Planes aren't designed to survive the loss of a wing. Why is this case any different? There are different kinds of failure. Permanent and transient. Following the permanent procedure for a transient case can be fatal. Take your airplane example. Loss of a wing is permanent. That would be like the CPU failing or an external cable being cut. But your engines shutting down can be permanent or transient. Just like disk I/O failing. You'd use the transient procedure in this case. Keep trying to restart the engines. If they restart, great! You've just saved the plane. Same with the disk I/O. The programmer should keep trying to restart the I/O. If it comes back, great! You've just saved the patient.

Avernar 11y ago

> When the disk is plugged back in, it should be able to restart without any problems. But I think that's a different kind of resiliency than what you're referring to. Yes and no. I was referring to restarting internally when the error condition went away but restarting the app and waiting for telemetry to return can be a valid solution. Think of your torrent software. If you crank your firewall to block it while it's running it will not crash. If your disk fills up it won't crash. When the network comes back or more drive space if freed it will restart it's internal mechanisms. You wouldn't want it to restart in these conditions. If it runs out of memory however choosing to exit might be the best recovery mechanism. I think a life critical medical application can at least strive for internal restart and do an external restart if all else failed. The article stated they had to reboot the machine to get it back. Now that's way worse. > The situation in the article seems closer to hardware failure than a design oversight. Hardware failure is almost always a permanent condition. This was a "my I/O stopped briefly and would have came back if my code could handle it".

sillysaurus3 11y ago
During a surgery, the program doesn't have the luxury of showing a screen that says "No telemetry available." Such a program would be considered equally unreliable. Worse, it would lead to confusion: "Why is the telemetry unavailable? What does 'Error Code 2931' mean?" A spectacular crash immediately led to pinpointing the problem: The antivirus. If the program's sole purpose is to transform a massive amount of data in real time, it must have disk access by definition. It can't not have disk access. What would you suggest it do?
- Avernar 11y ago
  Yes it does! Showing "no telemetry available" is exactly what it should do. Crashing = unreliable. Reporting an error condition = reliable. Immediately? Took them 5 minutes to reboot the computer. The scan of the folder would take seconds let alone minutes. Pinpointing the problem is secondary. Not killing the patient is primary. > If the program's sole purpose is to transform a massive amount of data in real time, it must have disk access by definition. It can't not have disk access. And that is the mind set the programmers of the software had. You have to take care of error conditions. The processing can't have no disk access but no disk access can occur temporarily or permanently. What can you do? Pause the processing part of your program. Or make the processing part treat "no data" as valid input and display something else. Imagine taking that viewpoint with an ECG machine: This machine displays a heart rate waveform. So it must have a heart rate input. If there is no heart rate we'll just crash requiring a 5 minute reboot. Hell no! Draw a straight line and set off a buzzer!

stevetrewick 11y ago

Software should absolutely try to account for these kinds of errors and routinely does. When was the last time you saw a word processor or spreadsheet bork a machine so hard it needed to be restarted just because an AV scan kicked in? ReadFile and ReadFileEx both hand you a specific error if some part of the file you are trying to read is locked by another process because it's hardly rare. 'Halt and catch fire' is not generally considered a proportionate response. I've no idea where you're getting this 'unplugged the disk' thing from, AV software does not work this way.