5 ms·

Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "ut

Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/killed-by-a-machine-the-therac-25/ https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther...

AnssiH 8y ago
I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.
- Jtsummers 8y ago
  The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.
  - jsjohnst 8y ago
    Still doesn’t negate the fact the technician should’ve still checked. Again, not saying the technician is at fault at all, but they aren’t free of being involved in the death either. I’ll use a terrible analogy to make the point, if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time, are you completely not responsible if you pull the trigger the sixth time and die?
    - perl4ever 8y ago
      "if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time" I'm not sure that's a valid analogy. Isn't it the case rather that the revolver said "that's bad" but a person said "it's fine, ignore that, it always does that"? Which, in my experience, happens all the time in the workplace. If you didn't trust other people you work with, it would defeat the purpose of being in an organization at all. Not to mention, if pulling the trigger repeatedly was required for normal operations, you really can't blame the operator at all.
      - jsjohnst 8y ago
        > Which, in my experience, happens all the time in the workplace. If you are doing work that could immediately cause someone’s death, I feel you have a duty to double check and not just go with the flow. Ymmv and I’m not claiming my POV is more right, just that it’s my POV.
  - hexane360 8y ago
    To add to this, the more specific and informative an error message is the more authoritative it appears. "Malfunction 54" is nowhere near as good as "Unable to to set therapy mode"
    - Jtsummers 8y ago
      Yep. A good book on this sort of thing is Tragic Design [0]. I was in a pilot "Software Safety" course last year and this book was published just after, but was an excellent companion text for the course. I've been meaning to follow up with them to see what became of that work (the pilot course, my employers at the time were considering making it a mandatory/highly-recommended course for most of their software engineers and designers). [0] https://www.tragicdesign.com https://www.tragicdesign.com
- jsjohnst 8y ago
  Exactly. I don’t blame the technician in the slightest for the race condition, I just called out that ignoring the error state notice was at best careless.
ghaff 8y ago
The pilot example probably isn't the best one.If a plane with that design was nonetheless deemed airworthy by the FAA, pilots would absolutely be expected to know the correct operating procedures for that plane. Look at the "unusual" choice of averaging inputs from the pilot and co-pilot that helped lead to AF 447. It's very reasonable to argue it's bad design, but it was the responsibility of the pilots to know how it worked.
- hexane360 8y ago
  Again, this was a completely undocumented (and unintentional) race condition. There was nothing in the manual stating that "the machine will do bad things if you try to change the mode twice in less than 8 seconds". Averaging of control inputs was a documented, intentional design choice. Being "expected to know the correct operating procedures" doesn't extend to "know how the machine works better than its designers" or "be omniscient".
  - ghaff 8y ago
    Yes. There is a huge difference between undocumented and documented. And also between pilots flying a plane they are certified on and nurses who are probably less specialized.
  - nv-vn 8y ago
    Right, I meant that the issue was not responding safely to a malfunction. If it was undocumented, the operator should have paused the operation and attempted to contact the manufacturer. I don't blame them for their response necessarily (in the same way I don't think the programmers were actually guilty of a crime), but I think that should have been the main takeaway from this.

AnssiH 8y ago

I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.

Jtsummers 8y ago
The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.
- jsjohnst 8y ago
  Still doesn’t negate the fact the technician should’ve still checked. Again, not saying the technician is at fault at all, but they aren’t free of being involved in the death either. I’ll use a terrible analogy to make the point, if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time, are you completely not responsible if you pull the trigger the sixth time and die?
  - perl4ever 8y ago
    "if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time" I'm not sure that's a valid analogy. Isn't it the case rather that the revolver said "that's bad" but a person said "it's fine, ignore that, it always does that"? Which, in my experience, happens all the time in the workplace. If you didn't trust other people you work with, it would defeat the purpose of being in an organization at all. Not to mention, if pulling the trigger repeatedly was required for normal operations, you really can't blame the operator at all.
    - jsjohnst 8y ago
      > Which, in my experience, happens all the time in the workplace. If you are doing work that could immediately cause someone’s death, I feel you have a duty to double check and not just go with the flow. Ymmv and I’m not claiming my POV is more right, just that it’s my POV.
- hexane360 8y ago
  To add to this, the more specific and informative an error message is the more authoritative it appears. "Malfunction 54" is nowhere near as good as "Unable to to set therapy mode"
  - Jtsummers 8y ago
    Yep. A good book on this sort of thing is Tragic Design [0]. I was in a pilot "Software Safety" course last year and this book was published just after, but was an excellent companion text for the course. I've been meaning to follow up with them to see what became of that work (the pilot course, my employers at the time were considering making it a mandatory/highly-recommended course for most of their software engineers and designers). [0] https://www.tragicdesign.com https://www.tragicdesign.com
jsjohnst 8y ago
Exactly. I don’t blame the technician in the slightest for the race condition, I just called out that ignoring the error state notice was at best careless.

ghaff 8y ago

The pilot example probably isn't the best one.If a plane with that design was nonetheless deemed airworthy by the FAA, pilots would absolutely be expected to know the correct operating procedures for that plane. Look at the "unusual" choice of averaging inputs from the pilot and co-pilot that helped lead to AF 447. It's very reasonable to argue it's bad design, but it was the responsibility of the pilots to know how it worked.

hexane360 8y ago
Again, this was a completely undocumented (and unintentional) race condition. There was nothing in the manual stating that "the machine will do bad things if you try to change the mode twice in less than 8 seconds". Averaging of control inputs was a documented, intentional design choice. Being "expected to know the correct operating procedures" doesn't extend to "know how the machine works better than its designers" or "be omniscient".
- ghaff 8y ago
  Yes. There is a huge difference between undocumented and documented. And also between pilots flying a plane they are certified on and nurses who are probably less specialized.
- nv-vn 8y ago
  Right, I meant that the issue was not responding safely to a malfunction. If it was undocumented, the operator should have paused the operation and attempted to contact the manufacturer. I don't blame them for their response necessarily (in the same way I don't think the programmers were actually guilty of a crime), but I think that should have been the main takeaway from this.