6 ms·

Honestly, I disagree slightly. Reading the article as well as the original report years ago, I wasn’t left with the feeling the operator made “no fault of their

Honestly, I disagree slightly. Reading the article as well as the original report years ago, I wasn’t left with the feeling the operator made “no fault of their own”. Are they to blame at all, no, but the operator certainly made mistakes. For example, assuming an error is innocuous when you are intentionally delivering radiation to a person is careless at best. Again, the machine is at fault solely, but that doesn’t mean the operator didn’t have a role in the death.

nv-vn 8y ago
Yeah, I took away the same thing. As an example, in the aviation industry something like this would simply not be tolerated. When you are operating a potentially dangerous device, you have to do so with the utmost care. This isn't to say the technician should be punished, but one of the results of this investigation should have been a focus on making technicians aware of how disastrous the consequences could be if they don't respond appropriately to an error.
- hexane360 8y ago
  Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/killed-by-a-machine-the-therac-25/ https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther...
  - AnssiH 8y ago
    I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.
    - Jtsummers 8y ago
      The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.
      - jsjohnst 8y ago
        Still doesn’t negate the fact the technician should’ve still checked. Again, not saying the technician is at fault at all, but they aren’t free of being involved in the death either. I’ll use a terrible analogy to make the point, if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time, are you completely not responsible if you pull the trigger the sixth time and die?
        perl4ever 8y ago
        "if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time" I'm not sure that's a valid analogy. Isn't it the case rather that the revolver said "that's bad" but a person said "it's fine, ignore that, it always does that"? Which, in my experience, happens all the time in the workplace. If you didn't trust other people you work with, it would defeat the purpose of being in an organization at all. Not to mention, if pulling the trigger repeatedly was required for normal operations, you really can't blame the operator at all.
        jsjohnst 8y ago
        > Which, in my experience, happens all the time in the workplace. If you are doing work that could immediately cause someone’s death, I feel you have a duty to double check and not just go with the flow. Ymmv and I’m not claiming my POV is more right, just that it’s my POV.
      - 8y ago
  - ghaff 8y ago
    The pilot example probably isn't the best one.If a plane with that design was nonetheless deemed airworthy by the FAA, pilots would absolutely be expected to know the correct operating procedures for that plane. Look at the "unusual" choice of averaging inputs from the pilot and co-pilot that helped lead to AF 447. It's very reasonable to argue it's bad design, but it was the responsibility of the pilots to know how it worked.
    - hexane360 8y ago
      Again, this was a completely undocumented (and unintentional) race condition. There was nothing in the manual stating that "the machine will do bad things if you try to change the mode twice in less than 8 seconds". Averaging of control inputs was a documented, intentional design choice. Being "expected to know the correct operating procedures" doesn't extend to "know how the machine works better than its designers" or "be omniscient".
      - ghaff 8y ago
        Yes. There is a huge difference between undocumented and documented. And also between pilots flying a plane they are certified on and nurses who are probably less specialized.
      - nv-vn 8y ago
        Right, I meant that the issue was not responding safely to a malfunction. If it was undocumented, the operator should have paused the operation and attempted to contact the manufacturer. I don't blame them for their response necessarily (in the same way I don't think the programmers were actually guilty of a crime), but I think that should have been the main takeaway from this.

nv-vn 8y ago

Yeah, I took away the same thing. As an example, in the aviation industry something like this would simply not be tolerated. When you are operating a potentially dangerous device, you have to do so with the utmost care. This isn't to say the technician should be punished, but one of the results of this investigation should have been a focus on making technicians aware of how disastrous the consequences could be if they don't respond appropriately to an error.

hexane360 8y ago
Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/killed-by-a-machine-the-therac-25/ https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther...
- AnssiH 8y ago
  I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.
  - Jtsummers 8y ago
    The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.
    - jsjohnst 8y ago
      Still doesn’t negate the fact the technician should’ve still checked. Again, not saying the technician is at fault at all, but they aren’t free of being involved in the death either. I’ll use a terrible analogy to make the point, if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time, are you completely not responsible if you pull the trigger the sixth time and die?
      - perl4ever 8y ago
        "if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time" I'm not sure that's a valid analogy. Isn't it the case rather that the revolver said "that's bad" but a person said "it's fine, ignore that, it always does that"? Which, in my experience, happens all the time in the workplace. If you didn't trust other people you work with, it would defeat the purpose of being in an organization at all. Not to mention, if pulling the trigger repeatedly was required for normal operations, you really can't blame the operator at all.
        jsjohnst 8y ago
        > Which, in my experience, happens all the time in the workplace. If you are doing work that could immediately cause someone’s death, I feel you have a duty to double check and not just go with the flow. Ymmv and I’m not claiming my POV is more right, just that it’s my POV.
    - 8y ago
- ghaff 8y ago
  The pilot example probably isn't the best one.If a plane with that design was nonetheless deemed airworthy by the FAA, pilots would absolutely be expected to know the correct operating procedures for that plane. Look at the "unusual" choice of averaging inputs from the pilot and co-pilot that helped lead to AF 447. It's very reasonable to argue it's bad design, but it was the responsibility of the pilots to know how it worked.
  - hexane360 8y ago
    Again, this was a completely undocumented (and unintentional) race condition. There was nothing in the manual stating that "the machine will do bad things if you try to change the mode twice in less than 8 seconds". Averaging of control inputs was a documented, intentional design choice. Being "expected to know the correct operating procedures" doesn't extend to "know how the machine works better than its designers" or "be omniscient".
    - ghaff 8y ago
      Yes. There is a huge difference between undocumented and documented. And also between pilots flying a plane they are certified on and nurses who are probably less specialized.
    - nv-vn 8y ago
      Right, I meant that the issue was not responding safely to a malfunction. If it was undocumented, the operator should have paused the operation and attempted to contact the manufacturer. I don't blame them for their response necessarily (in the same way I don't think the programmers were actually guilty of a crime), but I think that should have been the main takeaway from this.