4 ms·

Health boss unsure how many patients were overdosed due to Windows upgrade

26 points by kiyanwang 5y ago

jhabdas 5y ago
Finger pointing time for the DS.
aitchnyu 5y ago
Umm, what's the mechanism of this bug?
- besnn00 5y ago
  A very vague article if you ask me.
  - breakingcups 5y ago
    If you read between the lines you can see how many management layers the description of this bug went through before arriving here: "It was a generic issue in the prescribing software. It's a patch relating to upgrading to Microsoft 10." Like a game of telephone (chinese whispers?), the actual root cause gets buried in layers of misunderstanding. The root cause might be (purely as an example, not even speculative on this particular case): "We use a terminal emulator to interface with an ancient system, the particular emulator we use has a dependency on library XYZ which has always mishandled number inputs but nobody noticed due to an unexpected interaction with Windows' IME. In Windows 10, IME was reworked and the bug was no longer masked." Who's really at fault in such a case? Is it Microsoft for changing OS behavior? Is it the original manufacturer/developer for relying on unspecified behavior? Is it the hospitals for not fully testing each upgrade themselves?
    - lucasyvas 5y ago
      While I would assume that in most cases the primary fault would belong to the manufacturer/developer in your example, there is missing change control at every level. Each party should be asking "If something were to break, what's the worst that could happen?" This concept should be mandated at every level of software rollout that could endanger lives. Part of this is a failure of the software industry because we have perpetuated the narrative that stuff "Just works." which is dangerously misleading to the general public. The public needs to understand more about the practicalities of software through better education. As developers, we need to push robustness way further than the average company cares to produce.
- squarefoot 5y ago
  "It's added another digit and replicated the last digit, so if you were to have 17 milligrams, it would have prescribed 177 milligrams," Such an error seems possible only if treating numbers as strings, an error that would defeat a possible function to validate dosages according to the patient prescriptions and related min-max values. I wouldn't use Windows for anything critical such as medical prescriptions, but honestly that looks more like sloppy programming.
castillar76 5y ago
Reminiscent of the the Therac-25 (https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25) incident. The sad thing is that there will be a lot of finger-pointing, potentially some financial penalties that will get paid off without incident, and probably a dev or two thrown under the bus. I doubt we'll see any meaningful change as result, though.
- archontes 5y ago
  I would expect to see an officer of the company subpoenaed. The chain of accountability starts with them, at least in theory. To the FDA, a dev doing the wrong thing is evidence that the company doesn't have systematic control of their development and production.
MarkusWandel 5y ago
Software that can kill people if it malfunctions, has to be reliable. A monstrously complex OS that is prone to changing its behaviour through auto-updates isn't even remotely suitable. However, what works is if the complex system is monitored by a simple, auditable sanity check. In the case of the Therac-25, nothing bad would have happened if the simple mechanical interlocks ("if A isn't in place, don't turn on B") hadn't been optimized away due to misplaced faith in the software's reliability. Similarly, a trained medical person would look at the off-by-10x dosage number and say this isn't right, let's just check that. And a simple can't fail software check could probably flag this sort of thing as well.
- deleted 5y ago
  [deleted]
- jwalton 5y ago
  Fun story - as a kid in the 80s, I caught pneumonia, and a doctor misplaced the decimal point in my weight and computed a dosage for me that was literally a 10x overdose. Then two nurses both thought they were supposed to give me the shot in question, so I ended up getting a 20x overdose. I spent a week in the hospital.

26 points by kiyanwang 5y ago

jhabdas 5y ago

Finger pointing time for the DS.

aitchnyu 5y ago

Umm, what's the mechanism of this bug?

besnn00 5y ago
A very vague article if you ask me.
- breakingcups 5y ago
  If you read between the lines you can see how many management layers the description of this bug went through before arriving here: "It was a generic issue in the prescribing software. It's a patch relating to upgrading to Microsoft 10." Like a game of telephone (chinese whispers?), the actual root cause gets buried in layers of misunderstanding. The root cause might be (purely as an example, not even speculative on this particular case): "We use a terminal emulator to interface with an ancient system, the particular emulator we use has a dependency on library XYZ which has always mishandled number inputs but nobody noticed due to an unexpected interaction with Windows' IME. In Windows 10, IME was reworked and the bug was no longer masked." Who's really at fault in such a case? Is it Microsoft for changing OS behavior? Is it the original manufacturer/developer for relying on unspecified behavior? Is it the hospitals for not fully testing each upgrade themselves?
  - lucasyvas 5y ago
    While I would assume that in most cases the primary fault would belong to the manufacturer/developer in your example, there is missing change control at every level. Each party should be asking "If something were to break, what's the worst that could happen?" This concept should be mandated at every level of software rollout that could endanger lives. Part of this is a failure of the software industry because we have perpetuated the narrative that stuff "Just works." which is dangerously misleading to the general public. The public needs to understand more about the practicalities of software through better education. As developers, we need to push robustness way further than the average company cares to produce.
squarefoot 5y ago
"It's added another digit and replicated the last digit, so if you were to have 17 milligrams, it would have prescribed 177 milligrams," Such an error seems possible only if treating numbers as strings, an error that would defeat a possible function to validate dosages according to the patient prescriptions and related min-max values. I wouldn't use Windows for anything critical such as medical prescriptions, but honestly that looks more like sloppy programming.

castillar76 5y ago

Reminiscent of the the Therac-25 (https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25) incident. The sad thing is that there will be a lot of finger-pointing, potentially some financial penalties that will get paid off without incident, and probably a dev or two thrown under the bus. I doubt we'll see any meaningful change as result, though.

archontes 5y ago
I would expect to see an officer of the company subpoenaed. The chain of accountability starts with them, at least in theory. To the FDA, a dev doing the wrong thing is evidence that the company doesn't have systematic control of their development and production.

MarkusWandel 5y ago

Software that can kill people if it malfunctions, has to be reliable. A monstrously complex OS that is prone to changing its behaviour through auto-updates isn't even remotely suitable. However, what works is if the complex system is monitored by a simple, auditable sanity check. In the case of the Therac-25, nothing bad would have happened if the simple mechanical interlocks ("if A isn't in place, don't turn on B") hadn't been optimized away due to misplaced faith in the software's reliability. Similarly, a trained medical person would look at the off-by-10x dosage number and say this isn't right, let's just check that. And a simple can't fail software check could probably flag this sort of thing as well.

deleted 5y ago
[deleted]
jwalton 5y ago
Fun story - as a kid in the 80s, I caught pneumonia, and a doctor misplaced the decimal point in my weight and computed a dosage for me that was literally a 10x overdose. Then two nurses both thought they were supposed to give me the shot in question, so I ended up getting a 20x overdose. I spent a week in the hospital.