3 ms·
I found this comment from the article fascinating: > I am a physician who did a computer science degree before medical school. I frequently use the Therac-25 i
by jancsika 6y ago
I found this comment from the article fascinating:
> I am a physician who did a computer science degree before medical school. I frequently use the Therac-25 incident as an example of why we need dual experts who are trained in both fields. I must add two small points to this fantastic summary.
> 1. The shadow of the Therac-25 is much longer than those who remember it. In my opinion, this incident set medical informatics back 20 years. Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous, even if the individual physicians didn't know why. This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. It wasn't until the US federal government slammed down the hammer in the mid 2000's and said no payment unless you adopt electronic health records, that computers made real inroads into clinical medicine.
> 2. The medical profession, and the government agencies that regulate it, are accustomed to risk and have systems to manage it. The problem is that classical medicine is tuned to "continuous risks." If the Risk of 100 mg of aspirin is "1 risk unit" and the risk of 200 mg of aspirin is "2 risk units" then the risk of 150 mg of aspirin is strongly likely to be between 1 and 2, and it definitely won't be 1,000,000. The mechanisms we use to regulate medicine, with dosing trials, and pharmacokinetic studies, and so forth are based on this assumption that both benefit and harm are continuous functions of prescribed dose, and the physician's job is to find the sweet spot between them.
> When you let a computer handle a treatment you are exposed to a completely different kind of risk. Computers are inherently binary machines that we sometimes make simulate continuous functions. Because computers are binary, there is a potential for corner cases that expose erratic, and as this case shows, potentially fatal behavior. This is not new to computer science, but it is very foreign to medicine. Because of this, medicine has a built in blind spot in evaluating computer technology.
- mcguire 6y agoI'm not sure I buy that. Or, well, I suppose that those in the medical field believe it, but I don't think they're right. Consider something like a surgeon nicking and artery while performing some routine surgery, the patient not responding normally to anesthesia or the anesthetist not getting the mixture right and the patient not coming back the way they went in. Or that subset of patients that have poor responses to a vaccine. Everybody likes to think of the world as a linear system, but it's not.
- giantDinosaur 6y agoWell, an equivalent example is really something more like: the surgeon picking up their scalpel too quickly actually means that what they picked up wasn't a scalpel 10% of the time, even though what they saw with their eyes was a scalpel. That's a rather more terrifying error. It's not really possible in the normal world, but is easily possible in the software one (UI not corresponding to the real program state.) In other words: tangible objects usually correspond to what we see; in software, you have no way necessarily of knowing if the UI/interface is outright lying to you. It could be doing anything internally, and a single flipped bit deep in some subroutine could cause death.