14 ms·

How a robot pharmacist wound up giving an overdose

63 points by steven 12y ago

robertfw 12y ago
> (The computer symbol for swearing, #$@&%*!, was chosen because it’s the output when you’re trying to type numbers but you’ve inadvertently engaged the caps lock key.) Did this used to be true? I have never encountered this. Holding down Shift, perhaps, but not caps lock...
- quotedmycode 12y ago
  Typewriters had shift-lock, which would have changed the numbers. http://ux.stackexchange.com/questions/64230/why-does-the-caps-lock-not-toggle-upper-and-lower-case-for-selected-text http://ux.stackexchange.com/questions/64230/why-does-the-cap...
- iamcurious 12y ago
  Symbol swearing goes to at least 1900: http://languagelog.ldc.upenn.edu/nll/?p=2483 http://languagelog.ldc.upenn.edu/nll/?p=2483 Not sure about the Caps Lock thing though.
- Marcus316 12y ago
  I've never used a keyboard/driver which had this functionality (CAPS Lock only affects letters on any system I've used in the past). However, I believe the SHIFT LOCK key that existed on some typewriters functioned in the fashion described in the article, as the SHIFT LOCK key did, in fact, affect all keys on a few typewriters I have used.
matznerd 12y ago
Serious cliff hanger on this article, with the second part of how it actually made the mistake held off until tomorrow. Reminds me of one of those old tv shows...
coldcode 12y ago
UX design, especially for medical software, is incredibly important and yet is rarely done by people who actually understand how to do it. I remember when my mom had radiation treatments for brain cancer, which were so awful they killed her quicker than the disease would have (by a few weeks). A few months later I read about how terrible the design for calibrating these machines was leading to a large number of mis-treatments, my sister and I wondered what dose she actually received. During the Vietnam war pilots turned off all the alarms in some of their planes due to constant alarms that always happened during intense combat situations leading to an inability of focus on what mattered. UX design is not easy but has to be treated as more important than just making it pretty. Understanding how real people use these things is incredibly important and not just a last minute bullet check.
- at-fates-hands 12y ago
  >>>UX design is not easy but has to be treated as more important than just making it pretty. Understanding how real people use these things is incredibly important and not just a last minute bullet check. Amen brother. The last four large scale business enterprise applications I've worked on, design was left to either the back-end developers, project managers or done by committee. The ones done by committee were absolutely brutal. Nothing rips your soul out quicker than trying to get 20 or more people to agree on something. I have no idea when or where design just got completely buried and left to people who clearly have no idea what they're doing, but in your case, it's incredibly troubling and scary.
- llamataboot 12y ago
  Yes, my grandmother received approximately 100x the dose she was supposed to receive due to operator error which resulted in her intestines being fried and gave her digestive problems for the rest of her life. (Did kill the cancer though)
- deleted 12y ago
  [deleted]
  - stonemetal 12y ago
ccvannorman 12y ago
> Computer experts call this type of problem — when the same action can result in two very different results — a “mode error,” and it is especially problematic when the user is not focused on the mode (in this case, mg versus mg/kg) Terrifying example of one way AI or computer-augmented decisions can go horribly wrong. A "Mode error" is at the core of the difference between intelligence and common sense.
discardorama 12y ago
FTA: "There is no difference between a minuscule overdose — going 0.1 milligram over a recommended dose — and this very large overdose. They all look exactly the same." That here is the problem, right here. If we can have self-driving cars (which can look ahead and figure out the result of their actions), so can a system like EPIC. Why can't it calculate the probability of a Bad Thing(tm) happening, and suitable tag the alerts?
- vilhelm_s 12y ago
  Maybe this is just a case of comparing a tech demo (the car) against a deployed system. Presumably hospital software will gradually improve in the future.
Someone1234 12y ago
While I absolutely agree computers, UIs, and systems in general can stand between a patient and an error, can we just stop and acknowledge how incompetent the doctor was for prescribing 38 instead of 1 double strong dose? The UI screenshot in the article makes that mistake VERY clear, and their "are you 100% sure" question is obviously intended to illicit a particular response (who is ever really 100% sure of anything?). I know this article is about how systems can be altered to protect patients, but much like with the NTSB's findings, sometimes it legitimately is pilot error (or doctor error in this case). Can things be better? Yes. Absolutely. But so can doctors. These types of systems are stop-gaps, without them this doctor could have calculated the wrong dose by hand and made just of large of a mistake.
- BHSPitMonkey 12y ago
  > and their "are you 100% sure" question is obviously intended to illicit a particular response (who is ever really 100% sure of anything?). When you routinely get prompted with that question dozens (hundreds?) of times a day, it can't elicit that response successfully. It's just not how our psychology works.
  - Someone1234 12y ago
    I'm not sure I follow. The author of that article asked people that question at conferences when showing them the UI screenshot in the article.
    - BHSPitMonkey 12y ago
      Misunderstood; I thought you were defending the confirmation prompt in the software itself.
- shpx 12y ago
  You should actually read the article.
  - Someone1234 12y ago
revelation 12y ago
Give me a break. The system has alerts and it spells out the exact dosage in tablets in a dozen places, which is then apparently seen by at least three people. If these people in the course of their work choose to ignore alerts, are distracted and not actually doing what they were hired to do (hint: theres a reason there were three humans in the chain here), the fault lies on them, not on the software. You obviously can't have it both ways. We call it automation because things will happen automatically. At the end of the day, this system probably causes significantly less problems than the old way of trying to read some doctors handwriting only to pick up the wrong medicine regardless.
- jsprogrammer 12y ago
  Don't forget about the unwritten 'real policy', as opposed to the actual policy: >Every training program has a “hidden curriculum” (the way things are actually done around here, as opposed to what the policies say or what the administrators told you during that interminable orientation). One of them — passed down from senior residents to the newbies — was, “Ignore all the alerts.”
- llamataboot 12y ago
  Really, you think the UX of those screens is in anyway clear or that the alerts shouldn't be color coded or differentiated by thresholds from "hey this looks like it might be an error" to "this is out of the realm of possibility, do not pass go, do not collect 200, correct this immediately"
  - isaacdl 12y ago
    But who gets to set the threshold? One doctor might want a full stop at 3% over, one might want one at 10%, one might not necessarily want one until maybe 50% (perhaps in this case of a research hospital that is routinely trying experimental treatments). I don't disagree that this shouldn't be considered carefully, but it's a little harder than maybe you're making it seem.
    - llamataboot 12y ago
geoelectric 12y ago
> Like many other physicians, pharmacists, and nurses, Jenny Lucca found alerts to be a constant nuisance. Even giving Tylenol to a feverish child every four hours triggered an alert that the dose was approaching the maximum allowed. You know, given what we know about the liver toxicity of acetaminophen/paracetamol at this point, not sure this is the best example of a spurious alert. There's a good argument that this is one of the more dangerous drugs OTC, in part due to the widespread impression that it's not very dangerous. It is interesting looking at the story in general as an example of alert fatigue, though, and the need to differentiate different types of alerts. Unlike some of the other comments I see here, I don't think this points to incompetence so much as human nature. It takes a fair amount of cognitive power to overcome the natural tendency to assume that when the alert was spurious yesterday, it's not spurious today. That power would be better spent elsewhere, and points to a failure of automation to truly lighten the load.
- nucleardog 12y ago
  Not "exceeding the maximum dose" just "approaching the maximum dose". "Hey, you're almost taking exactly the amount of this thing we say you can take. Watch out!" is pretty much the definition of a spurious alert to me. It's telling you that what you expect to happen is happening.
  - geoelectric 12y ago
    Well, yeah, except pills up the amount in steps--it's not like turning a dial. So you'd want to warn before they can give another dose of medication that would take you over the line. Really depends on how far off the line the warning is. But either way, in this case I'm pretty sure Tylenol was stressed in the article because it was an example of an innocuous medication, which it really kind of isn't.
- PhantomGremlin 12y ago
  > given what we know about the liver toxicity of acetaminophen/paracetamol at this point, not sure this is the best example of a spurious alert Why is acetaminophen even in the picture? Isn't ibuprofen very similar in effect but much safer in an overdose?
Terr_ 12y ago
I work developing an internal company app that's already a little trigger-happy on the e-mail alerts (which are not at all life-and-death) and a lot of it boils down to an organizational/political malaise. The most common problem is when the people setting alert policies are divorced from the people receiving them. The managers have no direct incentive to keep the alerts sane or moderated. On an individual basis, every alert that exists shifts potential blame from the manager to the employee. ("It's all the employee's fault for not following the alerts.") In this way, over-alerting is often "automated micromanagement", but without even the feedback-loop and mutual-accountability of actual micromanagement. Until the people acting on alerts have some feedback into the alert-configuring process, the problem will continue. Another dysfunction is when people ask for alerts purely for an independent paper trail, either because they're paranoid about assigning blame, or because they want to use their e-mail client as a half-assed searchable database. P.S.: We've already had managers asking us if we could have e-mail alerts that users must confirm receipt of by visiting the website... and for the site to continuously e-mail followups until they do. Meanwhile, the IT staff have their own horror-stories of ancient inboxes grown too large to reliably read data from.
- angersock 12y ago
  This is exactly it--nurses are usually not empowered to set alarm thresholds or delays, but physicians are. So, docs ask for the most sensitive settings they can (because they want that intimacy of care), but they're gone 15 minutes later and the rest of the unit has to deal with the fallout for the rest of the day.
  - maddog99 12y ago
    Not sure what "intimacy of care" means, but most docs, like me, are frustrated and annoyed by the barrage of wolf-crying that comes out of these systems, of which Epic is one of the best and best-designed. (So far, all the others I've seen are FAR worse.) I WANT to see the meaningful alerts, but it's difficult to do with a sea of identical-appearing utter BS. Besides alert fatigue, there's a related problem with cognitive load. For this patient, I want to order A, B, C, and D. Now erect roadblocks in front of A, B, and C, and quite often D will fall by the wayside. By the way, I've never been able to set or even influence the alert policy. This was mandated by corporate risk management, whose policy appears always to be more alerts = safer.
angersock 12y ago
A few different observations: First, Epic does a lot of custom work--as noted in the article, each installation is "customized" to only have particular alerts. This makes it hard to count on a standard system behavior, because there isn't one. This continued coddling of clients makes it very hard to introduce new software into the market, because you can't do it without deep pockets and sloooooow sales cycles. Second, the physicians are very much overloaded with alarms and alerts--and instead of saying "Hey, let's figure out why we keep ending up with so many alarms" they instead do the short-term thing and just ignore or turn them off. The entire profession is rooted in split-second decisionmaking, for reasons I cannot fathom. They really, really, really want to be the heros that come in and save the day, and systematic approaches to operations optimization just aren't a priority. Third, the data coming off of these things (and at work we've seen this) usually indicates large operational deficiencies. Unfortunately, you can't just tell the clients "Hey, you all are doing things wrong, and here's the proof", because they either won't care, or they will fire you and bring in somebody who keeps them happy, or they'll care but be unable to make changes because that introduces uncertainty into their line of work and that could hurt someone--nevermind all the existing casualties.
- ams6110 12y ago
  The entire profession is rooted in split-second decisionmaking, for reasons I cannot fathom Because insurance companies, Medicare, and Medicaid reimbursements barely cover costs, so to be profitable they have to make it up on volume.
  - angersock 12y ago
    It's more fundamental than that, I think. It's core to the entire mindset of the profession. If you compare the operators of decades ago with the sysadmins of the 90s and the devops folks of today you can see a rough path of evolution that hopefully medicine will follow: a last-minute heroic effort to save a system is a failure, not a victory.
- FireBeyond 12y ago
  "The entire profession is rooted in split-second decisionmaking, for reasons I cannot fathom." Right. There are very few points in time where decisions in medicine need be like this. Even in the ED, where I interact most with physicians. Hell, the ED I see most often has checklists to "stop and review" (and they do) for things that most people would consider (and are) time-critical, cardiac arrests, intubation, and the like. It's odd to get used to, but seems to work very well.
Turbo_hedgehog 12y ago
Confused about one thing, did the robot put 38.5 pills in a bottle, or did it formulate a pill containing 38.5 pills worth of meds into one pill, which the resident gave to the patient? I'm thinking sitting there and giving the patient 38.5 pills for the patient to swallow would have been considered unusual?
- radicalbyte 12y ago
  From my experience with pharmacy robots (my wife owns a Pharmacy, their group have just decommissioned their expensive robot because it doesn't work): It almost certainly put 38.5 pills per day in a bottle. At which point the dispensing Pharmacist or Assistant should have started asking questions. Doctors -up these kind of things all the time; around 5% of prescriptions based on the very scientific measurement of the relative size of the good/bad stacks on my wife's desk.
- FireBeyond 12y ago
  It attached them in 39 little bags to a loop of plastic. You can see the picture near the bottom. If I see 39 individual pills in 39 individual bags like that and am being told that I am to administer them, at once, to a patient, alarm bells in my head are ringing, as a paramedic. Swallowing 39 tablets like that is not fun for anyone, even aside from risks of OD.
  - bentcorner 12y ago
    I'm wondering about this too. I guess we'll have to wait until Part 3. One other thing that went wrong here - through no fault of her own, the mom was in a different room with the brother. She knows what the patient normally takes, and would likely have started asking questions when she saw her son taking way more than what he usually took.
brandonb 12y ago
I wonder if somebody could build a bayesian filter for Epic alerts. If we can filter out the Nigerian prince scams from your email, why not apply that to save lives?
- Kalium 12y ago
  I suspect a great deal of lurking complexity and privacy concerns there. What's spurious in one case is critical in another, and the difference lies in HIPAA-sensitive data.
mcphage 12y ago
I'm torn on this. On one hand: we can see the process, and clearly identify a number of mistakes in the software that led to this problem. Serious mistakes, that need to get fixed. And on the other: while this was an error that wouldn't happen without computers and robots, it's clear that this process will reduce the number of overall errors, saving peoples lives. But it's a different set of people. This is similar to the "Trolley Problem" ethical thought experiment—we have a computerized system, that will save a lot of peoples' lives—at the cost of others, who wouldn't have otherwise died. It's a hard problem, with no easy solution. See: http://en.wikipedia.org/wiki/Trolley_problem http://en.wikipedia.org/wiki/Trolley_problem
viciousambition 12y ago
The title of this is basically false. The humans gave the overdose, the robot merely processed the order.
comex 12y ago
There are least two comments here already ascribing the error to incompetence; I expect more will come. In my opinion, this is a dangerous misperception. Hindsight is 20/20. You saw one screenshot displaying the incorrect dosage; the doctors see "scores" each day, day in, day out. The brain is hardwired to ignore repetitive stimuli. Humans are smarter than computers overall but have a much higher error rate; "competence" can decrease that rate, but not to zero. There is always a chance of a mistake. The software could have been designed to make the probability of requesting a 38.5x overdose from the robot pharmacist negligible, by reserving a special warning for extreme circumstances. This requires the programmers to do a non-shit (shall I say "competent"?) job once, but not constant vigilance by the doctor, who I'm sure can use the saved mental cycles to check for many other potential errors the machine can't handle completely trivially. I expect a more thorough design overhaul would cause a non-negligible reduction in the occurrence of a wide variety of mistakes, dangerous and not.
- FireBeyond 12y ago
  There was a remark made by one of those involved, in the article, that EPICs alerting didn't differentiate between a 'trivial' alert, say "Dose is 1.02x maximum recommended dosage" and a much more critical "Dose is 38.5x..."
  - bentcorner 12y ago
    A scale would be useful: Recommended dosage | ----|-x------------ vs ----|------------x- Also for mg vs mg/kg, that's a harder problem - off the top my head, get rid of the modes, and show both mg/kg and mg textboxes, with a radio button to select the one you want to edit. Edit it, and the non-selected text box is grayed-out and uneditable, and shows you the corresponding calculation (enter in mg/kg, and it shows you the mg amount of medicine; enter in mg, and it shows you the mg/kg ratio).
    - maddog99 12y ago
      As a sanity check, what about showing the ordering physician a picture of 38.5 tablets? "Is this correct?" I think that would grab my attention.
seagreen 12y ago
Pharmacists shouldn't be "clicking through" alerts. They should have a separate, joystick style device with an actual trigger (bright red). That trigger, and that trigger alone, can dismiss an alert. Seriously, what would that cost, like $20? (Obviously there are a lot more things wrong with this system than just using a mouse for too much, but this would help). We should be willing to invest more in our machines: http://www.loper-os.org/?p=861 http://www.loper-os.org/?p=861
- chucksmash 12y ago
  And if they are beset with hundreds of mostly innocuous alerts a day, the low signal/noise ratio of the alerts and their sheer quantity will train the pharmacists to press that button just a bit more blithely each time. That leads one the believe the correct course of action is to reduce the number of alerts but then that was at least part of the cause of the incident related in the article. My wife is a Pharm.D. at a hospital (that coincidentally is currently installing Epic). I know she worries (obsesses even) over the possibility of making an error that hurts or kills someone. I'd like to think other people doing that work feel the same way. Even if she can get three or four nines worth of confidence in every decision she makes however, that leads to some unfortunate probabilities when multiplied out by a career's worth of decisions. We can improve the technology to reduce the chances of significantly negative outcomes but they will always be non-zero. I don't envy her or the programmers working on systems like Epic. It's messy work with serious consequences.
  - seagreen 12y ago
    Reducing alerts is way more important than my idea, I totally agree with that.
sjg007 12y ago
This is why you must always always be at your kids beside and if you are in the hospital, always have a friend with you (if you are in the hospital and otherwise not capable), who can double check and ask questions on your behalf every time.
maddog99 12y ago
My proposal: usability experts should identify an acceptable range of alerts on a per-order, per-encounter basis. End-users of all disciplines should have the opportunity to provide feedback in real-time, in the live environment. Just like Amazon reviews, "Was this alert helpful?" If you always vote "no", your feedback will be down-weighted. With a few exceptions, like wrong patient or 10x overdose, all the other alerts will be ranked in order of utility and only the top x% will be shown.

63 points by steven 12y ago

> (The computer symbol for swearing, #$@&%*!, was chosen because it’s the output when you’re trying to type numbers but you’ve inadvertently engaged the caps lock key.) Did this used to be true? I have never encountered this. Holding down Shift, perhaps, but not caps lock...

quotedmycode 12y ago
Typewriters had shift-lock, which would have changed the numbers. http://ux.stackexchange.com/questions/64230/why-does-the-caps-lock-not-toggle-upper-and-lower-case-for-selected-text http://ux.stackexchange.com/questions/64230/why-does-the-cap...
iamcurious 12y ago
Symbol swearing goes to at least 1900: http://languagelog.ldc.upenn.edu/nll/?p=2483 http://languagelog.ldc.upenn.edu/nll/?p=2483 Not sure about the Caps Lock thing though.
Marcus316 12y ago
I've never used a keyboard/driver which had this functionality (CAPS Lock only affects letters on any system I've used in the past). However, I believe the SHIFT LOCK key that existed on some typewriters functioned in the fashion described in the article, as the SHIFT LOCK key did, in fact, affect all keys on a few typewriters I have used.

matznerd 12y ago

Serious cliff hanger on this article, with the second part of how it actually made the mistake held off until tomorrow. Reminds me of one of those old tv shows...

coldcode 12y ago

UX design, especially for medical software, is incredibly important and yet is rarely done by people who actually understand how to do it. I remember when my mom had radiation treatments for brain cancer, which were so awful they killed her quicker than the disease would have (by a few weeks). A few months later I read about how terrible the design for calibrating these machines was leading to a large number of mis-treatments, my sister and I wondered what dose she actually received. During the Vietnam war pilots turned off all the alarms in some of their planes due to constant alarms that always happened during intense combat situations leading to an inability of focus on what mattered. UX design is not easy but has to be treated as more important than just making it pretty. Understanding how real people use these things is incredibly important and not just a last minute bullet check.

at-fates-hands 12y ago
>>>UX design is not easy but has to be treated as more important than just making it pretty. Understanding how real people use these things is incredibly important and not just a last minute bullet check. Amen brother. The last four large scale business enterprise applications I've worked on, design was left to either the back-end developers, project managers or done by committee. The ones done by committee were absolutely brutal. Nothing rips your soul out quicker than trying to get 20 or more people to agree on something. I have no idea when or where design just got completely buried and left to people who clearly have no idea what they're doing, but in your case, it's incredibly troubling and scary.
llamataboot 12y ago
Yes, my grandmother received approximately 100x the dose she was supposed to receive due to operator error which resulted in her intestines being fried and gave her digestive problems for the rest of her life. (Did kill the cancer though)
deleted 12y ago
[deleted]
- stonemetal 12y ago

ccvannorman 12y ago

> Computer experts call this type of problem — when the same action can result in two very different results — a “mode error,” and it is especially problematic when the user is not focused on the mode (in this case, mg versus mg/kg) Terrifying example of one way AI or computer-augmented decisions can go horribly wrong. A "Mode error" is at the core of the difference between intelligence and common sense.

discardorama 12y ago

FTA: "There is no difference between a minuscule overdose — going 0.1 milligram over a recommended dose — and this very large overdose. They all look exactly the same." That here is the problem, right here. If we can have self-driving cars (which can look ahead and figure out the result of their actions), so can a system like EPIC. Why can't it calculate the probability of a Bad Thing(tm) happening, and suitable tag the alerts?

vilhelm_s 12y ago
Maybe this is just a case of comparing a tech demo (the car) against a deployed system. Presumably hospital software will gradually improve in the future.

Someone1234 12y ago

While I absolutely agree computers, UIs, and systems in general can stand between a patient and an error, can we just stop and acknowledge how incompetent the doctor was for prescribing 38 instead of 1 double strong dose? The UI screenshot in the article makes that mistake VERY clear, and their "are you 100% sure" question is obviously intended to illicit a particular response (who is ever really 100% sure of anything?). I know this article is about how systems can be altered to protect patients, but much like with the NTSB's findings, sometimes it legitimately is pilot error (or doctor error in this case). Can things be better? Yes. Absolutely. But so can doctors. These types of systems are stop-gaps, without them this doctor could have calculated the wrong dose by hand and made just of large of a mistake.

BHSPitMonkey 12y ago
> and their "are you 100% sure" question is obviously intended to illicit a particular response (who is ever really 100% sure of anything?). When you routinely get prompted with that question dozens (hundreds?) of times a day, it can't elicit that response successfully. It's just not how our psychology works.
- Someone1234 12y ago
  I'm not sure I follow. The author of that article asked people that question at conferences when showing them the UI screenshot in the article.
  - BHSPitMonkey 12y ago
    Misunderstood; I thought you were defending the confirmation prompt in the software itself.
shpx 12y ago
You should actually read the article.
- Someone1234 12y ago

revelation 12y ago

Give me a break. The system has alerts and it spells out the exact dosage in tablets in a dozen places, which is then apparently seen by at least three people. If these people in the course of their work choose to ignore alerts, are distracted and not actually doing what they were hired to do (hint: theres a reason there were three humans in the chain here), the fault lies on them, not on the software. You obviously can't have it both ways. We call it automation because things will happen automatically. At the end of the day, this system probably causes significantly less problems than the old way of trying to read some doctors handwriting only to pick up the wrong medicine regardless.

jsprogrammer 12y ago
Don't forget about the unwritten 'real policy', as opposed to the actual policy: >Every training program has a “hidden curriculum” (the way things are actually done around here, as opposed to what the policies say or what the administrators told you during that interminable orientation). One of them — passed down from senior residents to the newbies — was, “Ignore all the alerts.”
llamataboot 12y ago
Really, you think the UX of those screens is in anyway clear or that the alerts shouldn't be color coded or differentiated by thresholds from "hey this looks like it might be an error" to "this is out of the realm of possibility, do not pass go, do not collect 200, correct this immediately"
- isaacdl 12y ago
  But who gets to set the threshold? One doctor might want a full stop at 3% over, one might want one at 10%, one might not necessarily want one until maybe 50% (perhaps in this case of a research hospital that is routinely trying experimental treatments). I don't disagree that this shouldn't be considered carefully, but it's a little harder than maybe you're making it seem.
  - llamataboot 12y ago

geoelectric 12y ago

> Like many other physicians, pharmacists, and nurses, Jenny Lucca found alerts to be a constant nuisance. Even giving Tylenol to a feverish child every four hours triggered an alert that the dose was approaching the maximum allowed. You know, given what we know about the liver toxicity of acetaminophen/paracetamol at this point, not sure this is the best example of a spurious alert. There's a good argument that this is one of the more dangerous drugs OTC, in part due to the widespread impression that it's not very dangerous. It is interesting looking at the story in general as an example of alert fatigue, though, and the need to differentiate different types of alerts. Unlike some of the other comments I see here, I don't think this points to incompetence so much as human nature. It takes a fair amount of cognitive power to overcome the natural tendency to assume that when the alert was spurious yesterday, it's not spurious today. That power would be better spent elsewhere, and points to a failure of automation to truly lighten the load.

nucleardog 12y ago
Not "exceeding the maximum dose" just "approaching the maximum dose". "Hey, you're almost taking exactly the amount of this thing we say you can take. Watch out!" is pretty much the definition of a spurious alert to me. It's telling you that what you expect to happen is happening.
- geoelectric 12y ago
  Well, yeah, except pills up the amount in steps--it's not like turning a dial. So you'd want to warn before they can give another dose of medication that would take you over the line. Really depends on how far off the line the warning is. But either way, in this case I'm pretty sure Tylenol was stressed in the article because it was an example of an innocuous medication, which it really kind of isn't.
PhantomGremlin 12y ago
> given what we know about the liver toxicity of acetaminophen/paracetamol at this point, not sure this is the best example of a spurious alert Why is acetaminophen even in the picture? Isn't ibuprofen very similar in effect but much safer in an overdose?

Terr_ 12y ago

I work developing an internal company app that's already a little trigger-happy on the e-mail alerts (which are not at all life-and-death) and a lot of it boils down to an organizational/political malaise. The most common problem is when the people setting alert policies are divorced from the people receiving them. The managers have no direct incentive to keep the alerts sane or moderated. On an individual basis, every alert that exists shifts potential blame from the manager to the employee. ("It's all the employee's fault for not following the alerts.") In this way, over-alerting is often "automated micromanagement", but without even the feedback-loop and mutual-accountability of actual micromanagement. Until the people acting on alerts have some feedback into the alert-configuring process, the problem will continue. Another dysfunction is when people ask for alerts purely for an independent paper trail, either because they're paranoid about assigning blame, or because they want to use their e-mail client as a half-assed searchable database. P.S.: We've already had managers asking us if we could have e-mail alerts that users must confirm receipt of by visiting the website... and for the site to continuously e-mail followups until they do. Meanwhile, the IT staff have their own horror-stories of ancient inboxes grown too large to reliably read data from.

angersock 12y ago
This is exactly it--nurses are usually not empowered to set alarm thresholds or delays, but physicians are. So, docs ask for the most sensitive settings they can (because they want that intimacy of care), but they're gone 15 minutes later and the rest of the unit has to deal with the fallout for the rest of the day.
- maddog99 12y ago
  Not sure what "intimacy of care" means, but most docs, like me, are frustrated and annoyed by the barrage of wolf-crying that comes out of these systems, of which Epic is one of the best and best-designed. (So far, all the others I've seen are FAR worse.) I WANT to see the meaningful alerts, but it's difficult to do with a sea of identical-appearing utter BS. Besides alert fatigue, there's a related problem with cognitive load. For this patient, I want to order A, B, C, and D. Now erect roadblocks in front of A, B, and C, and quite often D will fall by the wayside. By the way, I've never been able to set or even influence the alert policy. This was mandated by corporate risk management, whose policy appears always to be more alerts = safer.

angersock 12y ago

A few different observations: First, Epic does a lot of custom work--as noted in the article, each installation is "customized" to only have particular alerts. This makes it hard to count on a standard system behavior, because there isn't one. This continued coddling of clients makes it very hard to introduce new software into the market, because you can't do it without deep pockets and sloooooow sales cycles. Second, the physicians are very much overloaded with alarms and alerts--and instead of saying "Hey, let's figure out why we keep ending up with so many alarms" they instead do the short-term thing and just ignore or turn them off. The entire profession is rooted in split-second decisionmaking, for reasons I cannot fathom. They really, really, really want to be the heros that come in and save the day, and systematic approaches to operations optimization just aren't a priority. Third, the data coming off of these things (and at work we've seen this) usually indicates large operational deficiencies. Unfortunately, you can't just tell the clients "Hey, you all are doing things wrong, and here's the proof", because they either won't care, or they will fire you and bring in somebody who keeps them happy, or they'll care but be unable to make changes because that introduces uncertainty into their line of work and that could hurt someone--nevermind all the existing casualties.

ams6110 12y ago
The entire profession is rooted in split-second decisionmaking, for reasons I cannot fathom Because insurance companies, Medicare, and Medicaid reimbursements barely cover costs, so to be profitable they have to make it up on volume.
- angersock 12y ago
  It's more fundamental than that, I think. It's core to the entire mindset of the profession. If you compare the operators of decades ago with the sysadmins of the 90s and the devops folks of today you can see a rough path of evolution that hopefully medicine will follow: a last-minute heroic effort to save a system is a failure, not a victory.
FireBeyond 12y ago
"The entire profession is rooted in split-second decisionmaking, for reasons I cannot fathom." Right. There are very few points in time where decisions in medicine need be like this. Even in the ED, where I interact most with physicians. Hell, the ED I see most often has checklists to "stop and review" (and they do) for things that most people would consider (and are) time-critical, cardiac arrests, intubation, and the like. It's odd to get used to, but seems to work very well.

Turbo_hedgehog 12y ago

Confused about one thing, did the robot put 38.5 pills in a bottle, or did it formulate a pill containing 38.5 pills worth of meds into one pill, which the resident gave to the patient? I'm thinking sitting there and giving the patient 38.5 pills for the patient to swallow would have been considered unusual?

radicalbyte 12y ago
From my experience with pharmacy robots (my wife owns a Pharmacy, their group have just decommissioned their expensive robot because it doesn't work): It almost certainly put 38.5 pills per day in a bottle. At which point the dispensing Pharmacist or Assistant should have started asking questions. Doctors -up these kind of things all the time; around 5% of prescriptions based on the very scientific measurement of the relative size of the good/bad stacks on my wife's desk.
FireBeyond 12y ago
It attached them in 39 little bags to a loop of plastic. You can see the picture near the bottom. If I see 39 individual pills in 39 individual bags like that and am being told that I am to administer them, at once, to a patient, alarm bells in my head are ringing, as a paramedic. Swallowing 39 tablets like that is not fun for anyone, even aside from risks of OD.
- bentcorner 12y ago
  I'm wondering about this too. I guess we'll have to wait until Part 3. One other thing that went wrong here - through no fault of her own, the mom was in a different room with the brother. She knows what the patient normally takes, and would likely have started asking questions when she saw her son taking way more than what he usually took.

brandonb 12y ago

I wonder if somebody could build a bayesian filter for Epic alerts. If we can filter out the Nigerian prince scams from your email, why not apply that to save lives?

Kalium 12y ago
I suspect a great deal of lurking complexity and privacy concerns there. What's spurious in one case is critical in another, and the difference lies in HIPAA-sensitive data.

mcphage 12y ago

I'm torn on this. On one hand: we can see the process, and clearly identify a number of mistakes in the software that led to this problem. Serious mistakes, that need to get fixed. And on the other: while this was an error that wouldn't happen without computers and robots, it's clear that this process will reduce the number of overall errors, saving peoples lives. But it's a different set of people. This is similar to the "Trolley Problem" ethical thought experiment—we have a computerized system, that will save a lot of peoples' lives—at the cost of others, who wouldn't have otherwise died. It's a hard problem, with no easy solution. See: http://en.wikipedia.org/wiki/Trolley_problem http://en.wikipedia.org/wiki/Trolley_problem

viciousambition 12y ago

The title of this is basically false. The humans gave the overdose, the robot merely processed the order.

comex 12y ago

There are least two comments here already ascribing the error to incompetence; I expect more will come. In my opinion, this is a dangerous misperception. Hindsight is 20/20. You saw one screenshot displaying the incorrect dosage; the doctors see "scores" each day, day in, day out. The brain is hardwired to ignore repetitive stimuli. Humans are smarter than computers overall but have a much higher error rate; "competence" can decrease that rate, but not to zero. There is always a chance of a mistake. The software could have been designed to make the probability of requesting a 38.5x overdose from the robot pharmacist negligible, by reserving a special warning for extreme circumstances. This requires the programmers to do a non-shit (shall I say "competent"?) job once, but not constant vigilance by the doctor, who I'm sure can use the saved mental cycles to check for many other potential errors the machine can't handle completely trivially. I expect a more thorough design overhaul would cause a non-negligible reduction in the occurrence of a wide variety of mistakes, dangerous and not.

FireBeyond 12y ago
There was a remark made by one of those involved, in the article, that EPICs alerting didn't differentiate between a 'trivial' alert, say "Dose is 1.02x maximum recommended dosage" and a much more critical "Dose is 38.5x..."
- bentcorner 12y ago
  A scale would be useful: Recommended dosage | ----|-x------------ vs ----|------------x- Also for mg vs mg/kg, that's a harder problem - off the top my head, get rid of the modes, and show both mg/kg and mg textboxes, with a radio button to select the one you want to edit. Edit it, and the non-selected text box is grayed-out and uneditable, and shows you the corresponding calculation (enter in mg/kg, and it shows you the mg amount of medicine; enter in mg, and it shows you the mg/kg ratio).
  - maddog99 12y ago
    As a sanity check, what about showing the ordering physician a picture of 38.5 tablets? "Is this correct?" I think that would grab my attention.

seagreen 12y ago

Pharmacists shouldn't be "clicking through" alerts. They should have a separate, joystick style device with an actual trigger (bright red). That trigger, and that trigger alone, can dismiss an alert. Seriously, what would that cost, like $20? (Obviously there are a lot more things wrong with this system than just using a mouse for too much, but this would help). We should be willing to invest more in our machines: http://www.loper-os.org/?p=861 http://www.loper-os.org/?p=861

chucksmash 12y ago
And if they are beset with hundreds of mostly innocuous alerts a day, the low signal/noise ratio of the alerts and their sheer quantity will train the pharmacists to press that button just a bit more blithely each time. That leads one the believe the correct course of action is to reduce the number of alerts but then that was at least part of the cause of the incident related in the article. My wife is a Pharm.D. at a hospital (that coincidentally is currently installing Epic). I know she worries (obsesses even) over the possibility of making an error that hurts or kills someone. I'd like to think other people doing that work feel the same way. Even if she can get three or four nines worth of confidence in every decision she makes however, that leads to some unfortunate probabilities when multiplied out by a career's worth of decisions. We can improve the technology to reduce the chances of significantly negative outcomes but they will always be non-zero. I don't envy her or the programmers working on systems like Epic. It's messy work with serious consequences.
- seagreen 12y ago
  Reducing alerts is way more important than my idea, I totally agree with that.

sjg007 12y ago

This is why you must always always be at your kids beside and if you are in the hospital, always have a friend with you (if you are in the hospital and otherwise not capable), who can double check and ask questions on your behalf every time.

maddog99 12y ago

My proposal: usability experts should identify an acceptable range of alerts on a per-order, per-encounter basis. End-users of all disciplines should have the opportunity to provide feedback in real-time, in the live environment. Just like Amazon reviews, "Was this alert helpful?" If you always vote "no", your feedback will be down-weighted. With a few exceptions, like wrong patient or 10x overdose, all the other alerts will be ranked in order of utility and only the top x% will be shown.