8 ms·
My insulin pump controller has a bug
- hturan 3y agoFor those unaware, misjudging a dose of rapid-acting insulin by even a few units can prove fatal. Can’t fathom expecting a quarter of a unit and getting 100x the amount.
- mtmail 3y agoNon-twitter link: https://nitter.net/morganherlocker/status/1730455721815527429 https://nitter.net/morganherlocker/status/173045572181552742... (edit: I'm not the author) "Notified today that my insulin pump controller has a bug where the leading decimal point will be dropped, ie: changing a dose of .21 units to 21 units. I can reproduce randomly ~1 in 5 times so probably a race condition. Easily one of the worst software bugs I have ever heard of."
- jmrjmr 3y agoWhat brand/model?
- patagurbon 3y agoOmnipod 5 when using the Android app as the controller
- stranded22 3y agoTy - wife is T1D and waiting for closed loop with omnipod. Might wait a little longer…
- yankoff 3y agoDamn, that's the one I'm using although with a separate controller. I've been looking forward for the iPhone app that is supposed to come out in 2024.
- numpad0 3y ago> using the Android app as the controller
- sidewndr46 3y agowouldn't this put someone into a state of insulin shock?
- yankoff 3y agoIt will drop the blood sugar level dangerously low and cause hypoglycemia. A person can lose consciousness, get into a coma, or even die.
- deleted 3y ago[deleted]
- wwilim 3y agoIf you wanted 0.66 units and got 66 units, I am pretty sure you would die. I would count on the tissue around the infusion site not having enough throughput to release it all into your system at once and swelling up. This would mean the problem would spread over a few hours, but it would be an extreme emergency anyway. To put it into perspective (at least with my insulin to carb ratio, it varies from person to person), if you wanted to drink a regular sized can of coke, you would dose 3.3u. 66u is therefore enough insulin for 20 cans of coke. Another way to put it is that 10g of sugar ~ 1u of insulin. You'd have to eat 660g of raw sugar. There is a medication called Glucagen which is used to treat severe hypo. It works by triggering release of stored glucose from the liver. The thing is, from what I've quickly googled, a typical 1mg dose is usually only enough to raise your blood sugar by about 100mg/dl, and then your store is depleted and further doses have no effect. With my insulin/carb ratio, I would only need about 2.5u to reduce my blood sugar by 100mg/dl. All of a sudden it feels scary to think that the reservoir in my pump holds about 300u of insulin...
- pixl97 3y agoHeh, yea, this is why when I went to DEFCON I disconnected the feed to my body. I mean it's not likely someone's going to try to murder me via bluetooth, but why take the risk when I didn't have to. In the days before I had a pump I did something similar to myself. I had a short acting insulin and long acting insulin both in pen form. Managed to give myself the large overnight dose which should have been the long acting with the short acting pen. After about 15 minutes I realized something was up when I felt the low coming on and spent the next 2 hours drinking a disgusting amount of orange juice.
- auggierose 3y agoFor software like that, not sure how anything other than formally proven correct software is acceptable. And note, no type system is gonna help you with a bug like that.
- chlorion 3y agoI don't know very much about formal methods but I would assume the type system plays an important role for encoding that specific states can or cant exist.
- asfarley 3y agoBut a high reading is a state that can exist. I don’t really see how formal methods would exclude a multiplication/unit error with respect to the displayed unit.
- layer8 3y agoWith the right type system, any formal proof can be represented in it, and the type checker will verify the proof. Any fault that remains can also remain with a formal proof. Look up Curry–Howard isomorphism.
- auggierose 3y agoObviously I mean an ML-like type system with automatic inference. Otherwise it is just general logic, and you definitely don't need types for that. And Curry-Howard is the most overrated isomorphism in history. It may make computer scientists head explode, but mathematicians can live without it. And yes, I know that Lean is based on it, and Tao is using it, and he just doesn't know better. Sorry.
- layer8 3y agoType inference is largely orthogonal to the question. The point is, any formal proof you could do outside a type system you can also do within a type system, so yes the right type system would help you avoid such bugs.
- e2le 3y agoIt sounds like the bug was in the Android app. Shouldn't the microcontroller (on the pump) be validating dose values received? (dose > MAX_DOSE) ? return DOSE_TOO_HIGH : return DOSE_ACCEPTED; 20 units is obviously dangerous for a single dose.
- jacquesm 3y agoAbsolutely. The App should be considered hostile by the pump firmware until proven otherwise.
- patagurbon 3y agoThis is quite the overreaction. There is both a confirmation screen and a maximum bolus setting to ameliorate the issue until an OTA update can be provided. Stranding users who use the app without access to our lifesaving medical device is... Bold, to say the least.
- jacquesm 3y agoIf you are a diabetic you should always have a plan B for your 'lifesaving medical device' because they can break. For instance: https://www.northwoodsendocrinology.com/wp-content/uploads/2018/10/Insulin-Pump-Back-Up-Plan.pdf https://www.northwoodsendocrinology.com/wp-content/uploads/2... Presumably whoever provided you your pump has a similar instruction. Risking using an app/pump combo that has a known flaw, especially a non-deterministic one is bold, to say the least. Safer to switch to the regular controller (assuming you have it) until this is fixed.
- grayhatter 3y agoWhat you mean to say is completely correct. What you actually said is wrong, there's a huge delta between hostile and unproven. The pump firmware should assert it's own safety and sanity checks. Which is different from hostility.
- jacquesm 3y ago
- bradknowles 3y agoTherac-25 walks into the chat. See: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- SuperNinKenDo 3y agoWow, that seems extremely serious. In any reql engineering field, this kind of thing would have serious repercussions for the engineer, and not merely the poor consumer.
- yread 3y agoLet's hope the notified body will stomp on them hard
- jacquesm 3y agoFull code audit, exhaustive testing. This is not toy stuff.
- ttyprintk 3y agoThe MAUDE database tracks this for FDA. The average consumer can sign up for alerts based on make and model: https://maude.innolitics.com/ https://maude.innolitics.com/
- Mountain_Skies 3y agoIn any real engineering field, the engineer wouldn't compete for a job with "engineers" who went through a two month boot camp to be an engineer. They'd also have power to say "No" when a project manager insisted on pushing out an improperly tested product.
- mardifoufs 3y agoI share your sentiment but in this case, I don't think bootcamp devs usually get to work on such a device. Not fresh of bootcamp devs at least. But yes, this general type of event is inevitably what happens when the entire field prides itself in not having any standards.
- Decabytes 3y agoI know the department of defense used to use Ada because it was a “safety critical” language. I feel like when it comes to this devices there should be more stringent guidelines around coding practices. Is that not the case?
- kragen 3y agothere are, that's why insulin pump firmware is developed using obsolete and error-prone methods by substandard engineers and why you can't get the source for your insulin pump firmware and reflash it with a bug-fixed version
- chpatrick 3y agoI see where you're coming from, but I don't think letting anyone mess with the equipment keeping them alive is a good idea either. I really love open source IoT stuff for example and I think it's a great that people are trying to take real ownership of their devices. However it's different if the penalty for failure is your lights not working or immediate death. I think a better solution would be if the source was required to be public so people can identify bugs, but without giving users the ability to flash.
- noir_lord 3y ago> I think a better solution would be if the source was required to be public so people can identify bugs, but without giving users the ability to flash. That would be a solution, still has the problem of "how do you confirm the bug if you can't run the code". Finding those kinds of subtle bugs pretty much requires you be able to execute it.
- kragen 3y agoi do, because not 'letting anyone mess with the equipment keeping them alive' is, obviously, condemning some of them to deaths that they would avoid if they were allowed to i think you would need an overwhelmingly large number of prevented suicides and avoided foolish accidents to justify such a monstrous evil the balance of deaths is very likely to work out the other way around, in fact, because people reflashing their own insulin pumps have much stronger incentives to maximize their survival chances than any possible regulatory regime would, much less the one that actually exists the policy of 'let us make laws and set up regulatory agencies to prevent people from harming themselves' has such an astoundingly bad track record that it always surprises me to hear someone advocating it in apparent seriousness
- andreasley 3y agoMay be this pump/issue: https://www.tandemdiabetes.com/docs/default-source/legal/company-update/basal-rate-field-correction-notice-legacy-pumps-qsf0010911.pdf https://www.tandemdiabetes.com/docs/default-source/legal/com... However, the document states that "there is no defect identified in relation to the use of this product", which would contradict the Twitter post, so maybe it's a different product or either the poster or the document is wrong.
- diogocp 3y agoIt does not contradict the Twitter post. They're saying that since there is a confirmation screen, it's the user's responsibility to catch the bug. this-is-fine.jpg
- unwind 3y agoNo, they (the manufacturer) are saying that users might "inadvertently" enter and confirm an incorrect value, which is not the same as the users enters one value but a bug causes it to be parsed, displayed and confirmed as a different value. The latter is what the Tweet claimed. Very confusing. Also, why would you ever post about a potentially dangerous bug in a medical device without naming it? That just seems ... oddly sensational, to me.
- patagurbon 3y agoIt is not this pump I believe. Instead it is the Omnipod 5, we received notifications this morning.
- thulle 3y ago> February 08, 2022 They'd need 20 months to manage to notify users?
- omh 3y agoI've got this Tandem pump and we discussed this exact bug when I received the pump. To my understanding this isn't the same type of bug. Tandem are just saying "it can be confusing to enter fractional rates". This bug is for a different pump and is "when you enter a fraction rate it will change the rate". That is much worse
- Ferret7446 3y agoI can't comprehend how that kind of bug even happens. Not that insulin pumps should have any bugs, but it doesn't even sound like an integer overflow or use after free which is "understandable". Are they doing some kind of scuffed string parsing?
- ajdude 3y agoThey probably wrote the controller in JavaScript
- pavel_lishin 3y agoAt my first job out of college, we were hired to create a point-of-sale system. We wrote it in Javascript, being unaware of how Javascript treats numbers. (Like a rented mule.) After the first few days, the client reached out and told us that the end-of-day numbers were sometimes off by a few cents - tax totals, things like that. After a few weeks, they'd be off by significantly more. Easily the most embarrassing incident of my career, even worse than when I accidentally left a `console.log("FUCK");` in production code as a debugging tool, or when we wrote some code to generate random four-letter codes that would spew out the occasional profanity.
- freedomben 3y agoWe had a similar issue with random four-letter codes. The simplest and easiest solution was to remove vowels from the pool of candidates. Consonant only codes are much harder to generate real words.
- Maxion 3y agoFCK!
- freedomben 3y agoThat's only 3 letters! FXCK was probably the closest we got with that change
- tibbydudeza 3y agoI worked for a bank - a bug (arithmetic overflow) tried to charge a customers credit card for more than a million kabillion on a payment - thankfully it was caught upstream and declined. One of the reasons why I will never work in the medical field - too much real life consequences - wrong account entries can be fixed by journal entries :).
- pixl97 3y agoI'm not sure about this unit, but with my Tandem there is a maximum dose setting on the unit. I don't use the app to dose, so I have no idea if the app obeys that. Even with the maximum dose setting there is a massive range of doses some people need, especially if they have resistance and don't manage carbs well. As others have mentioned in thread they use a 30u max dose setting, which for me would be around 1/3rd of the total daily intake.
- patagurbon 3y agoThis is the Omnipod 5 when using the Android app as the controller. I received the email this morning. This is indeed a pretty terrible bug. I'm hopeful the confirmation screen prevented any real harm from being done. E: I can reproduce. It seems to happen around 1 in 10 entries for me.
- testplzignore 3y agoI'm very curious to see this in action. Are you able to get a video of it showing typing in the dose and showing the confirmation screen missing the decimal point?
- patagurbon 3y agoBoth the Omnipod and Dexcom apps prevent screen recording. Possibly for user privacy, but I'm suspicious of that. I don't really have a way to get a second video recorder at the moment.
- schiffern 3y ago>I'm hopeful the confirmation screen prevented any real harm from being done. I hope so too, but I doubt it. Confirmation screens are notorious for becoming habituated and then ignored. Yes, even important ones. https://alistapart.com/article/neveruseawarning/ https://alistapart.com/article/neveruseawarning/
- pixl97 3y agoYep, when you use something a few times a day it's very easy to get interface blindness. I use a t.slim myself and catching something like .10 and 10 is going to be very, very easy to miss. Probably need a leading zero like 0.10 to make it stand out. And even then most the time I'm clicking the conformations to get on with my life and do something else.
- cadr 3y agoDo you know what version of the app you have? I've not been able to reproduce it yet on 1.0.91 (Following the steps as I saw at https://twitter.com/Tims_Pants/status/1730515134731182490 https://twitter.com/Tims_Pants/status/1730515134731182490)
- fifteen1506 3y agoCue the "you wouldn't steal a car" music. ♫ "You wouldn't demand open-source software for life-critical contexts instead of the industry-perfected closed software" ♫
- parthianshotgun 3y agoI am bad at reading this, can you elaborate what you mean? Should we open source life critical software?
- fsflover 3y agoWe should: https://news.ycombinator.com/item?id=38143180 https://news.ycombinator.com/item?id=38143180
- parthianshotgun 3y agoOh good! I agree, perhaps some of it can be defined as a social good. I remember the idea that was posed of having government created social media. Something open source and with a transparent algorithm. I really can't see a downside if it's done well.
- onetimeuse92304 3y agoI think you are missing the point. The point is that with commercial software is that there exists an entity that accepts responsibility for it under threat of severe penalties. You don't want somebody to be killed and then have just everybody shrug "it is not me" and not be able to find a culprit. Personally, I always include a note that my software is offered as is and I do not accept any liability for anything that it might do including damage to property or somebody's health or life. That, obviously, absolutely disqualifies it from any critical use. But that's only fair, if I am not compensated for my work I should not be held liable for it either. Now, once we find out the software pretty much must be developed commercially, there exists currently no way to force the commerce to make the code public and there are valid arguments in the debate that making it public could make it less safe.
- GTP 3y agoA new Thearc-25?
- jacquesm 3y agoIt may well be, depends on whether anybody died on account of this bug. 100x... mindbogglingly irresponsible to not have multiple safeguards that caught this before production. Note that these are marketed to children as well.
- Havoc 3y agoThat smells like a lawsuit. 10x insulin in 1 out of 5 is wild
- Aaargh20318 3y ago10x ? 21 instead of .21 is 100x This is basically Russian roulette with insulin with even worse odds.
- amelius 3y agoBut does the device contain that much insulin?
- phil21 3y agoThis device can hold up to 200 units of insulin in it's cartridge. I'm not familiar with it to say if it has any safety interlocks to administer some sort of max dose lower than that at a time though.
- patagurbon 3y agoThere is a user setting for maximum bolus. For me I have it set to 30, but some may set it lower. Indeed that might be recommended after this.
- pixl97 3y agoYea, you should never run the device without a max bolus set. Even 30 to me seems high, but insulin update can be wildly random between people.
- omh 3y agoYes. A typical pump will contain enough for several days. My pump right now has 100 units and is only half full.
- quitit 3y ago
- omh 3y agoA clearer description of the bug is here: https://twitter.com/Tims_Pants/status/1730515134731182490 https://twitter.com/Tims_Pants/status/1730515134731182490 It's wild that this sort of bug got through testing. As a diabetic it feels like our insulin pump software is very conservative and lacking in features especially compared to what some of the "closed loop" things would like to do. That seems reasonable if the manufacturers are having to do lots of safety testing. But if bugs like this are getting through then the testing obviously isn't anywhere near as robust as we'd like.
- fabian2k 3y agoThe part that makes this even more dangerous than this description might indicate is that it appears to be somewhat random. So most of the time entering a value as e.g. ".5" would work, but sometimes it would not. This is probably more dangerous than if it would always fail to do that, as the first few times you use it you likely pay much more attention to the confirmation dialog than later when you're used to it.
- phkahler 3y agoThat linked description is very interesting. They list 3 steps that need to happen to get an incorrect dose. The 3rd step is that YOU confirm the dose. The next section emphasizes the importance of confirming it. That's all great, and yeah, the user basically hit OK but that does not change the fact that they have a software bug. Also, since it happens intermittently with that kind of input I have to seriously question how the software is put together. If the input box shows the decimal there's no way it should slip past the parser. Something smells very wrong with how their app is put together, and that would make me concerned about other issues we just don't know about yet.
- omh 3y agoAlso it would be very possible to misread the confirmation. If I've just entered "0.21" then when the confirmation screen reads "21" it's not immediately obvious that it's wrong.
- 3y ago
- lkt 3y agoHere is the safety notification: https://www.omnipod.com/en-gb/fsn-11-2023 https://www.omnipod.com/en-gb/fsn-11-2023
- deltaburnt 3y agoI'm not familiar with insulin dosing, but I'm guessing a sharp 10x jump should warrant my app telling me to go to the hospital? I'd expect medical devices like this to have a last line of defence safety check against bugs. Basically checking all common sense edge cases like: is this dose lethal to humans, is this dose significantly more/less than historical doses, is this dose below a level in which most humans would get no benefit. These are all easily testable both in terms of software testing and regulatory testing.
- Kudos 3y agoIt's a 100x dose.
- bradley13 3y agoGood software would certainly include that kind of multi-layered defense. Insulin dosage does not vary by a factor of 100. Sadly, medical software is not written any better than any other software. In this case, the firmware was probably written by engineers who don't think about possible user errors, because they are focused on the hardware. Meanwhile, the app was probably outsourced to the lowest bidder from Elbonia.
- mlsu 3y agoI can't speak for Omnipod in particular, but insulin pump firmware is absolutely written with user error in mind. There is probably an entire team at the company ("human factors") that will do testing to make sure that the planned UX for the device would not (or, better yet, cannot) be misinterpreted or misused. Also, other commenters here are insinuating that it's possible to wash your hands of something like this by just putting some text warning like "you should always check the dose before delivery." That is not true, neither for the FDA nor for the internal standards for the manufacturer.
- wwilim 3y agoI nearly died in my sleep 2 years ago from about 10 units unexpectedly releasing into my system from swollen tissue left behind by a broken infusion site. What the author describes is potentially lethal. The thing is, there is no such thing as a lethal dose of insulin per se. If you want to eat half a large pizza and wash it down with a can of coke, you'll likely dose 15 or more units. But dose the same amount and end up not eating your meal and you are in extremely big trouble within half an hour.
- deleted 3y ago[deleted]
- cypherpunks01 3y ago"We understand this may impact your experience and are here to support you." I love this - I've never heard the euphemism impact your experience used to mean that your life may end. What a wild PR assignment (language off the company's safety notice https://www.omnipod.com/en-gb/fsn-11-2023 https://www.omnipod.com/en-gb/fsn-11-2023)
- stalfosknight 3y agoWhy do so many people / companies insist on using grossly euphemistic managerspeak like this?
- nerbert 3y agoBecause anything different from that would be admitting that it is a life threatening situation and they are not going to treat it as such.
- deleted 3y ago[deleted]
- theshrike79 3y agoLess risk of liability and cancellation.
- numpad0 3y agoBecause it works. People who don't will not get to keep their jobs in corporate environments, even when nothing is going on.
- stalfosknight 3y agoI work in a corporate environment and I refuse to use managerspeak. I haven't lost my job.
- randomdata 3y agoLike all business communication (and communication in general, really): Fashion. While you occasionally find some trailblazers to set what is new in fashion, mostly people just copy what other people are doing.
- looperhacks 3y agoNow I know why my wifes insulin pump's (Medtronic) Android app doesn't support setting the bolus and only allows observation :D It's likely better that way. If the app would allow this, I'd already decompiled the app and tried to reverse engineer the protocol to allow her to set a bolus when she's wearing something that makes it hard to access the pump or carry her phone with her (like a dress). Setting the bolus via e.g. a smartwatch would be really nice, but I'd be very afraid of creating a life-threatening bug.
- code_duck 3y agoThe app and firmware for the Tandem T-Slim were updated last year or so to allow dosage from the phone app. People on Twitter and reddit were very excited - finally! How convenient! My thought was wow, that is something I really don’t want at all, from a standpoint of possible bugs, security risk and accidental dosage. I don’t want someone who hacks my phone or insulin pump to be able to kill me. Of course, nobody on reddit took my opinion seriously.
- tenebrisalietum 3y agoWonder if it's possible to enter "0.21" and avoid the bug.
- patagurbon 3y agoThe notification we received specifically mentioned that typing ".21" was the issue, without a leading 0
- MauranKilom 3y agoFrom the twitter description it wasn't clear whether this was just a display bug or the device actually failing to record the user input. I naively assumed it was just failing to display the decimal point. But no, going by the manufacturer statement, the actual value entered and used will ignore the decimal point. You still have to confirm the (correctly-displayed-incorrectly-received-value). If you don't pay attention (because the last 10 times it worked) you might just die... That's unbelievable. Aside: This is reminding me of the relatively-orthogonal-but-still-awful-imo-UX where entering numbers into payment fields (e.g. card reader or PayPal app) just floods in the numbers from the right, with every digit passing through the decimal places as you type. I don't know why it irks me so much more than "normal" number entering, but it just feels so wrong. I'm entering a quantity in (e.g.) dollars, but I have to type it as a number of pennies. Normally, entering the decimal point would disambiguate my intent clearly even if I type too few or too many decimal zeros...
- thfuran 3y ago>I don't know why it irks me so much more than "normal" number entering Because it's a grievous violation of your (and presumably every user's; I know I hate those too) understanding of how a text field works. And certain types of edits are made significantly less convenient by that behavior as well.
- jacquesm 3y agoBecause there are countries that use '.' and countries that use ',' interchangeably and some are so messed up they use both. So entering pennies is the safest option.
- patagurbon 3y agoOne other thing to note here, with the Dexcom and Omnipod running together you really shouldn't be giving yourself boluses directly. Instead you type in the number of carbs you intake, and let the system make adjustments. In my experience taking direct insulin messes with the algorithm that tries to manage your sugar. I suspect this is why this wasn't noticed immediately.
- adamwright0 3y agoHi medical software researcher / professor here. This is definitely a bad bug! I’d like to use it as a teaching example — any chance you could post a video of the bug? Especially one that shows how it sometimes takes the value correctly and sometimes changes it?