12 ms·
Siblings miss crucial life-extending treatment because of CrowdStrike outage
- h2odragon 2y agohttps://archive.ph/yfbek https://archive.ph/yfbek
- nullindividual 2y agoFull title: Siblings miss crucial life-extending treatment at Seattle Children’s because of CrowdStrike outage
- sgbeal 2y agoJust FYI: site is inaccessible from outside the US.
- linacica 2y agoOoof yeah, just found out about too, first time seeing something not available in my country
- throwaway3306a 2y agoWhile I appreciate the effect this kind of downtime can have, I just don't understand these stories. Presumably it was planned in advance, so the patients know the time of their appointment and the doctor knows what was planned, and everything necessary to physically perform the treatment is already prepared at the hospital. What's stopping them from doing it without filling it into a digital system? Why is it impossible to make a paper record and fill it into the computer system later? If somebody was literally dying, would they stand around the computer like confused characters in a The Sims game who can't find the door, instead of saving the life? And if not, why is this less urgent case different?
- j-bos 2y agoSee some of the comments from affected medical staff on the main outage story, but the tldr is, tightly coupled systems.
- throwaway3306a 2y agoI get that, my point is, why is it absolutely necessary to use the computer system? Why don't they just knock on the door, go grab the medicine and tools, apply it, then fill it into the system later? I understand they would just postpone whatever can be postponed to save the headache, I don't get the stories about life/health threatening situations.
- MostlyStable 2y agoI can imagine that for something like this procedure, which is an infusion of medication into the brain it sounds like?, that the "tools" to perform the procedure themselves are computer based or computer dependent. It might not be as simple as injecting a drug into an IV line. Note that I am not a doctor and have absolutely no specific knowledge beyond what is in the original article, but I am guessing at potential explanations. Additionally, the article states that there is some "wiffle [sic] room" around the timing of the infusions. So it may be that the delay is not quite as serious as the title makes it sound.
- throwaway3306a 2y agoPresumably they would fix these computers first thing during the night from a backup? If not, is this really about CrowdStrike, and not about a hospital unable to keep their absolutely critical computers backed up and restored in a timely manner? Again, I understand that restoring a complex net of servers is hard and takes time. But they surely have local hospital IT admins for these absolutely critical computers who are always available on site and can do it individually - it's not like there will be more than a hundred of these at a particular hospital? Hack it a little if you have to, disable the SSO etc - all that can be fixed later.
- CoastalCoder 2y agoDo it really make sense to blame just CrowdStrike for this? They were one link in what appears to be a pretty fragile dependency graph. For example, wouldn't it possibly make sense to also blame: * Regulators / insurers / etc. who require passing the audits that mandate using services like this. * System designers who failed to implement disaster recovery plans for this scenario. * Auditors who failed to highlight this risk. * Device vendors who made medical equipment susceptible to this kind of DoS. * U.S. FDA / DEA who allowed and/or mandated systems with this kind of vulnerability. * Voters (in democracies) who ultimately bear responsibility for their government's actions/inactions. Etc.?
- TeMPOraL 2y agoThere's lot of blame to pass around, and a lot of systems to reconsider, but at least initially, the blame lies with people who had a kill switch to critical infrastructure in multiple countries, were fully aware of that fact, and yet were so careless they accidentally pulled it.
- stefan_ 2y agoI don't exactly care who is blamed for this in the chain of stupidity, but it must happen. This corrosive attitude of "oops software problems nothing we can do" must end fast.
- deleted 2y ago[deleted]
- warkdarrior 2y agoThe more you spread the blame, the less likely it is anything will change.
- Qem 2y ago*because of CrowdStrike and Microsoft
- Bilal_io 2y agoAccording to some comments in Yesterday's thread Debian was also hit with something similar back in April. Another comment in this thread quotes Crowdstrike's ToS which states that their software should not be used on critical systems. I blame the hospital for its inability to operate with pen and paper in the event of a computer crash or a power outage.
- lambdaone 2y agoIt's pretty difficult to operate a CT scanner with pen and paper, to name just one thing that fell over yesterday. CT scanners are life-critical.
- freehorse 2y agoWhy any machine related to the operation of a CT scanner itself has to be connected to the internet? The problem is not "using technology" in general. The problem is internet connectivity being not correctly identified as a liability in designing our technology infrastructure systems.
- nullindividual 2y agoFrom what I've seen when I've had CTs (I am not a medical professional and have no direct ties to their industry), the machine sends the images in real time to a technician in another room. Those images are then sent to an offsite service for review by a radiologist, then returned to the doctor to give you the results and they're uploaded to Epic where I can review them online at my leisure. It's all on a network for a reason. If it's on a network, it has to comply with all regulations that govern the service.
- freehorse 2y ago
- Twirrim 2y agoIt's probably been raised before, but the CrowdStrike terms of use (https://www.crowdstrike.com/software-terms-of-use/ https://www.crowdstrike.com/software-terms-of-use/), section 6.1, have the usual blurb on them (emphasis mine): > Neither the software or any other Crowdstrike offerings are for use in the operation or aircraft navigation, nuclear facilities, communication systems, weapons systems, DIRECT OR INDIRECT LIFE-SUPPORT SYSTEMS, air traffic control, or any application OR INSTALLATION WHERE FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL INJURY, or property damage. SOFTWARE USER agrees that it is SOFTWARE USER’S RESPONSIBILITY TO ENSURE SAFE USE OF SOFTWARE AND ANY OTHER CROWDSTRIKE OFFERING IN SUCH APPLICATIONS AND INSTALLATIONS. We don't really think long and hard enough about isolation of systems, and what levels of access they actually need to be able to do their tasks. It's entirely practical to build completely isolated networks. US Government (and most major governments) operate classified networks with air gaps, network diodes and the like. We don't have to make everything actually internet accessible, while still retaining the ability to get data in to such isolated networks.
- nneonneo 2y agoI suspect that if the operators of such facilities obeyed all of the terms of use for every product they wanted to use, they’d be using pen and paper for everything…
- niemandhier 2y agoNo, that’s why medical and aviation products are expensive and these are difficult markets to penetrate. The degree of reliability that is required is insane, I cannot read the article since I am outside the US, BUT if these are the terms of service and the product was used in any area the was excluded under these terms, the entity that used the product might very well be guilty of gross negligence.
- Frieren 2y agoand it was not. No plane was flying with that software. It was booking services and similar needs. Planes could fly just fine, it was impossible to book people, thou. My guess is that it's similar in this case. (Site is down)
- niemandhier 2y agoPeople ridicule the German fetish for doing things on paper and using cash, but many things tend to work here even if the computers stop working. My general practitioner once treated me during a power outage, all I had to do was come back and have my insurance scanned later.
- odiroot 2y agoThat's a really moot point because the legacy banking industry there got heavily affected. The nation's weird affinity for pen and paper is nothing but Luddism. Once you have experience living in a country with good e-governance you'd roll your eyes at Germany and their love for faxes.
- niemandhier 2y agoWell I did not notice whatever affected the banking industry of my country, which at least for me makes my point. Good e-governance is incredibly difficult, which explains our love for faxes. E.g. Microsoft faced a lot of scrutiny in the congress hearing in June, some people go as far as saying MS is a danger to the national security of the USA. If the US, which is the home of these companies and can put pressure in ways the German government cannot, still can‘t force them to deliver secure systems, a fax ( at least an encrypted one) looks pretty attractive. When the EU parliament still used faxes, the US at least hat to break into the offices and manually install components to the machines to get access.
- Rinzler89 2y ago>Good e-governance is incredibly difficult And yet Estonia, a former impoverished communist country significantly less wealthy than Germany did it, and did it well. But no, Germans always have a laundry list bingo of FUD excuses as to why it can't possibly work. The bingo usually starts with "but m'uh privacy!" even though BAMF, Schufa and every law firm and government agency remotely interested in you can find out everything about you if they want to fine you for something.
- 2y ago
- 999900000999 2y agohttps://crowdstrike.wd5.myworkdayjobs.com/crowdstrikecareers https://crowdstrike.wd5.myworkdayjobs.com/crowdstrikecareers Looks like Crowdstrike outsources their SDET/QA while keeping most software engineers stateside. I generally don't have an issue with outsourcing, but it's obvious they're trying to save money on QA here. A few 200k SDETs could of probably caught this. I see this at tons of companies, they see QA as less important...
- alexchamberlain 2y agoThere are 3 axes of risk: probability that something goes wrong, the impact of something going wrong and the time to remediation when something goes wrong. You're arguing that on shoring QA would reduce the probability of something going wrong. I'm neither going to agree nor disagree. However, I think the failure here is to mitigate the impact of something going wrong. Their rollout plan was fundamentally flawed - it shouldn't have taken out so many machines at the same time. It should have been rolled out in stages, with only 1 machine at most at any given customer receiving early versions. It's best to assume a bug will get through 1 day or another, and spend some time mitigating the other axes too.
- 999900000999 2y agoMy argument is they decided to cut cost on QA. It's very likely a higher paid QA team would of caught this. A higher paid QA might of told management, hey this is a very high risk change. If we're going to roll this out let's limit it to reduce the numbers of people affected. If you on shore your core development, but outsource all of your QA, I'm forced to assume you value QA less.
- ctxc 2y ago"If we're going to roll this out let's limit it to reduce the numbers of people affected." Ime this is something senior developers would themselves do - and not only for changes they deem "high risk", but also by default. I say this because this case a data file was changed. Probably done thousands of times without an issue. QA would have never said "we need a staged rollout for this". Developers and those who set the process should do it.
- lambdaone 2y agoI had to get someone life-critical medicine yesterday. My GP practice's computers were down because, presumably, of Crowdstrike. Manual pen-and-paper processes saved the day. I wonder how many people didn't get so lucky?
- Baeocystin 2y agoI was due to pick up my ADHD meds yesterday, and couldn't, for crowdstrike and reversion to paper reasons. For me, it's mildly annoying, but I've got an emergency supply. The lines of truly desperate people with much more urgent needs than mine were long, and there was a lot of crying and despair in the lobby. I can only imagine the situation in larger cities.
- dylan604 2y agoWas this a first time prescription? If not, was there something that prevented the script from being filled earlier?
- norgie 2y agoIs that relevant?
- dylan604 2y agoIf there's something that says you can't fill the script until X days before previous runs out because of some "regulation", then yes, it is relevant. It's just another example of short sighted JIT style expectations that only compound situations like this where the supply chain is interrupted.
- mindslight 2y agoIn usual quiescent conditions, the "insurance" companies often do their damnedest to delay medical care as long as they can get away with. I've had to suffer them only allowing one week of a prescription to be filled at a time, with fulfillment done by overnight shipping. I don't even think it's about saving money per se (overnight shipping of a refrigerated package isn't cheap, and I sure as shit wouldn't have been eating the cost of a misdelivery), but rather control for control's sake. So ultimately it's not bona fide regulation taking away that last few weeks of slack and creating a needless mad rush, but rather the common setup of the "free market" unaccountably setting uniform policy in lock step, while you might get to choose which hold music you listen to.
- whalesalad 2y agoSiblings miss critical life-extending treatment because the hospital IT department didn’t architect their endpoint update strategy correctly. This should have rolled out in small, incremental steps to verify no failures. A mass “select * from hosts” global update with no testing (even if the vendor says it’s good) is entirely foolish. Hopefully folks learn from this.
- bongodongobob 2y agoIn the 20 years I've been in IT, in a variety of industries, with all the legacy manufacturing systems I've dealt with, I've never seen a software patch that blue screens all computers. Firmware, yes, rare, but never software. This is way outside of the norm.
- whalesalad 2y agoAnything operating that low in the stack is going to be a risk if shit goes wrong. It’s essentially a untrusted third party kernel update. Any sysadmin worth a dime knows not to blindly upgrade something like a kernel without staging it first - this is no different. But honestly windows admins don’t understand the systems they’re maintaining.. so this was inevitable.
- bongodongobob 2y agoYou have no idea what you're yapping about. Bye.
- whalesalad 2y agoDo you?
- CAP_NET_ADMIN 2y agoMaybe try reading what happened during the Crowdstrike fiasco and then comment about it. Crowdstrike auto updates itself, the agent allows you to select update cadence, but this was an update to the "channel" files which auto update themselves a few times per day and you don't have any control over it.
- russdill 2y agoJust to be a little fair here, healthcare providers are a major target of ransomware. How many ransomware attacks has crowd strike thwarted?
- hypercube33 2y agoWhy isn't this stuff air gapped?
- ikekkdcjkfke 2y agoShould be, or with a clearly defined interface against the open web
- heraldgeezer 2y ago[flagged]
- ctxc 2y agoIt's not "fair" that a product can brick your appliances if they have previously protected against it. In my mind, that is because the protection is what they're paid bags of cash to do. If Crowdstrike was a charity I might find myself agreeable on your description of fair in favor of Crowdstrike.
- akira2501 2y agoWhy don't EMA/EHR systems have write only encrypted journal storage that they use to guarantee data safety against this problem? I mean, just for _basic_ audits, you would hope to have that. If ransomeware can destroy your entire facility, than an angry insider can do much worse.
- russdill 2y agoEven with encrypted append only storage, any actor just needs to be more patient. Wait a week, 2 weeks, a month, whatever.
- sureglymop 2y ago
- blackeyeblitzar 2y agoThere’s are lots of reports like this. In Boston, Mass General and Brigham both shut down normal operations from what I heard.
- ugh123 2y agoArticle is void of any information about why they missed the treatment that day, just that their appointments were canceled, and thats it. What terrible reporting.
- justinclift 2y agoUgh: Error 451 It appears you are attempting to access this website from a country outside of the United States, therefore access cannot be granted at this time. Fortunately the archive.today link works.
- Ylpertnodi 2y agoAs does my vpn.
- valiant55 2y agoWhy is this a thing?
- throwaway4pp24 2y agoGDPR
- 1vuio0pswjnm7 2y agoWorks where archive.ph is blocked: https://web.archive.org/web/20240720155219/https://www.kiro7.com/news/local/siblings-miss-crucial-life-extending-treatment-seattle-childrens-because-crowdstrike-outage/Y5EMEXGGQJE7PHDEAEKCWG6U4M/ https://web.archive.org/web/20240720155219/https://www.kiro7...