17 ms·

Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

291 points by akehrer 10y ago

callesgg 10y ago
Putting antivirus on equipment at all indicates a much bigger problem. That the equipment is somehow configured to be susceptible to viruses.
- datenwolf 10y ago
  Well at least in EN62304 the installation of AV on medical devices is recommended. The whole thing reads like as if it was written by people who picked up a few buzzwords and read a few articles in a computer magazine.
11thEarlOfMar 10y ago
/rant/ I can't tell you how many times we've chased down field problems that ultimately were the result of antivirus scans. It's been so bad, that one of the first questions we now ask when we get a tool-down report is "is there antivirus running and what is the configuration?" Bringing Windows into the architecture of any type of capital equipment control system is a bane. A scourge. I mean to say, it really is a misappropriation of software. Imagine, "Yeah, Frank only knows VB, so that's what we used for the aircraft's cockpit GUI." /xrant/
- copperx 10y ago
  I would expect these kinds of systems to be running a soft realtime OS. Or at the very least a run of the mill OS with no extraneous software running in the background.
  - Dwolb 10y ago
    This. How are these devices not running on some sort of hardened OS seen in airplanes and automotive? Medical applications are mission critical (or some variant) and should have same (or better!) certification procedures set up for correctness and security.
    - aavotins 10y ago
      Second this. It is terrifying to know that mission critical, medical grade software runs on a consumer operating system. Military/aerospace systems have numerous requirements and clearly defined practices and ways of developing these systems, often going through various layers of documentation and using specifically designed programming languages(like the Z programming language) to write specifications, which are then re-written into code, but it seems like medical industry has been neglected.
      - ktRolster 10y ago
        Then this story about viruses at a nuclear power plant won't make you feel any better: http://www.reuters.com/article/us-nuclearpower-cyber-germany-idUSKCN0XN2OS http://www.reuters.com/article/us-nuclearpower-cyber-germany... Some great quotes: "Mikko Hypponen, chief research officer for Finland-based F-Secure, said that infections of critical infrastructure were surprisingly common" "Hypponen said he had recently spoken to a European aircraft maker that said it cleans the cockpits of its planes every week of malware designed for Android phones. The malware spread to the planes only because factory employees were charging their phones with the USB port in the cockpit."
        zyxley 10y ago
        > The malware spread to the planes only because factory employees were charging their phones with the USB port in the cockpit. The moral of the story: If you include any kind of port in something, people WILL plug things into it sooner or later.
        ams6110 10y ago
        Sounds suspect to me. Aircraft computers are not running Android.
        copperx 10y ago
        I'm pretty sure the entertainment system runs on some version of Android.
- beachstartup 10y ago
  correlation doesn't imply causation.... except when someone selected windows, and then made 50 other moronic decisions also. half the work done in this industry is just dealing with stupid decisions made by stupid people. i just accept this now.
  - tluyben2 10y ago
    But when Windows is chosen because it is easier to find coders who will 'remain within the budget' (cheapcheap) and cannot even make sure the virus scanner doesn't run during procedures or at all it goes a bit too far.
    - pdkl95 10y ago
      "Cheaper" or "remain within the budget" doesn't excuse using inadequate parts that don't meet the design requirements. Unfortunately, this total disregard for safety isn't just software anymore. When we stat skipping lessons that we've know for a looooooonng time (such as why a split bobbin is an important feature in a transformer[1]), we have evidence of a serious need for strongly enforced regulation. [1] https://news.ycombinator.com/item?id=11474730 https://news.ycombinator.com/item?id=11474730
      - tluyben2 10y ago
        I know it doesn't, but unfortunately it is often how it works. More than I want to remember I have seen things like 'but lives depend on this!' or '100s of millions can get lost if this doesn't work!' and yet when the RFPs come back and something like InfoSys is chosen because big name and cheaper than experts in the field. Edit; > we have evidence of a serious need for strongly enforced regulation. Better education? But I guess strongly enforced regulation will force companies to not go for the cheapest solutions they can get away with which in turn will require people with actual knowledge in the field which will require better education, somehow.
        pdkl95 10y ago
        Better education is always a great idea. Unfortunately, regulation becomes a necessary fix for immediate problems. Note that regulation is the nicer option; the other way to force people to get the necessary education is liability, which could get really ugly in the case of medical devices.
- raverbashing 10y ago
  This This machines costs hundreds of thousands of dollars. There should be no excuse for using Windows. None. I would not be surprised if the "antivirus" thing was some PHB requirement
  - Amezarak 10y ago
    Is there a reason to believe that choosing Windows was a bad decision? The bad decision was installing antivirus software. Otherwise, most any modern OS would be fine. This machine probably shouldn't be connected to a network (if it was), the USB ports should be disabled; data can come off on burned CDs, autorun should be disabled, etc. That's how you deal with IA concerns on a standalone mission-critical system, not by installing antivirus.
    - raverbashing 10y ago
      You're right. My bet is that they use windows because they want to save or get something from the machine (either to a USB drive or network) And I would have followed the same steps you mentioned. The machine could work with Windows. Windows CE might have been a better pick, so you can have things like RO filesystem, etc "There is no reason to use Windows" is, as you mentioned, not a bad decision in itself, but since they do it the lazy way, it is awful. Not sure if there's an out-of-the-box way of firewalling everything in the Windows versions available at the time that machine was built
      - enraged_camel 10y ago
        A lot of those machines have Windows embedded in them, because a lot of doctors and nurses are really, really bad with technology and cannot use anything more complicated than Windows's familiar user interface. Training involves "OK, now use the mouse and double-click this icon on the desktop to start the program." I wish I was kidding.
        stephengillie 10y ago
        Windows is a familiar interface for computers. Is there a good reason for not using a familiar user interface?
- rwmj 10y ago
  And if you want PCI-DSS [credit card handling] certification, then you'd better be running AV software, even when it's completely inappropriate.
- rsync 10y ago
  "Bringing Windows into the architecture of any type of capital equipment control system is a bane. A scourge." We need to go much further than this, since many people will "solve" this problem by using a different platform than Windows. In reality, bringing networking of any type into capital equipment control systems or critical infrastructure is the bane ... the scourge. Whatever convenience or perceived function that networking (including very local networking, like USB) is dramatically outweighed by the additional attack surface. Go back to sneakernet and check your facebook at home, Mr. Nuclear Plant Worker.
  - CaptSpify 10y ago
    The issue with this is: Many of these systems need to send data to other system on the network. You'll have to send the data over some kind of PACS network, or you'll just have to use a USB. I agree that's a "better" setup than networking, but I still don't think having staff plug random usb devices into your medical equipment is a great idea either.
mtgx 10y ago
Microsoft must be relieved this wasn't yet another Windows 10 upgrade horror story.
YeGoblynQueenne 10y ago
>> The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure. >> The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side. So "RTFM"? Not very helpful.
GunboatDiplomat 10y ago
Why on earth is medical equipment running standard Windows? This is the ideal location for some basic RTOS or even just an embedded Linux. Seems like a huge cost and risk for no gain.
- riyadparvez 10y ago
  That was my first thought too. And why there is an anti-virus running? This equipment should not be connected to the internet nor some staff should plug-in a flash drive on the first place.
  - ars 10y ago
    That's not true. They have to keep a record of the data for the patient file. So it does have to communicate remotely in some fashion.
    - GunboatDiplomat 10y ago
      Could communicate with a gateway bridging a private medical devices network and a public network. That seems a reasonable way to provide control and access.
- tluyben2 10y ago
  It's cheap to find Windows programmers and even cheaper to find ones that are not hindered by knowledge about software quality and safety. That's not their fault; no-one ever told them something like that exists.
  - nonbel 10y ago
    >"hindered by knowledge" This is a great phrase. "Joe was hindered by knowledge of what a p-value means and so didn't claim he discovered a key to understanding the disease."
- kosmic_k 10y ago
  I never realized just how lucky I was to have been born when I was. I can't imagine building embed devices which aren't running a very simply super loop or an ARM RTOS.
kinai 10y ago
this reminds me of IT crowds bomb disposal robot: https://www.youtube.com/watch?v=z88b96ECZCE https://www.youtube.com/watch?v=z88b96ECZCE just perfect
CaptSpify 10y ago
https://xkcd.com/463/ https://xkcd.com/463/ The whole structure is wrong. I used to work in medical equipment repair. Windows Embedded is running so many devices it's not funny. But it's not just Windows that's the problem. I put a linux-system on a PACS network to diagnose equipment. It was a headless, and we asked the IT group to block it off from the Internet. Hospital IT: "Does it have antivirus?" Me: "..."
- SixSigma 10y ago
  List of FDA medical equipment recalls for 2016 http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm480134.htm http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm48... At least three of them are Class 1 - May cause death And all of those are software related, none run Windows http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm481966.htm http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm48... http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm489108.htm http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm48... http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm485790.htm http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm48...
  - kosmic_k 10y ago
    That is astoundingly horrifying, especially the Class 1's which were distributed for over five years.
    - SixSigma 10y ago
      I read every recall, food and medical, from 2000-2015 for a university research project. tbh I'm surprised anyone is still alive !
    - icegreentea 10y ago
      The bar for recall is actually relatively low. Basically, when you find a fault (somehow, maybe in regular QC in manufacturing, or something bad actually happens in the field, or some engineer is fucking around), the question is "can this affect patient safety/outcome in the field". If the answer is at all not a certain "no", then you're probably thinking recall at that point, unless you can adequately root cause and contain it. Since the nature of "oh oops" is that they tend to affect systems in ways that are not anticipated, there's often insufficient evidence to rule out danger to patients, and therefore there's a recall. For example, if you sold 10,000 diagnostics machines, and then discovered that because of stack up of tolerances in electrical components, something like 1 in 100,000 machines will have a fault that affects the customer safety. However, because your original analysis (during design phase) did not show this problem, you never bothered recording the actual performance characteristics of 40% of the components involved in the stack up. Now you're in a pretty awkward situation, that could result in a recall. And it could very well by that all 100,000 machines sold are just fine.
pdkl95 10y ago
Is it going to take more deaths to convince people to learn from the Therac-25[1]? If you aren't designing for safety first, you have no business working on medical devices or anything else that might be a dangerous when it misbehaves. [1] http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf
- chestervonwinch 10y ago
  I am not the parent poster, but may I ask why is this comment being down-voted? I'm not speaking for the parent, but he or she seems to be implying that medical equipment with anti-virus software with automatic updates (used as such) may potentially compromise a patient's safety, and may be indicative of further bad design practices, which could result in, at worst, death. Is this somehow off-topic, or not worthy of discussion?
  - pdkl95 10y ago
    That's exactly right. The article mentions that the doctors were fortunate enough to have five minutes during which they could reboot the device. If they were in the middle of some other procedure that had tighter time constraints, a reboot could have easily killed the patient. Just like the Therac-25, this isn't about a single problem (the antivirus or the race condition in the Therac-25's software). Designing for safety has to happen at all levels of design. Using Windows (or Linux, or any other complex OS) in a medical device shows that the designer wasn't even considering the safety of major parts of their design. Designing medical devices with an OS that can be infected with malware (and thus need an antivirus) is the same kind of idiocy that puts a car's steering and brakes on the same CAN bus as the music player and emergency radio. It's a sign that the designer needs either more education or a different job before someone is injured or killed.
  - jschwartzi 10y ago
    Because it's really disingenuous to say that the medical device industry hasn't learned anything from Therac 25. The concept of two-fault failures is an industry standard that was learned from Therac. The fact is that in the Medical software industry the best practice is to manage the entire software configuration of the medical device. Failing to do so, and especially failing to adhere to the guidelines of the manufacturer, is negligent at best. We all know that the behavior that led to the hazard is the wrong thing to do and that somebody screwed up. The only other real insight that can be gained from this incident is that it's very important to have configuration management procedures that are easy to follow, and it's important to verify that they were correctly followed. I can't tell whether they were in this case, but I suspect given the use of Off-the-shelf software that there was some manual sequence of steps required to adhere to the approved configuration. Given that, I would have expected an error of this magnitude, because it's well known that humans make mistakes whenever they are made to follow a sequence of steps. The configuration should have been verified at installation time, at least. If you're interested in the kinds of things the industry has to consider in the US, take a look at the FDA guidance for the 510k submittal process.
steven2012 10y ago
Antivirus scans are one of those things added on IT checklists to cover their ass whenever something wrong happens. But it rarely is useful. It only causes problems. We've seen so many issues related to virus scans throughout the years it's crazy. What's better is to lock down the servers with only minimal access. I haven't used virus scan on my main desktop for over 10 years because I don't click on weird emails and I don't go to sketchy websites ever. Sure there's the risk of malware from ads I suppose, but I'm not that worried.
- jconley 10y ago
  Most of the time IT is just implementing policy from the CIO, which is basing it on the requirements of the company's insurers. Insurance companies require some very annoying things like Anti-virus. It's like having a lock on your office. You do it so the insurance company will pay you if someone comes in and steals your stuff.
  - AnthonyMouse 10y ago
    It's more like the requirements cronies put into defense contracts to make sure the contractors make a lot of money. The reason "security requirements" documents require antivirus is that companies like Symantec make sure they're in the right position to be the ones asked when someone is writing up a security requirements document, so that their answer can be "make sure you install antivirus (and here's the contact info for our volume licensing center)."
- deleted 10y ago
  [deleted]
- jcrawfordor 10y ago
  Yeah, you don't click on weird emails and don't go to sketchy websites. Try managing IT security for an enterprise of 10,000 employees. A/V will save your ass hundreds of times every single day. Computer professionals rarely understand the use case for A/V precisely because they are not the use case. In most all applications, A/V serves first as a safeguard against stupid user behavior, and only second as a safeguard against more advanced penetration (and in the latter case, one with only rare success). I'd bet that the #1 way enterprises are getting breached is still malicious email attachments, that's certainly true in my experience.
deleted 10y ago
[deleted]
ezoe 10y ago
This situation is even funnier(and sadly very seriously flawed) in Japan. Medical equipment require an authorization to use. Any change to the medical equipment requires another authorization or it's prohibited. By "any change" , it includes Windows Update(it changes the system obviously). The result: they use anti-malware software to protect(or rather, believed to protect) unpatched Windows. At least one anti-malware software company(Trend Micro), marketing that their software can protect the medical equipment in such situation.
- exhilaration 10y ago
  But... what about AV/malware definition updates? Doesn't that fall under "any change"?
  - symtos 10y ago
    and what about security updates to the snakeoil they sell, eg. https://bugs.chromium.org/p/project-zero/issues/detail?id=693 https://bugs.chromium.org/p/project-zero/issues/detail?id=69...
- Blackthorn 10y ago
  > Any change to the medical equipment requires another authorization or it's prohibited. Honestly, this isn't a bad decision. If the device was tested and certified with specific software, a software upgrade is not guaranteed to not cause a problem.
  - symtos 10y ago
    using software with known problems in order to avoid potential problems from an upgrade does not seem like a non-bad decision
    - Blackthorn 10y ago
      Is the medical device working right now? Yes. Could, upon upgrading, the device stop working, possibly in a subtle way that might kill somebody? Yes. The approval process for medical devices is rightfully difficult. Software upgrades, even if they seem trivial, should not be a backdoor process of bypassing testing and approval.
Kristine1975 10y ago
Why is there a virus scanner on a PC inside the operating room? Don't tell me that PC is connected to the internet...
- rs999gti 10y ago
  I was going to ask this as well. Why does this PC need to be connected to the internet? If it doesn't need to phone home while operating as a heart monitor then there is no need to have antivirus or have this PC connected to the internet. Also, plenty of devices not connected to the internet run Windows: ATM's, Billboard, Monitors, etc. Dumb IT is to blame for this mistake.
  - jcrawfordor 10y ago
    > Also, plenty of devices not connected to the internet run Windows: ATM's, Billboard, Monitors, etc. I hate to break it to you, but, in practice... these things are all typically connected to the internet.
  - mirimir 10y ago
    The need to get updates, I bet.
coldcode 10y ago
I worked at a financial company that ran its production Oracle database servers on Windows in the same network as the staff (no firewall) and ran virus checkers on them. Performance was terrible of course.
angersock 10y ago
Okay, seriously, I need to say something, because I doubt most of the people commenting in this thread have ever dealt with either health IT, healthcare software, or any of the related nonsense. There are kinda four flavors of machine setup I ran into while in that field: big server banks for on-site hosting (think huge enterprise VM farms, for data warehousing and record storage and virtual desktop hosting), care provider systems (think like tablets, doctor office computers, nurse workstations, room workstations), cart computers (used for things like running the sonogram or cardiogram equipment, or for other studies), and actual integrated devices (for, say, data collection). The care provider systems are usually comically locked-down, tablets and phones having the meanest management software they can (no apps, limited connectivity, remote wiping, and so forth). Workstations tend to be centrally managed, have images pushed regularly (ha!), and often use AD and smartcards to handle authentication. One place I've seen took this a step further, and basically just booted users directly into a VM hosted on the server farms mentioned earlier. You can't use USB devices, you have highly-regulated clipboard access, and so forth--this is done to prevent HIPAA breaches. Which is kinda silly given other workarounds, but whatever makes people feel safe and the CIO happy. These workstations run some enterprise version of Windows, probably 7 Pro. Those silly-long extended service agreements you see on Microsoft? Hospitals are some of the people keeping that alive, and they will pay obnoxious amounts of money for the privilege. The cart computers are typically like the workstations in terms of functionality, but they may have software specific to the device they're talking to. They might not be as locked down (e.g., only acting as thin clients to a remote VM), but they are still running Windows. The device computers may run some kind of RTOS. In some cases, they'll be running a customized Windows CE installation--which is totally reasonable. There are a lot of good guarantees that that can give a development shop, least of all that they can call up Microsoft instead of StackOverflow and say "Hey, this function does x, it's documented as y, and we're paying you a lot of money, so what the fuck?". Windows Embedded (which is I think the successor, am not sure). In all of these cases, Windows itself works pretty damned well. It runs the software everybody needs, it has the enterprise deployment stuff figured out through decades of improvement, and really there is no reason to be scoffing at its choice. Now, if folks have goofed up and thrown a stupid AV policy on the machine, that's a different question entirely. Health IT is full to the brim of people basically just punching a clock and being unable to get anything done in a reasonable amount of time. Sometimes, they do awesome things, but mainly they are just custodians standing between doctors and really really stupid policy decisions that seemed good at the time. EDIT: Removed unrelated example at top.
iask 10y ago
There are a couple of things here from my POV, first - I would replace the head of their IT and any senior IT staff - who seem to look for the quickest-then-cheap solutions. Dumb ducks who don't spend the extra time understanding the importance of the infrastructure and the software they install. And also replace the service vendor, if they have one. I've seen this happen time and again, where companies have some 3rd party service vendors who would install AV software on anything they can get their hand on, even a microwave or coffee machine - just to tell the client "my bill is expensive, but you can feel secure, we installed AV". I despise these folks with a passion. The problem is not Windows. It's a lack of knowledge and understanding. Simple. For god's sake - it's 2016 - dump the Anti Virus software. I am gonna make t-shirts this summer with this ;)
- technion 10y ago
  I would replace the head of their IT and any senior IT staff It's a very good bet the senior IT team were following orders from somewhere else in the chain here.
fencepost 10y ago
I see a bunch of folks talking about whether PCs are connected to the Internet and "why was it running antivirus in the first place?" It's called Defense in Depth. It Does Not Matter if the device is connected to/able to reach the Internet. First, it probably can reach the Internet in some way simply by being networked. I don't think I've ever seen a medical office (can't speak about hospitals) where medical diagnostic equipment was on a fully-separate network able only to talk to other network equipment and specified data destinations (PACS servers). Second, I'm not concerned about unpatched, unprotected machines being infected from the Internet. Odds are they're running a restricted version of Windows, with a custom shell and a lot of stuff stripped out. I'm concerned that they're going to be infected by another machine on the network that's gotten infected. With all the past SQL Server security issues a decade or more ago, how many people think those SQL Server boxes could be directly reached from outside the local network? The conjunction of those two is that even if you firewall all that stuff off, the PACS servers are still on both networks, and are probably running much more interesting and vulnerable stuff than the device controllers. Sure you can fully wall everything off - it's really easy, just do your X-rays onto film, burn your MRIs and ultrasounds onto CDs, and print your EKGs for later scanning. Oh, and listen to people complain about how out-of-date your systems and procedures are. There are other factors that come in as well - sure, every device manufacturer could provide fully bespoke diagnostic displays developed from the ground up in artisanal software shops providing full employment for assembly programmers working on embedded systems, along with cohorts of graphic designers creating glorious steampunk-styled interfaces. That's a beautiful dream, keep having it. For the rest of the world, creating a UI on that custom embedded system running on something from RIM/Blackberry (yeah, they own QNX) is just going to get them crap from people because of A) how clunky it probably looks and B) How could they even consider allowing direct user interaction with the RTOS that was chosen to ensure that the dangerous bits in contact with patients/radiation/irradiated patients were safe? There's a beautiful world out there somewhere where everything is safe and secure and seamless and updated. The rest of us live in worlds where Joe in Marketing's PC gets infected with something that allows an attacker to start scanning the network for unpatched vulnerabilities on any system, which leads to an out-of-date install of IIS on a legacy server that hasn't been updated because there's no longer a contract with the vendor (or no vendor) but it's around because there's a statutory requirement to keep the data on that system for 7-10 years. There's a lot of ugliness out there. Antivirus is a way to try to ensure that when (not if) some of it hits you the repercussions are minimized.
fla 10y ago
How can a medical device be certified for running on 'user hardware' (=uncontrolled environment). Something is probably missing from the article. IMO, the device in question wasn't critical at all, and a failure could be expected.
fiatjaf 10y ago
> The problem is not Windows. It's a lack of knowledge and understanding. Simple. Yes, if there was a minimum of knowledge Windows wouldn't have been used.
- dlp211 10y ago
  Get out of here with that nonsense. You may not be a fan of paid software, but the Windows Kernel is just as good as any FOSS kernel today in regards to stability. The Server and embedded SKUs also come with a ton of the extraneous stuff removed that one would ever be worried about. The issue here is that someone decided that a machine that should only ever be connected to an air gapped network needed anti-virus software. Disclaimer: I work for Microsoft.
  - wyager 10y ago
    >the Windows Kernel is just as good as any FOSS kernel today in regards to stability. I disagree strongly, but this is beside the point. Medical hardware should not be using any operating system that's not hard real-time and thoroughly vetted. Ideally they would use no OS at all. Even Linux, which is drastically more appropriate for embedded systems, is a questionable choice for medical equipment.
  - zxcvcxz 10y ago
    > the Windows Kernel is just as good as any FOSS kernel today in regards to stability. No it's really not. I've used Windows and Linux a lot and in 10 years of using Linux I've only twice had a kernel panic while using non-experimental software. On Windows I've had countless BSODs.
    - dlp211 10y ago
      I'm guessing that your use cases for Windows and FOSS kernels are different, but I'll counter your anecdote with some of my own. Almost every single BSOD today is a hardware/driver problem. After Server 2003, MSFT got serious about kernel stability. I haven't had a BSOD from a kernel problem since Vista. Secondly, I also run multiple Server 2012 R2 servers with server hardware, and have never had a BSOD with any of those machines. Finally, Microsoft runs Azure on Win Server, if it wasn't stable, it's cloud platform would be in serious trouble. Disclaimer: I still work for Microsoft
Avernar 10y ago
"Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly. The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side." Here's how I read that: The programmers of this piece of software assumed that some I/O operation would never fail and when it does the program shits itself. So instead of hardening their software to withstand loss of telemetry gracefully, which would cost time and money for the company, they just give instructions to disable scans on their folder. Odds are good that somewhere this scan will happen (and it did). Either IT doesn't read the release notes or goofs the configuration or an antivirus update clears the white list. Might not even be the antivirus that interferes with the telemetry briefly. But instead of having resilient software it's "the anitvirus software's fault" or "it's IT's fault" when something goes wrong because of their bad management/engineering decision.
- sillysaurus3 10y ago
  Software shouldn't necessarily try to account for errors in that manner. Usually, the most graceful thing to do is to exit cleanly. For example, if there is a massive amount of data, it has to be stored on disk. It's too large to keep in memory. And if the point of the program is to transform that data in real time, then it has to have access to the disk. The antivirus basically unplugged the disk. What can it do to recover? There's nothing to be done. It should be able to survive that situation, of course. When the disk is plugged back in, it should be able to restart without any problems. But I think that's a different kind of resiliency than what you're referring to. In this case, the only way to recover would be to copy the frozen data to a new area of the hard drive, assuming it retained read access. But such complexities result in brittle implementations, prone to acquiring bugs. What if the disk space runs out? So you check beforehand whether there's enough space. But what if some other program starts consuming disk space in the middle of your copy operation? And so on. It's an endless spiral of design complexity. The situation in the article seems closer to hardware failure than a design oversight.
saganus 10y ago
Wtf? "The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure." Who configures an antiviurs for an hourly scan on a doctor's computer?
- pritambaral 10y ago
  It wasn't even a doctor's computer, it was apparently an operating-room equipment computer.
malbs 10y ago
We've had issues with the latest versions of kaspersky. A burst of network activity is almost guaranteed to crash a machine. It took us a while to isolate Kaspersky 10, and it's not even any particular component inside of Kaspersky, but only when all features are enabled. We tried different permutations of features to try and isolate the cause of our crashes, but as soon as you have any one feature disabled in, the crashes stop, Very frustrating because ultimately our clients laid the blame at my feet (new software feature, new release, blah blah blah), and not exactly much you can do in the way of hardening against this particular crash, the app generates a burst of network data, and boom, blue screen/instant reboot.
dchichkov 10y ago
Let me surprise you, with the code quality that sometimes is running in what is actually 'life-critical' software. Back in the nineties, I wrote a nice piece of some 300kb of C code, for DOS/x86. It was a complete software package, controlling medical equipment that was testing speed of blood coagulation. These tests are crucial in the patient post-operation recovery. This piece of C code had some hardware control code, some statistics, a bit of math, some visualisation, GUI, etc. Normally, you'd imagine a team of 2-3 people, carefully written test cases, dedicated QA person, and a year of time to write something like it. And independend lab, that would certify the thing. Well... in that case, yes, there was independent certification... but... It was just one developer, and I was 13, when I wrote it ;) During after-school time, in around 4-6 months. And I must say, I still sometimes have chills, when I think of the code quality, and, um, unorthodox solutions of 13-year-old myself. Yes, I've had some years of experience at the time, both writing software and designing hardware, and advice from my parents, who both could write software. But, at the time, I've had zero formal training, aside from reading K&R and PC XT manuals ;). So, you might imagine the code quality ;) Even, no need to imagine, I actually still have it somewhere in the archives :)
- state 10y ago
  I understand why you probably don't want to put it up, but boy would it be fun to look at the code you're describing.
  - dchichkov 10y ago
    I probably will put it up. It's a nice inspirational story for teens out here. Doubt there'd be any repercussions, no one cares about some random code on GitHub. And the equipment is hopefully taken out of service years ago, it was more than 20 years back. I wish I knew how long it had been used, but there've been only about 10-20 units sold, I think. I vaguely remember adding extra features for a year or so (like adding support for HP laserjet printer). But one of the founders of the company (on the business side) had some health problems, and I guess that had played role in very small number of units sold. The only feedback that I've had, is pretty much that my father took me to a lab once, that had a unit deployed, for a support call. And I've seen some real printouts with patient names, from the unit. The lab assistant seemed to be happy with the device. I remember them showing me some blood plasma and teaching me to count cells, during lab tour ;)
combatentropy 10y ago
> they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side. And the hospital included full instructions to the software company on how to properly perform a heart transplant, so they were baffled why the programmer just let his teammate die of heart failure. Come on, this kind of stuff should be a zero-configuration hardware-based black box, with its own buttons, screen, etc. --- not something that needs to be (or even can be) connected to something outside the vendor's total control.
firebones 10y ago
For what it is worth, Merge is now part of IBM Watson. http://www.merge.com/News/Article.aspx?ItemID=660 http://www.merge.com/News/Article.aspx?ItemID=660 Welcome to the Health Cloud Powered by Watson.
stevetrewick 10y ago
From the linked report : Based upon the available information, the cause for the reported event was due to the customer not following instructions concerning the installation of anti-virus software; therefore, there is no indication that the reported event was related to product malfunction or defect I beg to differ. I'd consider a momentary loss of file I/O due to lock contention causing a machine to require a reboot a shocking defect in - say - a word processor (which, notably, do not have this problem). That this risk is apparently known and the vendor's sole mitigation is to document a 'Don't do that then' is absolutely 100% an indication of a product defect, even in the absence of an actual occurrence.
billforsternz 10y ago
"A critical medical equipment crashed during a heart procedure due to a timely scan triggered by the antivirus software installed on the PC to which the said device was sending data for logging and monitoring." That should be untimely. The opposite of timely.
toomanythings2 10y ago
I don't think they say this device is controlled by Windows but it must be. Why professional software and instruments even consider using Windows is beyond me.
tmptmp 10y ago
>>The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side. These Merge Hemo fucktards should learn a bit about systems design from Margaret Hamilton [1] [2] before they start coding for such life critical software. Another point is no closed source software (including the greatest ones like Windows) should be allowed to use in such institutions. [1] http://www.nasa.gov/home/hqnews/2003/sep/HQ_03281_Hamilton_Honor.html http://www.nasa.gov/home/hqnews/2003/sep/HQ_03281_Hamilton_H... [2] https://en.m.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%29 https://en.m.wikipedia.org/wiki/Margaret_Hamilton_%28scienti... edit: added point about closed-source software
vilniuse 10y ago
Just like any other software package... :(

291 points by akehrer 10y ago

Putting antivirus on equipment at all indicates a much bigger problem. That the equipment is somehow configured to be susceptible to viruses.

datenwolf 10y ago
Well at least in EN62304 the installation of AV on medical devices is recommended. The whole thing reads like as if it was written by people who picked up a few buzzwords and read a few articles in a computer magazine.

11thEarlOfMar 10y ago

/rant/ I can't tell you how many times we've chased down field problems that ultimately were the result of antivirus scans. It's been so bad, that one of the first questions we now ask when we get a tool-down report is "is there antivirus running and what is the configuration?" Bringing Windows into the architecture of any type of capital equipment control system is a bane. A scourge. I mean to say, it really is a misappropriation of software. Imagine, "Yeah, Frank only knows VB, so that's what we used for the aircraft's cockpit GUI." /xrant/

copperx 10y ago
I would expect these kinds of systems to be running a soft realtime OS. Or at the very least a run of the mill OS with no extraneous software running in the background.
- Dwolb 10y ago
  This. How are these devices not running on some sort of hardened OS seen in airplanes and automotive? Medical applications are mission critical (or some variant) and should have same (or better!) certification procedures set up for correctness and security.
  - aavotins 10y ago
    Second this. It is terrifying to know that mission critical, medical grade software runs on a consumer operating system. Military/aerospace systems have numerous requirements and clearly defined practices and ways of developing these systems, often going through various layers of documentation and using specifically designed programming languages(like the Z programming language) to write specifications, which are then re-written into code, but it seems like medical industry has been neglected.
    - ktRolster 10y ago
      Then this story about viruses at a nuclear power plant won't make you feel any better: http://www.reuters.com/article/us-nuclearpower-cyber-germany-idUSKCN0XN2OS http://www.reuters.com/article/us-nuclearpower-cyber-germany... Some great quotes: "Mikko Hypponen, chief research officer for Finland-based F-Secure, said that infections of critical infrastructure were surprisingly common" "Hypponen said he had recently spoken to a European aircraft maker that said it cleans the cockpits of its planes every week of malware designed for Android phones. The malware spread to the planes only because factory employees were charging their phones with the USB port in the cockpit."
      - zyxley 10y ago
        > The malware spread to the planes only because factory employees were charging their phones with the USB port in the cockpit. The moral of the story: If you include any kind of port in something, people WILL plug things into it sooner or later.
      - ams6110 10y ago
        Sounds suspect to me. Aircraft computers are not running Android.
        copperx 10y ago
        I'm pretty sure the entertainment system runs on some version of Android.
beachstartup 10y ago
correlation doesn't imply causation.... except when someone selected windows, and then made 50 other moronic decisions also. half the work done in this industry is just dealing with stupid decisions made by stupid people. i just accept this now.
- tluyben2 10y ago
  But when Windows is chosen because it is easier to find coders who will 'remain within the budget' (cheapcheap) and cannot even make sure the virus scanner doesn't run during procedures or at all it goes a bit too far.
  - pdkl95 10y ago
    "Cheaper" or "remain within the budget" doesn't excuse using inadequate parts that don't meet the design requirements. Unfortunately, this total disregard for safety isn't just software anymore. When we stat skipping lessons that we've know for a looooooonng time (such as why a split bobbin is an important feature in a transformer[1]), we have evidence of a serious need for strongly enforced regulation. [1] https://news.ycombinator.com/item?id=11474730 https://news.ycombinator.com/item?id=11474730
    - tluyben2 10y ago
      I know it doesn't, but unfortunately it is often how it works. More than I want to remember I have seen things like 'but lives depend on this!' or '100s of millions can get lost if this doesn't work!' and yet when the RFPs come back and something like InfoSys is chosen because big name and cheaper than experts in the field. Edit; > we have evidence of a serious need for strongly enforced regulation. Better education? But I guess strongly enforced regulation will force companies to not go for the cheapest solutions they can get away with which in turn will require people with actual knowledge in the field which will require better education, somehow.
      - pdkl95 10y ago
        Better education is always a great idea. Unfortunately, regulation becomes a necessary fix for immediate problems. Note that regulation is the nicer option; the other way to force people to get the necessary education is liability, which could get really ugly in the case of medical devices.
raverbashing 10y ago
This This machines costs hundreds of thousands of dollars. There should be no excuse for using Windows. None. I would not be surprised if the "antivirus" thing was some PHB requirement
- Amezarak 10y ago
  Is there a reason to believe that choosing Windows was a bad decision? The bad decision was installing antivirus software. Otherwise, most any modern OS would be fine. This machine probably shouldn't be connected to a network (if it was), the USB ports should be disabled; data can come off on burned CDs, autorun should be disabled, etc. That's how you deal with IA concerns on a standalone mission-critical system, not by installing antivirus.
  - raverbashing 10y ago
    You're right. My bet is that they use windows because they want to save or get something from the machine (either to a USB drive or network) And I would have followed the same steps you mentioned. The machine could work with Windows. Windows CE might have been a better pick, so you can have things like RO filesystem, etc "There is no reason to use Windows" is, as you mentioned, not a bad decision in itself, but since they do it the lazy way, it is awful. Not sure if there's an out-of-the-box way of firewalling everything in the Windows versions available at the time that machine was built
    - enraged_camel 10y ago
      A lot of those machines have Windows embedded in them, because a lot of doctors and nurses are really, really bad with technology and cannot use anything more complicated than Windows's familiar user interface. Training involves "OK, now use the mouse and double-click this icon on the desktop to start the program." I wish I was kidding.
      - stephengillie 10y ago
        Windows is a familiar interface for computers. Is there a good reason for not using a familiar user interface?
rwmj 10y ago
And if you want PCI-DSS [credit card handling] certification, then you'd better be running AV software, even when it's completely inappropriate.
rsync 10y ago
"Bringing Windows into the architecture of any type of capital equipment control system is a bane. A scourge." We need to go much further than this, since many people will "solve" this problem by using a different platform than Windows. In reality, bringing networking of any type into capital equipment control systems or critical infrastructure is the bane ... the scourge. Whatever convenience or perceived function that networking (including very local networking, like USB) is dramatically outweighed by the additional attack surface. Go back to sneakernet and check your facebook at home, Mr. Nuclear Plant Worker.
- CaptSpify 10y ago
  The issue with this is: Many of these systems need to send data to other system on the network. You'll have to send the data over some kind of PACS network, or you'll just have to use a USB. I agree that's a "better" setup than networking, but I still don't think having staff plug random usb devices into your medical equipment is a great idea either.

mtgx 10y ago

Microsoft must be relieved this wasn't yet another Windows 10 upgrade horror story.

YeGoblynQueenne 10y ago

>> The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure. >> The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side. So "RTFM"? Not very helpful.

GunboatDiplomat 10y ago

Why on earth is medical equipment running standard Windows? This is the ideal location for some basic RTOS or even just an embedded Linux. Seems like a huge cost and risk for no gain.

riyadparvez 10y ago
That was my first thought too. And why there is an anti-virus running? This equipment should not be connected to the internet nor some staff should plug-in a flash drive on the first place.
- ars 10y ago
  That's not true. They have to keep a record of the data for the patient file. So it does have to communicate remotely in some fashion.
  - GunboatDiplomat 10y ago
    Could communicate with a gateway bridging a private medical devices network and a public network. That seems a reasonable way to provide control and access.
tluyben2 10y ago
It's cheap to find Windows programmers and even cheaper to find ones that are not hindered by knowledge about software quality and safety. That's not their fault; no-one ever told them something like that exists.
- nonbel 10y ago
  >"hindered by knowledge" This is a great phrase. "Joe was hindered by knowledge of what a p-value means and so didn't claim he discovered a key to understanding the disease."
kosmic_k 10y ago
I never realized just how lucky I was to have been born when I was. I can't imagine building embed devices which aren't running a very simply super loop or an ARM RTOS.

kinai 10y ago

this reminds me of IT crowds bomb disposal robot: https://www.youtube.com/watch?v=z88b96ECZCE https://www.youtube.com/watch?v=z88b96ECZCE just perfect

CaptSpify 10y ago

https://xkcd.com/463/ https://xkcd.com/463/ The whole structure is wrong. I used to work in medical equipment repair. Windows Embedded is running so many devices it's not funny. But it's not just Windows that's the problem. I put a linux-system on a PACS network to diagnose equipment. It was a headless, and we asked the IT group to block it off from the Internet. Hospital IT: "Does it have antivirus?" Me: "..."

SixSigma 10y ago
List of FDA medical equipment recalls for 2016 http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm480134.htm http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm48... At least three of them are Class 1 - May cause death And all of those are software related, none run Windows http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm481966.htm http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm48... http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm489108.htm http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm48... http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm485790.htm http://www.fda.gov/MedicalDevices/Safety/ListofRecalls/ucm48...
- kosmic_k 10y ago
  That is astoundingly horrifying, especially the Class 1's which were distributed for over five years.
  - SixSigma 10y ago
    I read every recall, food and medical, from 2000-2015 for a university research project. tbh I'm surprised anyone is still alive !
  - icegreentea 10y ago
    The bar for recall is actually relatively low. Basically, when you find a fault (somehow, maybe in regular QC in manufacturing, or something bad actually happens in the field, or some engineer is fucking around), the question is "can this affect patient safety/outcome in the field". If the answer is at all not a certain "no", then you're probably thinking recall at that point, unless you can adequately root cause and contain it. Since the nature of "oh oops" is that they tend to affect systems in ways that are not anticipated, there's often insufficient evidence to rule out danger to patients, and therefore there's a recall. For example, if you sold 10,000 diagnostics machines, and then discovered that because of stack up of tolerances in electrical components, something like 1 in 100,000 machines will have a fault that affects the customer safety. However, because your original analysis (during design phase) did not show this problem, you never bothered recording the actual performance characteristics of 40% of the components involved in the stack up. Now you're in a pretty awkward situation, that could result in a recall. And it could very well by that all 100,000 machines sold are just fine.

pdkl95 10y ago

Is it going to take more deaths to convince people to learn from the Therac-25[1]? If you aren't designing for safety first, you have no business working on medical devices or anything else that might be a dangerous when it misbehaves. [1] http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf

chestervonwinch 10y ago
I am not the parent poster, but may I ask why is this comment being down-voted? I'm not speaking for the parent, but he or she seems to be implying that medical equipment with anti-virus software with automatic updates (used as such) may potentially compromise a patient's safety, and may be indicative of further bad design practices, which could result in, at worst, death. Is this somehow off-topic, or not worthy of discussion?
- pdkl95 10y ago
  That's exactly right. The article mentions that the doctors were fortunate enough to have five minutes during which they could reboot the device. If they were in the middle of some other procedure that had tighter time constraints, a reboot could have easily killed the patient. Just like the Therac-25, this isn't about a single problem (the antivirus or the race condition in the Therac-25's software). Designing for safety has to happen at all levels of design. Using Windows (or Linux, or any other complex OS) in a medical device shows that the designer wasn't even considering the safety of major parts of their design. Designing medical devices with an OS that can be infected with malware (and thus need an antivirus) is the same kind of idiocy that puts a car's steering and brakes on the same CAN bus as the music player and emergency radio. It's a sign that the designer needs either more education or a different job before someone is injured or killed.
- jschwartzi 10y ago
  Because it's really disingenuous to say that the medical device industry hasn't learned anything from Therac 25. The concept of two-fault failures is an industry standard that was learned from Therac. The fact is that in the Medical software industry the best practice is to manage the entire software configuration of the medical device. Failing to do so, and especially failing to adhere to the guidelines of the manufacturer, is negligent at best. We all know that the behavior that led to the hazard is the wrong thing to do and that somebody screwed up. The only other real insight that can be gained from this incident is that it's very important to have configuration management procedures that are easy to follow, and it's important to verify that they were correctly followed. I can't tell whether they were in this case, but I suspect given the use of Off-the-shelf software that there was some manual sequence of steps required to adhere to the approved configuration. Given that, I would have expected an error of this magnitude, because it's well known that humans make mistakes whenever they are made to follow a sequence of steps. The configuration should have been verified at installation time, at least. If you're interested in the kinds of things the industry has to consider in the US, take a look at the FDA guidance for the 510k submittal process.

steven2012 10y ago

Antivirus scans are one of those things added on IT checklists to cover their ass whenever something wrong happens. But it rarely is useful. It only causes problems. We've seen so many issues related to virus scans throughout the years it's crazy. What's better is to lock down the servers with only minimal access. I haven't used virus scan on my main desktop for over 10 years because I don't click on weird emails and I don't go to sketchy websites ever. Sure there's the risk of malware from ads I suppose, but I'm not that worried.

jconley 10y ago
Most of the time IT is just implementing policy from the CIO, which is basing it on the requirements of the company's insurers. Insurance companies require some very annoying things like Anti-virus. It's like having a lock on your office. You do it so the insurance company will pay you if someone comes in and steals your stuff.
- AnthonyMouse 10y ago
  It's more like the requirements cronies put into defense contracts to make sure the contractors make a lot of money. The reason "security requirements" documents require antivirus is that companies like Symantec make sure they're in the right position to be the ones asked when someone is writing up a security requirements document, so that their answer can be "make sure you install antivirus (and here's the contact info for our volume licensing center)."
deleted 10y ago
[deleted]
jcrawfordor 10y ago
Yeah, you don't click on weird emails and don't go to sketchy websites. Try managing IT security for an enterprise of 10,000 employees. A/V will save your ass hundreds of times every single day. Computer professionals rarely understand the use case for A/V precisely because they are not the use case. In most all applications, A/V serves first as a safeguard against stupid user behavior, and only second as a safeguard against more advanced penetration (and in the latter case, one with only rare success). I'd bet that the #1 way enterprises are getting breached is still malicious email attachments, that's certainly true in my experience.

deleted 10y ago

[deleted]

ezoe 10y ago

This situation is even funnier(and sadly very seriously flawed) in Japan. Medical equipment require an authorization to use. Any change to the medical equipment requires another authorization or it's prohibited. By "any change" , it includes Windows Update(it changes the system obviously). The result: they use anti-malware software to protect(or rather, believed to protect) unpatched Windows. At least one anti-malware software company(Trend Micro), marketing that their software can protect the medical equipment in such situation.

exhilaration 10y ago
But... what about AV/malware definition updates? Doesn't that fall under "any change"?
- symtos 10y ago
  and what about security updates to the snakeoil they sell, eg. https://bugs.chromium.org/p/project-zero/issues/detail?id=693 https://bugs.chromium.org/p/project-zero/issues/detail?id=69...
Blackthorn 10y ago
> Any change to the medical equipment requires another authorization or it's prohibited. Honestly, this isn't a bad decision. If the device was tested and certified with specific software, a software upgrade is not guaranteed to not cause a problem.
- symtos 10y ago
  using software with known problems in order to avoid potential problems from an upgrade does not seem like a non-bad decision
  - Blackthorn 10y ago
    Is the medical device working right now? Yes. Could, upon upgrading, the device stop working, possibly in a subtle way that might kill somebody? Yes. The approval process for medical devices is rightfully difficult. Software upgrades, even if they seem trivial, should not be a backdoor process of bypassing testing and approval.

Kristine1975 10y ago

Why is there a virus scanner on a PC inside the operating room? Don't tell me that PC is connected to the internet...

rs999gti 10y ago
I was going to ask this as well. Why does this PC need to be connected to the internet? If it doesn't need to phone home while operating as a heart monitor then there is no need to have antivirus or have this PC connected to the internet. Also, plenty of devices not connected to the internet run Windows: ATM's, Billboard, Monitors, etc. Dumb IT is to blame for this mistake.
- jcrawfordor 10y ago
  > Also, plenty of devices not connected to the internet run Windows: ATM's, Billboard, Monitors, etc. I hate to break it to you, but, in practice... these things are all typically connected to the internet.
- mirimir 10y ago
  The need to get updates, I bet.

coldcode 10y ago

I worked at a financial company that ran its production Oracle database servers on Windows in the same network as the staff (no firewall) and ran virus checkers on them. Performance was terrible of course.

angersock 10y ago

Okay, seriously, I need to say something, because I doubt most of the people commenting in this thread have ever dealt with either health IT, healthcare software, or any of the related nonsense. There are kinda four flavors of machine setup I ran into while in that field: big server banks for on-site hosting (think huge enterprise VM farms, for data warehousing and record storage and virtual desktop hosting), care provider systems (think like tablets, doctor office computers, nurse workstations, room workstations), cart computers (used for things like running the sonogram or cardiogram equipment, or for other studies), and actual integrated devices (for, say, data collection). The care provider systems are usually comically locked-down, tablets and phones having the meanest management software they can (no apps, limited connectivity, remote wiping, and so forth). Workstations tend to be centrally managed, have images pushed regularly (ha!), and often use AD and smartcards to handle authentication. One place I've seen took this a step further, and basically just booted users directly into a VM hosted on the server farms mentioned earlier. You can't use USB devices, you have highly-regulated clipboard access, and so forth--this is done to prevent HIPAA breaches. Which is kinda silly given other workarounds, but whatever makes people feel safe and the CIO happy. These workstations run some enterprise version of Windows, probably 7 Pro. Those silly-long extended service agreements you see on Microsoft? Hospitals are some of the people keeping that alive, and they will pay obnoxious amounts of money for the privilege. The cart computers are typically like the workstations in terms of functionality, but they may have software specific to the device they're talking to. They might not be as locked down (e.g., only acting as thin clients to a remote VM), but they are still running Windows. The device computers may run some kind of RTOS. In some cases, they'll be running a customized Windows CE installation--which is totally reasonable. There are a lot of good guarantees that that can give a development shop, least of all that they can call up Microsoft instead of StackOverflow and say "Hey, this function does x, it's documented as y, and we're paying you a lot of money, so what the fuck?". Windows Embedded (which is I think the successor, am not sure). In all of these cases, Windows itself works pretty damned well. It runs the software everybody needs, it has the enterprise deployment stuff figured out through decades of improvement, and really there is no reason to be scoffing at its choice. Now, if folks have goofed up and thrown a stupid AV policy on the machine, that's a different question entirely. Health IT is full to the brim of people basically just punching a clock and being unable to get anything done in a reasonable amount of time. Sometimes, they do awesome things, but mainly they are just custodians standing between doctors and really really stupid policy decisions that seemed good at the time. EDIT: Removed unrelated example at top.

iask 10y ago

There are a couple of things here from my POV, first - I would replace the head of their IT and any senior IT staff - who seem to look for the quickest-then-cheap solutions. Dumb ducks who don't spend the extra time understanding the importance of the infrastructure and the software they install. And also replace the service vendor, if they have one. I've seen this happen time and again, where companies have some 3rd party service vendors who would install AV software on anything they can get their hand on, even a microwave or coffee machine - just to tell the client "my bill is expensive, but you can feel secure, we installed AV". I despise these folks with a passion. The problem is not Windows. It's a lack of knowledge and understanding. Simple. For god's sake - it's 2016 - dump the Anti Virus software. I am gonna make t-shirts this summer with this ;)

technion 10y ago
I would replace the head of their IT and any senior IT staff It's a very good bet the senior IT team were following orders from somewhere else in the chain here.

fencepost 10y ago

I see a bunch of folks talking about whether PCs are connected to the Internet and "why was it running antivirus in the first place?" It's called Defense in Depth. It Does Not Matter if the device is connected to/able to reach the Internet. First, it probably can reach the Internet in some way simply by being networked. I don't think I've ever seen a medical office (can't speak about hospitals) where medical diagnostic equipment was on a fully-separate network able only to talk to other network equipment and specified data destinations (PACS servers). Second, I'm not concerned about unpatched, unprotected machines being infected from the Internet. Odds are they're running a restricted version of Windows, with a custom shell and a lot of stuff stripped out. I'm concerned that they're going to be infected by another machine on the network that's gotten infected. With all the past SQL Server security issues a decade or more ago, how many people think those SQL Server boxes could be directly reached from outside the local network? The conjunction of those two is that even if you firewall all that stuff off, the PACS servers are still on both networks, and are probably running much more interesting and vulnerable stuff than the device controllers. Sure you can fully wall everything off - it's really easy, just do your X-rays onto film, burn your MRIs and ultrasounds onto CDs, and print your EKGs for later scanning. Oh, and listen to people complain about how out-of-date your systems and procedures are. There are other factors that come in as well - sure, every device manufacturer could provide fully bespoke diagnostic displays developed from the ground up in artisanal software shops providing full employment for assembly programmers working on embedded systems, along with cohorts of graphic designers creating glorious steampunk-styled interfaces. That's a beautiful dream, keep having it. For the rest of the world, creating a UI on that custom embedded system running on something from RIM/Blackberry (yeah, they own QNX) is just going to get them crap from people because of A) how clunky it probably looks and B) How could they even consider allowing direct user interaction with the RTOS that was chosen to ensure that the dangerous bits in contact with patients/radiation/irradiated patients were safe? There's a beautiful world out there somewhere where everything is safe and secure and seamless and updated. The rest of us live in worlds where Joe in Marketing's PC gets infected with something that allows an attacker to start scanning the network for unpatched vulnerabilities on any system, which leads to an out-of-date install of IIS on a legacy server that hasn't been updated because there's no longer a contract with the vendor (or no vendor) but it's around because there's a statutory requirement to keep the data on that system for 7-10 years. There's a lot of ugliness out there. Antivirus is a way to try to ensure that when (not if) some of it hits you the repercussions are minimized.

fla 10y ago

How can a medical device be certified for running on 'user hardware' (=uncontrolled environment). Something is probably missing from the article. IMO, the device in question wasn't critical at all, and a failure could be expected.

fiatjaf 10y ago

> The problem is not Windows. It's a lack of knowledge and understanding. Simple. Yes, if there was a minimum of knowledge Windows wouldn't have been used.

dlp211 10y ago
Get out of here with that nonsense. You may not be a fan of paid software, but the Windows Kernel is just as good as any FOSS kernel today in regards to stability. The Server and embedded SKUs also come with a ton of the extraneous stuff removed that one would ever be worried about. The issue here is that someone decided that a machine that should only ever be connected to an air gapped network needed anti-virus software. Disclaimer: I work for Microsoft.
- wyager 10y ago
  >the Windows Kernel is just as good as any FOSS kernel today in regards to stability. I disagree strongly, but this is beside the point. Medical hardware should not be using any operating system that's not hard real-time and thoroughly vetted. Ideally they would use no OS at all. Even Linux, which is drastically more appropriate for embedded systems, is a questionable choice for medical equipment.
- zxcvcxz 10y ago
  > the Windows Kernel is just as good as any FOSS kernel today in regards to stability. No it's really not. I've used Windows and Linux a lot and in 10 years of using Linux I've only twice had a kernel panic while using non-experimental software. On Windows I've had countless BSODs.
  - dlp211 10y ago
    I'm guessing that your use cases for Windows and FOSS kernels are different, but I'll counter your anecdote with some of my own. Almost every single BSOD today is a hardware/driver problem. After Server 2003, MSFT got serious about kernel stability. I haven't had a BSOD from a kernel problem since Vista. Secondly, I also run multiple Server 2012 R2 servers with server hardware, and have never had a BSOD with any of those machines. Finally, Microsoft runs Azure on Win Server, if it wasn't stable, it's cloud platform would be in serious trouble. Disclaimer: I still work for Microsoft

Avernar 10y ago

"Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly. The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side." Here's how I read that: The programmers of this piece of software assumed that some I/O operation would never fail and when it does the program shits itself. So instead of hardening their software to withstand loss of telemetry gracefully, which would cost time and money for the company, they just give instructions to disable scans on their folder. Odds are good that somewhere this scan will happen (and it did). Either IT doesn't read the release notes or goofs the configuration or an antivirus update clears the white list. Might not even be the antivirus that interferes with the telemetry briefly. But instead of having resilient software it's "the anitvirus software's fault" or "it's IT's fault" when something goes wrong because of their bad management/engineering decision.

sillysaurus3 10y ago
Software shouldn't necessarily try to account for errors in that manner. Usually, the most graceful thing to do is to exit cleanly. For example, if there is a massive amount of data, it has to be stored on disk. It's too large to keep in memory. And if the point of the program is to transform that data in real time, then it has to have access to the disk. The antivirus basically unplugged the disk. What can it do to recover? There's nothing to be done. It should be able to survive that situation, of course. When the disk is plugged back in, it should be able to restart without any problems. But I think that's a different kind of resiliency than what you're referring to. In this case, the only way to recover would be to copy the frozen data to a new area of the hard drive, assuming it retained read access. But such complexities result in brittle implementations, prone to acquiring bugs. What if the disk space runs out? So you check beforehand whether there's enough space. But what if some other program starts consuming disk space in the middle of your copy operation? And so on. It's an endless spiral of design complexity. The situation in the article seems closer to hardware failure than a design oversight.

saganus 10y ago

Wtf? "The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure." Who configures an antiviurs for an hourly scan on a doctor's computer?

pritambaral 10y ago
It wasn't even a doctor's computer, it was apparently an operating-room equipment computer.

malbs 10y ago

We've had issues with the latest versions of kaspersky. A burst of network activity is almost guaranteed to crash a machine. It took us a while to isolate Kaspersky 10, and it's not even any particular component inside of Kaspersky, but only when all features are enabled. We tried different permutations of features to try and isolate the cause of our crashes, but as soon as you have any one feature disabled in, the crashes stop, Very frustrating because ultimately our clients laid the blame at my feet (new software feature, new release, blah blah blah), and not exactly much you can do in the way of hardening against this particular crash, the app generates a burst of network data, and boom, blue screen/instant reboot.

dchichkov 10y ago

Let me surprise you, with the code quality that sometimes is running in what is actually 'life-critical' software. Back in the nineties, I wrote a nice piece of some 300kb of C code, for DOS/x86. It was a complete software package, controlling medical equipment that was testing speed of blood coagulation. These tests are crucial in the patient post-operation recovery. This piece of C code had some hardware control code, some statistics, a bit of math, some visualisation, GUI, etc. Normally, you'd imagine a team of 2-3 people, carefully written test cases, dedicated QA person, and a year of time to write something like it. And independend lab, that would certify the thing. Well... in that case, yes, there was independent certification... but... It was just one developer, and I was 13, when I wrote it ;) During after-school time, in around 4-6 months. And I must say, I still sometimes have chills, when I think of the code quality, and, um, unorthodox solutions of 13-year-old myself. Yes, I've had some years of experience at the time, both writing software and designing hardware, and advice from my parents, who both could write software. But, at the time, I've had zero formal training, aside from reading K&R and PC XT manuals ;). So, you might imagine the code quality ;) Even, no need to imagine, I actually still have it somewhere in the archives :)

state 10y ago
I understand why you probably don't want to put it up, but boy would it be fun to look at the code you're describing.
- dchichkov 10y ago
  I probably will put it up. It's a nice inspirational story for teens out here. Doubt there'd be any repercussions, no one cares about some random code on GitHub. And the equipment is hopefully taken out of service years ago, it was more than 20 years back. I wish I knew how long it had been used, but there've been only about 10-20 units sold, I think. I vaguely remember adding extra features for a year or so (like adding support for HP laserjet printer). But one of the founders of the company (on the business side) had some health problems, and I guess that had played role in very small number of units sold. The only feedback that I've had, is pretty much that my father took me to a lab once, that had a unit deployed, for a support call. And I've seen some real printouts with patient names, from the unit. The lab assistant seemed to be happy with the device. I remember them showing me some blood plasma and teaching me to count cells, during lab tour ;)

combatentropy 10y ago

> they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side. And the hospital included full instructions to the software company on how to properly perform a heart transplant, so they were baffled why the programmer just let his teammate die of heart failure. Come on, this kind of stuff should be a zero-configuration hardware-based black box, with its own buttons, screen, etc. --- not something that needs to be (or even can be) connected to something outside the vendor's total control.

firebones 10y ago

For what it is worth, Merge is now part of IBM Watson. http://www.merge.com/News/Article.aspx?ItemID=660 http://www.merge.com/News/Article.aspx?ItemID=660 Welcome to the Health Cloud Powered by Watson.

stevetrewick 10y ago

From the linked report : Based upon the available information, the cause for the reported event was due to the customer not following instructions concerning the installation of anti-virus software; therefore, there is no indication that the reported event was related to product malfunction or defect I beg to differ. I'd consider a momentary loss of file I/O due to lock contention causing a machine to require a reboot a shocking defect in - say - a word processor (which, notably, do not have this problem). That this risk is apparently known and the vendor's sole mitigation is to document a 'Don't do that then' is absolutely 100% an indication of a product defect, even in the absence of an actual occurrence.

billforsternz 10y ago

"A critical medical equipment crashed during a heart procedure due to a timely scan triggered by the antivirus software installed on the PC to which the said device was sending data for logging and monitoring." That should be untimely. The opposite of timely.

toomanythings2 10y ago

I don't think they say this device is controlled by Windows but it must be. Why professional software and instruments even consider using Windows is beyond me.

tmptmp 10y ago

>>The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side. These Merge Hemo fucktards should learn a bit about systems design from Margaret Hamilton [1] [2] before they start coding for such life critical software. Another point is no closed source software (including the greatest ones like Windows) should be allowed to use in such institutions. [1] http://www.nasa.gov/home/hqnews/2003/sep/HQ_03281_Hamilton_Honor.html http://www.nasa.gov/home/hqnews/2003/sep/HQ_03281_Hamilton_H... [2] https://en.m.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%29 https://en.m.wikipedia.org/wiki/Margaret_Hamilton_%28scienti... edit: added point about closed-source software

vilniuse 10y ago

Just like any other software package... :(