15 ms·

> "Microsoft rolled out a patch for the vulnerability last March, but hackers took advantage of the fact that vulnerable targets — particularly hospitals — had

> "Microsoft rolled out a patch for the vulnerability last March, but hackers took advantage of the fact that vulnerable targets — particularly hospitals — had yet to update their systems." > "The malware was circulated by email; targets were sent an encrypted, compressed file that, once loaded, allowed the ransomware to infiltrate its targets." It sounds like the basic (?) security practices recommended by professionals - keep systems up-to-date, pay attention to whether an email is suspicious - would have covered your network. Of course, as @mhogomchunu points out in his comment - is this the sort of thing where only one weak link is needed? Still. Maybe this will help the proponents of keeping government systems updated? And/or, maybe this will prompt companies like MS to roll out security-only updates, to make it easier for sysadmins to keep their systems up-to-date...? (presumably, a reason why these systems weren't updated is due to functionality concerns with updates...?)

dekhn 9y ago
If you run a large installation of computers, taking updates can be a huge risk. Often they can break things, and then you're in the position of being blamed for running an update. Not updating can often lead to much higher stability. In previous environments I've worked that were "regulated", any change to the environent, such as a firmware upgrade, triggered an entire re-regulation process (testing, paperwork, etc).
- stonogo 9y ago
  That's wrong. If you run a large installation of computers, and you do not have a plan and a process for quickly deploying security patches, you should be fired with cause. In this specific case, there are mitigations available that do not require installation of software, but merely a configuration change. Also in this specific case, the people who run IT at NHS are completely incompetent, and this has been well-documented for several years. In the general case, "I have a lot of machines" is an excuse provided by the unable to evade being held responsible by the uninformed.
  - bluGill 9y ago
    Easy for your to say. I have been unable to do my job for a several days because some update broke a service I was using. Sure the service was badly written, but we didn't know that until the patch was applied. The phone company used to have (they still might, I'm not in the business anymore) large labs that were small replications of their network. I've been in meetings where the goal was to decide if we should try to get our latest release through their process - if yes and we were successful they would pay big for the latest features, but if yes and it failed [I can't remember, I think we had to pay them for test time, but the contract was complex]. A lot of time was spent looking at every known bug to decide if it was important.
    - linkregister 9y ago
      Was the update a Microsoft Security Update?
  - dekhn 9y ago
    if your job is to keep a bunch of computers working, keeping the systems running is the goal. Deploying security patches quickly is not always considered a requirement. Again, the problem is that rolling out patches quickly often leads to unplanned problems that can't be easily detected or rolled back from. That can cause problems worse than leaving security issues unpatched.
    - PeterisP 9y ago
      If your systems are exposed to the Internet, then deploying security patches quickly is a part of keeping the systems running - as illustrated by this case, where the systems obviously are not running and can't be easily rolled back to a working state. The business of cybercrime is changing. With the growing popularity of ransomware, we should expect a gradual decrease in time between a published remote vulnerability and your systems getting attacked. It may be useful to delay patches by a day to see if there aren't any glaring problems encountered by others - but it's not a reason do leave open holes that were patched in March. Frankly, there was no good reason why this attack hadn't happened a month ago; next time the gap may be much smaller. Yes, there is a chance that installing a security update to break your systems. But there's also a chance that not installing a security update will break your systems, and that chance, frankly, is much higher. Furthermore, "That can cause problems worse than leaving security issues unpatched" seems trivially untrue. Every horrible thing that might happen because of a patch broken in a weird way may also happen because of an unpatched security issue. Leaving security issues unpatched can take down all your systems and data, plus also expose confidential information. A MS patch, on the other hand, assuming that it's tested in any way whatsoever, won't do that - at most, it will take down some of your systems, which is bad, but not as bad as e.g. Spain's Telefonica is experiencing right now. What patch could have caused them even worse problems?
- linkregister 9y ago
  Downloading Microsoft security updates is simple and safe. You just download the monthly rollup: http://www.catalog.update.microsoft.com/search.aspx?q=4012215 http://www.catalog.update.microsoft.com/search.aspx?q=401221... Any competent sysadmin will have these available on their internal update server and push updates+restart during off-peak hours. Receptionist computers that can open websites with untrusted JavaScript can't reasonably be held to this certification. Certification isn't what kept the NHS from applying patches.
  - baq 9y ago
    I'd like to hear r/sysadmin opinion on that.
    - linkregister 9y ago
      Translation: "My feelings make me feel that the statement isn't right. Instead of finding out, I'm just going to say that I wish someone would tell this commenter they're wrong."
      - CamperBob2 9y ago
        Translation: "Microsoft has lied about the content of their updates before."
  - tjalfi 9y ago
    Some vertical markets use a lot of software that integrates with Microsoft Office applications. The result is that there is a much higher change of a Microsoft update breaking a critical application. [0] is a recent (September 2015) example of two Microsoft patches that were widely blocked in the legal industry until Microsoft released a follow up patch. iManage and Workshare, the products mentioned in the blog entry, are considered critical applications in any law firm that uses them. iManage is a widely used document management system (think primitive VCS with Office add-ins). All documents are stored in the DMS so access to it is critical to the business. Workshare is used for document comparison and metadata scrubbing. Metadata scrubbing is used on all outgoing emails. [0] http://www.kraftkennedy.com/block-2-microsoft-patches-prevent-awful-bug/ http://www.kraftkennedy.com/block-2-microsoft-patches-preven...
- lomnakkus 9y ago
  > If you run a large installation of computers, taking updates can be a huge risk. Often they can break things, and then you're in the position of being blamed for running an update. Not updating can often lead to much higher stability. There is such a thing as staged rollouts for this exact type of scenario.
elorant 9y ago
Well this justifies MS's decision for forced updates in Win10. Not that I like it, just saying.
- digi_owl 9y ago
  Keep in mind that on business grade installs the updates are not forced. And there have already been situations where updates have caused problems. Maybe not as severe as a full on attack, but enough to potentially disrupt production and thus risk someones job.
- devrandomguy 9y ago
  It justifies security updates for all operating systems. It does not justify the installation of spyware or changes to the user interface.
  - a_imho 9y ago
    Sadly this distinction is rarely made and imo intentionally kept ambiguous. Lovejoy's law is used for justifying spy, bloat and crapware.
- notalaser 9y ago
  > Well this justifies MS's decision for forced updates in Win10. Not that I like it, just saying. Unfortunately, I think the active hours period cannot be set to more than twelve hours, which is less than the time required for some surgical interventions. I can almost imagine it: OK everyone, ten-minute break while Windows installs its updates, this guy who's been on life support for the last ten hours can wait a little longer. That's why updates are not forced on business-grade installs, and forcing them would be a very, very stupid decision. Forced updates make sense for home users, since Microsoft can't depend on someone requiring them to keep their networks secure. For other types of users, second-guessing update policies is always a bad idea.
  - sp332 9y ago
    Win10 Pro is more flexible, although you might have to drop down to Group Policy to do it. http://pureinfotech.com/defer-windows-10-upgrades-updates/ http://pureinfotech.com/defer-windows-10-upgrades-updates/ At the very least, the workstations can be pointed to internal WSUS servers which control the rollouts. I'm guessing that's how most of the currently-vulnerable computers stayed vulnerable until now.
eggbrain 9y ago
> It sounds like the basic (?) security practices recommended by professionals - keep systems up-to-date, pay attention to whether an email is suspicious - would have covered your network. This is secondhand information (so take it for what it's worth, there could be pieces I'm missing), but I talked with a startup that was focusing on this problem, and the issue was not quite the computers and servers that IT were using (although sometimes it was), it was that many medical devices (like CT scanners, pumps, etc) come shipped with old outdated versions of operating systems and libraries. No big deal right? Just make sure those are up to date too? Well, many times the support contract for these medical devices are so strict that you can invalidate the warranty by installing third party software like an antivirus, or even doing something like Windows update. Even worse, many hospitals don't even know what devices they have -- it's easy for IT to know about laptops and computers, but when every single medical device more complicated than a stethoscope has a chip in it and may respond to calls on certain ports, it's a tougher picture to know. The startup was https://www.virtalabs.com/ https://www.virtalabs.com/ by the way, they really are doing some cool things to help with this.
- MichaelGG 9y ago
  Why are those devices being connected to an unsecure network? Surely they should have super limited data exchange features?
  - pyre 9y ago
    As is commonly the case, hardware vendors are more concerned with selling you the hardware and probably spend bottom-dollar for their software developers. I can't say that I've worked in such an environment, but my impression is that management at such companies probably see software dev as a cost-centre rather than something to actually spend money on for quality.
    - MichaelGG 9y ago
      But the hospital management shouldn't be plugging them onto the same network where end-users have access, no?
eliudnir 9y ago
I'm sure this sort of stuff doesn't help with speedy updates. https://arstechnica.com/tech-policy/2017/03/public-university-lays-off-79-it-workers-after-they-train-h-1b-replacements/ https://arstechnica.com/tech-policy/2017/03/public-universit...
- Joeri 9y ago
  I saw this at work from the inside of a big telco that did this. They replaced one guy who needed only vague instructions to configure complex software replaced by a team of five who needed detailed step-by-step manuals written out by the vendor and still took twice as long and couldn't cope with any hiccups along the way. I do not believe outsourcing saves money. It only does so either by cutting quality of service, or in cases where the IT department was heavily mismanaged anyway. Bring in capable management and you don't need to outsource.
  - billharrison 9y ago
    I've never seen a case where outsourcing of general-purpose IT things saves money over the long-term. It might make the budget look better for a year or two. Which, I think, is the motivation for a lot of the people making decisions to outsource. It is cheaper right now, so who cares about later? Special-purpose stuff can still be cheaper to outsource, though. If I need something to work next week and it would take my staff a month to get up to speed, I'd spend the money on outsourcing it.
jordigh 9y ago
Karen Sandler has an interesting story about medical devices and how they are, literally, putting her life on the line. She's both a lawyer and a hacker, and you should hear the stories she tells about how people distrust her for this and think she's trying to trick them when all she wants to do is learn about the software and hardware that is keeping her alive: https://www.youtube.com/watch?v=iMKHqO28FcI https://www.youtube.com/watch?v=iMKHqO28FcI
- phrh8 9y ago
  Wow, that video deserves more views than it has. Really interesting and relevant for a two year old talk.
alkonaut 9y ago
Functionality is always second to security. In this situation you patch all the machines and then you test which still work . If the machines at the hospital dont work because some software was incompatible with the vulnerability fix (which is almost unthinkable in most cases)- then those coputers are simply unavailable and surgeries are cancelled or whatever the impact may be.
ernst_klim 9y ago
>It sounds like the basic (?) security practices recommended by professionals The problem with windows is that crap can not be upgraded without stopping workflow and rebooting. With linux distros you can upgrade packages in a background (I think that's because you can upgrade the file being executed in linux, while in windows you can't, but I'm not sure) without even rebooting after upgrade. You can even patch your kernel without reboot. It windows you have to see an upgrade screen for an hour without an opportunity to do something useful, and after that you have to reboot. That sucks.

dekhn 9y ago

If you run a large installation of computers, taking updates can be a huge risk. Often they can break things, and then you're in the position of being blamed for running an update. Not updating can often lead to much higher stability. In previous environments I've worked that were "regulated", any change to the environent, such as a firmware upgrade, triggered an entire re-regulation process (testing, paperwork, etc).

stonogo 9y ago
That's wrong. If you run a large installation of computers, and you do not have a plan and a process for quickly deploying security patches, you should be fired with cause. In this specific case, there are mitigations available that do not require installation of software, but merely a configuration change. Also in this specific case, the people who run IT at NHS are completely incompetent, and this has been well-documented for several years. In the general case, "I have a lot of machines" is an excuse provided by the unable to evade being held responsible by the uninformed.
- bluGill 9y ago
  Easy for your to say. I have been unable to do my job for a several days because some update broke a service I was using. Sure the service was badly written, but we didn't know that until the patch was applied. The phone company used to have (they still might, I'm not in the business anymore) large labs that were small replications of their network. I've been in meetings where the goal was to decide if we should try to get our latest release through their process - if yes and we were successful they would pay big for the latest features, but if yes and it failed [I can't remember, I think we had to pay them for test time, but the contract was complex]. A lot of time was spent looking at every known bug to decide if it was important.
  - linkregister 9y ago
    Was the update a Microsoft Security Update?
- dekhn 9y ago
  if your job is to keep a bunch of computers working, keeping the systems running is the goal. Deploying security patches quickly is not always considered a requirement. Again, the problem is that rolling out patches quickly often leads to unplanned problems that can't be easily detected or rolled back from. That can cause problems worse than leaving security issues unpatched.
  - PeterisP 9y ago
    If your systems are exposed to the Internet, then deploying security patches quickly is a part of keeping the systems running - as illustrated by this case, where the systems obviously are not running and can't be easily rolled back to a working state. The business of cybercrime is changing. With the growing popularity of ransomware, we should expect a gradual decrease in time between a published remote vulnerability and your systems getting attacked. It may be useful to delay patches by a day to see if there aren't any glaring problems encountered by others - but it's not a reason do leave open holes that were patched in March. Frankly, there was no good reason why this attack hadn't happened a month ago; next time the gap may be much smaller. Yes, there is a chance that installing a security update to break your systems. But there's also a chance that not installing a security update will break your systems, and that chance, frankly, is much higher. Furthermore, "That can cause problems worse than leaving security issues unpatched" seems trivially untrue. Every horrible thing that might happen because of a patch broken in a weird way may also happen because of an unpatched security issue. Leaving security issues unpatched can take down all your systems and data, plus also expose confidential information. A MS patch, on the other hand, assuming that it's tested in any way whatsoever, won't do that - at most, it will take down some of your systems, which is bad, but not as bad as e.g. Spain's Telefonica is experiencing right now. What patch could have caused them even worse problems?
linkregister 9y ago
Downloading Microsoft security updates is simple and safe. You just download the monthly rollup: http://www.catalog.update.microsoft.com/search.aspx?q=4012215 http://www.catalog.update.microsoft.com/search.aspx?q=401221... Any competent sysadmin will have these available on their internal update server and push updates+restart during off-peak hours. Receptionist computers that can open websites with untrusted JavaScript can't reasonably be held to this certification. Certification isn't what kept the NHS from applying patches.
- baq 9y ago
  I'd like to hear r/sysadmin opinion on that.
  - linkregister 9y ago
    Translation: "My feelings make me feel that the statement isn't right. Instead of finding out, I'm just going to say that I wish someone would tell this commenter they're wrong."
    - CamperBob2 9y ago
      Translation: "Microsoft has lied about the content of their updates before."
- tjalfi 9y ago
  Some vertical markets use a lot of software that integrates with Microsoft Office applications. The result is that there is a much higher change of a Microsoft update breaking a critical application. [0] is a recent (September 2015) example of two Microsoft patches that were widely blocked in the legal industry until Microsoft released a follow up patch. iManage and Workshare, the products mentioned in the blog entry, are considered critical applications in any law firm that uses them. iManage is a widely used document management system (think primitive VCS with Office add-ins). All documents are stored in the DMS so access to it is critical to the business. Workshare is used for document comparison and metadata scrubbing. Metadata scrubbing is used on all outgoing emails. [0] http://www.kraftkennedy.com/block-2-microsoft-patches-prevent-awful-bug/ http://www.kraftkennedy.com/block-2-microsoft-patches-preven...
lomnakkus 9y ago
> If you run a large installation of computers, taking updates can be a huge risk. Often they can break things, and then you're in the position of being blamed for running an update. Not updating can often lead to much higher stability. There is such a thing as staged rollouts for this exact type of scenario.

elorant 9y ago

Well this justifies MS's decision for forced updates in Win10. Not that I like it, just saying.

digi_owl 9y ago
Keep in mind that on business grade installs the updates are not forced. And there have already been situations where updates have caused problems. Maybe not as severe as a full on attack, but enough to potentially disrupt production and thus risk someones job.
devrandomguy 9y ago
It justifies security updates for all operating systems. It does not justify the installation of spyware or changes to the user interface.
- a_imho 9y ago
  Sadly this distinction is rarely made and imo intentionally kept ambiguous. Lovejoy's law is used for justifying spy, bloat and crapware.
notalaser 9y ago
> Well this justifies MS's decision for forced updates in Win10. Not that I like it, just saying. Unfortunately, I think the active hours period cannot be set to more than twelve hours, which is less than the time required for some surgical interventions. I can almost imagine it: OK everyone, ten-minute break while Windows installs its updates, this guy who's been on life support for the last ten hours can wait a little longer. That's why updates are not forced on business-grade installs, and forcing them would be a very, very stupid decision. Forced updates make sense for home users, since Microsoft can't depend on someone requiring them to keep their networks secure. For other types of users, second-guessing update policies is always a bad idea.
- sp332 9y ago
  Win10 Pro is more flexible, although you might have to drop down to Group Policy to do it. http://pureinfotech.com/defer-windows-10-upgrades-updates/ http://pureinfotech.com/defer-windows-10-upgrades-updates/ At the very least, the workstations can be pointed to internal WSUS servers which control the rollouts. I'm guessing that's how most of the currently-vulnerable computers stayed vulnerable until now.

eggbrain 9y ago

> It sounds like the basic (?) security practices recommended by professionals - keep systems up-to-date, pay attention to whether an email is suspicious - would have covered your network. This is secondhand information (so take it for what it's worth, there could be pieces I'm missing), but I talked with a startup that was focusing on this problem, and the issue was not quite the computers and servers that IT were using (although sometimes it was), it was that many medical devices (like CT scanners, pumps, etc) come shipped with old outdated versions of operating systems and libraries. No big deal right? Just make sure those are up to date too? Well, many times the support contract for these medical devices are so strict that you can invalidate the warranty by installing third party software like an antivirus, or even doing something like Windows update. Even worse, many hospitals don't even know what devices they have -- it's easy for IT to know about laptops and computers, but when every single medical device more complicated than a stethoscope has a chip in it and may respond to calls on certain ports, it's a tougher picture to know. The startup was https://www.virtalabs.com/ https://www.virtalabs.com/ by the way, they really are doing some cool things to help with this.

MichaelGG 9y ago
Why are those devices being connected to an unsecure network? Surely they should have super limited data exchange features?
- pyre 9y ago
  As is commonly the case, hardware vendors are more concerned with selling you the hardware and probably spend bottom-dollar for their software developers. I can't say that I've worked in such an environment, but my impression is that management at such companies probably see software dev as a cost-centre rather than something to actually spend money on for quality.
  - MichaelGG 9y ago
    But the hospital management shouldn't be plugging them onto the same network where end-users have access, no?

eliudnir 9y ago

I'm sure this sort of stuff doesn't help with speedy updates. https://arstechnica.com/tech-policy/2017/03/public-university-lays-off-79-it-workers-after-they-train-h-1b-replacements/ https://arstechnica.com/tech-policy/2017/03/public-universit...

Joeri 9y ago
I saw this at work from the inside of a big telco that did this. They replaced one guy who needed only vague instructions to configure complex software replaced by a team of five who needed detailed step-by-step manuals written out by the vendor and still took twice as long and couldn't cope with any hiccups along the way. I do not believe outsourcing saves money. It only does so either by cutting quality of service, or in cases where the IT department was heavily mismanaged anyway. Bring in capable management and you don't need to outsource.
- billharrison 9y ago
  I've never seen a case where outsourcing of general-purpose IT things saves money over the long-term. It might make the budget look better for a year or two. Which, I think, is the motivation for a lot of the people making decisions to outsource. It is cheaper right now, so who cares about later? Special-purpose stuff can still be cheaper to outsource, though. If I need something to work next week and it would take my staff a month to get up to speed, I'd spend the money on outsourcing it.

jordigh 9y ago

Karen Sandler has an interesting story about medical devices and how they are, literally, putting her life on the line. She's both a lawyer and a hacker, and you should hear the stories she tells about how people distrust her for this and think she's trying to trick them when all she wants to do is learn about the software and hardware that is keeping her alive: https://www.youtube.com/watch?v=iMKHqO28FcI https://www.youtube.com/watch?v=iMKHqO28FcI

phrh8 9y ago
Wow, that video deserves more views than it has. Really interesting and relevant for a two year old talk.

alkonaut 9y ago

Functionality is always second to security. In this situation you patch all the machines and then you test which still work . If the machines at the hospital dont work because some software was incompatible with the vulnerability fix (which is almost unthinkable in most cases)- then those coputers are simply unavailable and surgeries are cancelled or whatever the impact may be.

ernst_klim 9y ago

>It sounds like the basic (?) security practices recommended by professionals The problem with windows is that crap can not be upgraded without stopping workflow and rebooting. With linux distros you can upgrade packages in a background (I think that's because you can upgrade the file being executed in linux, while in windows you can't, but I'm not sure) without even rebooting after upgrade. You can even patch your kernel without reboot. It windows you have to see an upgrade screen for an hour without an opportunity to do something useful, and after that you have to reboot. That sucks.