9 ms·

Wow. Patching (or using cloud mail providers) would have mitigated the risk for this one...and many others in the past (and the future). The cleanup from this

Wow. Patching (or using cloud mail providers) would have mitigated the risk for this one...and many others in the past (and the future). The cleanup from this is big for those who were hit. Launching attacks during major news events surely also helped the attackers stay under the radar for longer.

walrus01 6y ago
If I had to guess it's a huge laundry-list of organizations that for some legacy reason (Going back 10, 15, 20 years) are running on-premises Exchange, and don't have a full time person one of whose roles is to keep up on patches, security advisories and such.
- logifail 6y ago
  > to keep up on patches, security advisories and such Until you've personally experienced the full horror of attempting to keep on-premises Exchange patched, especially in the SME space where you may have few servers, it's hard to imagine how awful this is. Cumulative Updates are essentially "completely uninstall Exchange" and then "reinstall Exchange again". This is not what one might call a "patch". Then you get into dependencies on .Net and suddenly you need to upgrade the OS as well while you're in the middle of completely-uninstalling-and-reinstalling-Exchange. Last time I got sucked into this, I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system.
  - walrus01 6y ago
    Thankfully for my mental well being it has been 15+ years since I touched Exchange.
  - EvanAnderson 6y ago
    It's hardly a "full horror". I manage on-prem Exchange in the SME space, with single-server installations and multi-server installations (with and without high availability). The patching process is, arguably, inefficient (doing full installs over top of the existing installation) but, in terms of success rate, I've had good luck. I wouldn't put out any new on-prem Exchange today, but the ones I support have reasons to be on-prem or planned migration off-prem. Aside: I've been administering Exchange since version 4.0. I've never experienced "horrors" like so many people talk about. Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems that I've been privy to by way of friends, emergency engagements with non-Customers, etc.
    - logifail 6y ago
      > Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems I'm sure there are some SMEs who are happy to throw serious budget at doing on-prem Exchange "right". For everyone else, I'm not sure what they're supposed to do.
      - EvanAnderson 6y ago
        Everyone else pays for monthly Office 365 subscriptions and ends up spending more money. (Which is what I recommend now, but it galls me to no end.) I don't buy the "Exchange is expensive to support" argument. It's cheaper on-prem than paying for the subscription. We always saw break-even at around 16 - 20 months. I have billing records for a small business Customer w/ a single Exchange 2016 server for last year that amount to 6.5 hours for the entire year, including installing CU's 16 thru 18 (CU 19 fell in this year). Yes-- a piece of their overall Windows Update application budget applies to Exchange, as does the amortized cost of backup software, and server computer and support hardware. Even w/ the OS license, Exchange license, and CALs at 120x an Office 365 E3 monthly subscription they're still money ahead over the 4+ years they've been running Exchange.
        walrus01 6y ago
        However from the point of view of a medium sized business paying for office365, in terms of dollar per month per employee, they're getting much more than just exchange, they're getting onedrive, sharepoint, teams, and the office suite software itself as well.
        EvanAnderson 6y ago
        For sure. And then there's the CapEx/OpEx tax games to take advantage of, too. It's not a bad deal on the whole, but I think it's overhyped as being better than it really is. Moving to subscriptions results in a net increase in spend for organizations that were executing on-prem IT well and frugally. That's the only game now. I just think it's disingenuous to say that it's a cost savings. I reject the massive availability increase argument too, at least in the US, because of the lack of competition in the ISP space and the tier of service that is available to SMEs in their budget. You spend more for the same stuff, are forced to "upgrade" (read: lose features, see changes in UI) at the whim of a third party, and may experiece decreased availability if you're unwilling to spend more on Internet connectivity. There "upsides" for sure, but too many people peddling hosted solutions fail to recognize downsides.
  - Spooky23 6y ago
    I used to run a large on-prem exchange system with about 75k users. It’s literally the only product I’ve ever seen where the admins were the biggest, loudest advocates for outsourcing it to the predecessor to O365. It was more beastly back to run back then though. We did reduce our risk profile at the time by putting OWA behind a sslvpn and only allowing BlackBerry.
  - ocdtrekkie 6y ago
    It'd be nice if CUs were easier to install, but on-prem Exchange management isn't that much work once it's running smoothly. It'd be nice if they made it easier to firewall off more from your AD environment too. But most Exchange management I do is mailbox management, and you have to do that if it's in the cloud too.
    - EvanAnderson 6y ago
      This jibes with my experience. My Customers who have migrated to Office 365 have been using roughly the same labor as when they had on-prem Exchange. (If anything, they're using a little more.)
  - cutemonster 6y ago
    > I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system What did they reply?
- datavirtue 6y ago
  These people are always pulling double duty with Exchange administration being a bemoaned afterthought. I can barely get my head around the scope of this.
brundolf 6y ago
The cloud angle is interesting; on one hand, it creates an even-more-centralized single point of failure. On the other hand, given that virtually every computing system out there is a house of cards, letting the experts focus on securing (and updating!) just a single one might be the best defense.
- mywittyname 6y ago
  The cloud providers can afford to hire and train elite teams to handle security. I remember seeing a post about a guy trying to break out of the docker container used by Cloud SQL on GCP, and apparently the GCP admins made it known that he was being watched pretty early on. I believe the issue was patched fairly quickly too. It's possible that <Random F500 Co> has a great security team. But it's also possible that <Other F500 Co> doesn't.
  - brundolf 6y ago
    Really what we need is the ability to self-host reasonably secure systems without a team of experts working round the clock... but that doesn't appear to be the hand we've been dealt
    - belval 6y ago
      I might be biased because I work at AWS, but I really doubt that there are enough sys admins that know what they are doing and keep up to date let alone find vulnerabilities in the software they use to protect all companies. A Fortune 500 maybe, but at some point you simply can't afford someone who knows what he's doing and at that point you might as well have everything in the cloud so you can focus on your actual money making business.
      - kaliszad 6y ago
        Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution to actually sit down and make simpler systems that a few skilled people can fit in their heads and actually understand?
EvanAnderson 6y ago
The vulnerabilities being exploited were all zero-day. Up-to-date installations were still vulnerable.
kaliszad 6y ago
The proper mitigation would be actually using much simpler, better quality software. Microsoft Exchange Server is quite famous for being an attack vector on corporate networks. At my previous job, the company was advised (by a very capable and expensive security consulting company) to keep Exchange as separate as possible from the corporate network - this of course is a bit counter intuitive, when you want to use e.g. Single Sign-On, contacts and more typically with Active Directory (AD). Thankfully my job wasn't to administer or develop any solutions for AD or Exchange so I just took a note. Obviously, no engineer can have even a sufficient overview of the full Exchange Server implementation not speaking of full understanding. In such a situation security, quality and user (or admin for that matter) experience always take a big hit. It doesn't help Exchange Server is most likely developed using programming languages and approaches that more or less demand complecting the solution with OOP-related ceremony. Supporting two decades or more of legacy features and protocols doesn't help. Some companies even want to connect AD and Exchange to SharePoint... which is at least as complex as Exchange. The problem companies don't understand is that you have to work on simplifying, which is very hard - much harder than adding features. If you don't, the interactions between components will overwhelm even the largest and best skilled team on the planet. The result is, we see breaches and security issues like this every day and realistically, nobody who can decide anything in the corporate environment gives a f** anymore because nobody pays the more or less laughable fines with their own money and nobody really goes to jail but the user data is lost, peoples lives are shattered.
- EvanAnderson 6y ago
  Exchange product architecture was absolutely to blame for this. Very particualrly, the "/ECP" directory should have never been allowed to be Internet accessible. (I believe the upcoming version finally rectifies that in a "supported" way.) In general, though, Microsoft hasn't focused enough on making Exchange more compartmentalized. The servers' privileges in Active Directory are too high (though this is supposedly being addressed in the upcoming version too.)
  - kaliszad 6y ago
    Thank you for the insight. Certainly, "in the upcoming version" is a bit late for those affected and most of those other Exchange-related hacks in the past. The thinking around Exchange is still more or less left in the 20th century and it shows.
fbelzile 6y ago
> (or using cloud mail providers) Why? I don't see moving to a cloud solution being much better. The cloud service itself would be the single point of failure and would be just as vulnerable to a zero day. The organization would have even fewer risk mitigation options like NAT, firewalls, etc.
- jasdine817 6y ago
  As someone who surveys different organizations networks day in and out, the amount of unpatched and out of date Exchange servers (and other internet facing services) I see is ridiculous. Most sysadmins don't have a tangible idea of the risk they take when they set this stuff up. At least Office 365 is patched and monitored on a regular basis and has actual security teams tasked with looking for potential exploits.
jtdev 6y ago
The patches went out Tuesday... after many organizations were already compromised.
weare138 6y ago
It was a 0-day exploit. The patch wasn't released until March 2nd but the vulnerability was being exploited at least since January.

walrus01 6y ago

If I had to guess it's a huge laundry-list of organizations that for some legacy reason (Going back 10, 15, 20 years) are running on-premises Exchange, and don't have a full time person one of whose roles is to keep up on patches, security advisories and such.

logifail 6y ago
> to keep up on patches, security advisories and such Until you've personally experienced the full horror of attempting to keep on-premises Exchange patched, especially in the SME space where you may have few servers, it's hard to imagine how awful this is. Cumulative Updates are essentially "completely uninstall Exchange" and then "reinstall Exchange again". This is not what one might call a "patch". Then you get into dependencies on .Net and suddenly you need to upgrade the OS as well while you're in the middle of completely-uninstalling-and-reinstalling-Exchange. Last time I got sucked into this, I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system.
- walrus01 6y ago
  Thankfully for my mental well being it has been 15+ years since I touched Exchange.
- EvanAnderson 6y ago
  It's hardly a "full horror". I manage on-prem Exchange in the SME space, with single-server installations and multi-server installations (with and without high availability). The patching process is, arguably, inefficient (doing full installs over top of the existing installation) but, in terms of success rate, I've had good luck. I wouldn't put out any new on-prem Exchange today, but the ones I support have reasons to be on-prem or planned migration off-prem. Aside: I've been administering Exchange since version 4.0. I've never experienced "horrors" like so many people talk about. Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems that I've been privy to by way of friends, emergency engagements with non-Customers, etc.
  - logifail 6y ago
    > Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems I'm sure there are some SMEs who are happy to throw serious budget at doing on-prem Exchange "right". For everyone else, I'm not sure what they're supposed to do.
    - EvanAnderson 6y ago
      Everyone else pays for monthly Office 365 subscriptions and ends up spending more money. (Which is what I recommend now, but it galls me to no end.) I don't buy the "Exchange is expensive to support" argument. It's cheaper on-prem than paying for the subscription. We always saw break-even at around 16 - 20 months. I have billing records for a small business Customer w/ a single Exchange 2016 server for last year that amount to 6.5 hours for the entire year, including installing CU's 16 thru 18 (CU 19 fell in this year). Yes-- a piece of their overall Windows Update application budget applies to Exchange, as does the amortized cost of backup software, and server computer and support hardware. Even w/ the OS license, Exchange license, and CALs at 120x an Office 365 E3 monthly subscription they're still money ahead over the 4+ years they've been running Exchange.
      - walrus01 6y ago
        However from the point of view of a medium sized business paying for office365, in terms of dollar per month per employee, they're getting much more than just exchange, they're getting onedrive, sharepoint, teams, and the office suite software itself as well.
        EvanAnderson 6y ago
        For sure. And then there's the CapEx/OpEx tax games to take advantage of, too. It's not a bad deal on the whole, but I think it's overhyped as being better than it really is. Moving to subscriptions results in a net increase in spend for organizations that were executing on-prem IT well and frugally. That's the only game now. I just think it's disingenuous to say that it's a cost savings. I reject the massive availability increase argument too, at least in the US, because of the lack of competition in the ISP space and the tier of service that is available to SMEs in their budget. You spend more for the same stuff, are forced to "upgrade" (read: lose features, see changes in UI) at the whim of a third party, and may experiece decreased availability if you're unwilling to spend more on Internet connectivity. There "upsides" for sure, but too many people peddling hosted solutions fail to recognize downsides.
- Spooky23 6y ago
  I used to run a large on-prem exchange system with about 75k users. It’s literally the only product I’ve ever seen where the admins were the biggest, loudest advocates for outsourcing it to the predecessor to O365. It was more beastly back to run back then though. We did reduce our risk profile at the time by putting OWA behind a sslvpn and only allowing BlackBerry.
- ocdtrekkie 6y ago
  It'd be nice if CUs were easier to install, but on-prem Exchange management isn't that much work once it's running smoothly. It'd be nice if they made it easier to firewall off more from your AD environment too. But most Exchange management I do is mailbox management, and you have to do that if it's in the cloud too.
  - EvanAnderson 6y ago
    This jibes with my experience. My Customers who have migrated to Office 365 have been using roughly the same labor as when they had on-prem Exchange. (If anything, they're using a little more.)
- cutemonster 6y ago
  > I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system What did they reply?
datavirtue 6y ago
These people are always pulling double duty with Exchange administration being a bemoaned afterthought. I can barely get my head around the scope of this.

brundolf 6y ago

The cloud angle is interesting; on one hand, it creates an even-more-centralized single point of failure. On the other hand, given that virtually every computing system out there is a house of cards, letting the experts focus on securing (and updating!) just a single one might be the best defense.

mywittyname 6y ago
The cloud providers can afford to hire and train elite teams to handle security. I remember seeing a post about a guy trying to break out of the docker container used by Cloud SQL on GCP, and apparently the GCP admins made it known that he was being watched pretty early on. I believe the issue was patched fairly quickly too. It's possible that <Random F500 Co> has a great security team. But it's also possible that <Other F500 Co> doesn't.
- brundolf 6y ago
  Really what we need is the ability to self-host reasonably secure systems without a team of experts working round the clock... but that doesn't appear to be the hand we've been dealt
  - belval 6y ago
    I might be biased because I work at AWS, but I really doubt that there are enough sys admins that know what they are doing and keep up to date let alone find vulnerabilities in the software they use to protect all companies. A Fortune 500 maybe, but at some point you simply can't afford someone who knows what he's doing and at that point you might as well have everything in the cloud so you can focus on your actual money making business.
    - kaliszad 6y ago
      Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution to actually sit down and make simpler systems that a few skilled people can fit in their heads and actually understand?

EvanAnderson 6y ago

The vulnerabilities being exploited were all zero-day. Up-to-date installations were still vulnerable.

kaliszad 6y ago

The proper mitigation would be actually using much simpler, better quality software. Microsoft Exchange Server is quite famous for being an attack vector on corporate networks. At my previous job, the company was advised (by a very capable and expensive security consulting company) to keep Exchange as separate as possible from the corporate network - this of course is a bit counter intuitive, when you want to use e.g. Single Sign-On, contacts and more typically with Active Directory (AD). Thankfully my job wasn't to administer or develop any solutions for AD or Exchange so I just took a note. Obviously, no engineer can have even a sufficient overview of the full Exchange Server implementation not speaking of full understanding. In such a situation security, quality and user (or admin for that matter) experience always take a big hit. It doesn't help Exchange Server is most likely developed using programming languages and approaches that more or less demand complecting the solution with OOP-related ceremony. Supporting two decades or more of legacy features and protocols doesn't help. Some companies even want to connect AD and Exchange to SharePoint... which is at least as complex as Exchange. The problem companies don't understand is that you have to work on simplifying, which is very hard - much harder than adding features. If you don't, the interactions between components will overwhelm even the largest and best skilled team on the planet. The result is, we see breaches and security issues like this every day and realistically, nobody who can decide anything in the corporate environment gives a f** anymore because nobody pays the more or less laughable fines with their own money and nobody really goes to jail but the user data is lost, peoples lives are shattered.

EvanAnderson 6y ago
Exchange product architecture was absolutely to blame for this. Very particualrly, the "/ECP" directory should have never been allowed to be Internet accessible. (I believe the upcoming version finally rectifies that in a "supported" way.) In general, though, Microsoft hasn't focused enough on making Exchange more compartmentalized. The servers' privileges in Active Directory are too high (though this is supposedly being addressed in the upcoming version too.)
- kaliszad 6y ago
  Thank you for the insight. Certainly, "in the upcoming version" is a bit late for those affected and most of those other Exchange-related hacks in the past. The thinking around Exchange is still more or less left in the 20th century and it shows.

fbelzile 6y ago

> (or using cloud mail providers) Why? I don't see moving to a cloud solution being much better. The cloud service itself would be the single point of failure and would be just as vulnerable to a zero day. The organization would have even fewer risk mitigation options like NAT, firewalls, etc.

jasdine817 6y ago
As someone who surveys different organizations networks day in and out, the amount of unpatched and out of date Exchange servers (and other internet facing services) I see is ridiculous. Most sysadmins don't have a tangible idea of the risk they take when they set this stuff up. At least Office 365 is patched and monitored on a regular basis and has actual security teams tasked with looking for potential exploits.

jtdev 6y ago

The patches went out Tuesday... after many organizations were already compromised.

weare138 6y ago

It was a 0-day exploit. The patch wasn't released until March 2nd but the vulnerability was being exploited at least since January.