25 ms·

30k U.S. organizations newly hacked via holes in Microsoft Exchange Server

1038 points by parsecs 6y ago

waynesoftware 6y ago
Wow. Patching (or using cloud mail providers) would have mitigated the risk for this one...and many others in the past (and the future). The cleanup from this is big for those who were hit. Launching attacks during major news events surely also helped the attackers stay under the radar for longer.
- walrus01 6y ago
  If I had to guess it's a huge laundry-list of organizations that for some legacy reason (Going back 10, 15, 20 years) are running on-premises Exchange, and don't have a full time person one of whose roles is to keep up on patches, security advisories and such.
  - logifail 6y ago
    > to keep up on patches, security advisories and such Until you've personally experienced the full horror of attempting to keep on-premises Exchange patched, especially in the SME space where you may have few servers, it's hard to imagine how awful this is. Cumulative Updates are essentially "completely uninstall Exchange" and then "reinstall Exchange again". This is not what one might call a "patch". Then you get into dependencies on .Net and suddenly you need to upgrade the OS as well while you're in the middle of completely-uninstalling-and-reinstalling-Exchange. Last time I got sucked into this, I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system.
    - walrus01 6y ago
      Thankfully for my mental well being it has been 15+ years since I touched Exchange.
    - EvanAnderson 6y ago
      It's hardly a "full horror". I manage on-prem Exchange in the SME space, with single-server installations and multi-server installations (with and without high availability). The patching process is, arguably, inefficient (doing full installs over top of the existing installation) but, in terms of success rate, I've had good luck. I wouldn't put out any new on-prem Exchange today, but the ones I support have reasons to be on-prem or planned migration off-prem. Aside: I've been administering Exchange since version 4.0. I've never experienced "horrors" like so many people talk about. Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems that I've been privy to by way of friends, emergency engagements with non-Customers, etc.
      - logifail 6y ago
        > Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems I'm sure there are some SMEs who are happy to throw serious budget at doing on-prem Exchange "right". For everyone else, I'm not sure what they're supposed to do.
        EvanAnderson 6y ago
        Everyone else pays for monthly Office 365 subscriptions and ends up spending more money. (Which is what I recommend now, but it galls me to no end.) I don't buy the "Exchange is expensive to support" argument. It's cheaper on-prem than paying for the subscription. We always saw break-even at around 16 - 20 months. I have billing records for a small business Customer w/ a single Exchange 2016 server for last year that amount to 6.5 hours for the entire year, including installing CU's 16 thru 18 (CU 19 fell in this year). Yes-- a piece of their overall Windows Update application budget applies to Exchange, as does the amortized cost of backup software, and server computer and support hardware. Even w/ the OS license, Exchange license, and CALs at 120x an Office 365 E3 monthly subscription they're still money ahead over the 4+ years they've been running Exchange.
        walrus01 6y ago
        However from the point of view of a medium sized business paying for office365, in terms of dollar per month per employee, they're getting much more than just exchange, they're getting onedrive, sharepoint, teams, and the office suite software itself as well.
    - Spooky23 6y ago
      I used to run a large on-prem exchange system with about 75k users. It’s literally the only product I’ve ever seen where the admins were the biggest, loudest advocates for outsourcing it to the predecessor to O365. It was more beastly back to run back then though. We did reduce our risk profile at the time by putting OWA behind a sslvpn and only allowing BlackBerry.
    - ocdtrekkie 6y ago
      It'd be nice if CUs were easier to install, but on-prem Exchange management isn't that much work once it's running smoothly. It'd be nice if they made it easier to firewall off more from your AD environment too. But most Exchange management I do is mailbox management, and you have to do that if it's in the cloud too.
      - EvanAnderson 6y ago
        This jibes with my experience. My Customers who have migrated to Office 365 have been using roughly the same labor as when they had on-prem Exchange. (If anything, they're using a little more.)
    - cutemonster 6y ago
      > I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system What did they reply?
  - datavirtue 6y ago
    These people are always pulling double duty with Exchange administration being a bemoaned afterthought. I can barely get my head around the scope of this.
- brundolf 6y ago
  The cloud angle is interesting; on one hand, it creates an even-more-centralized single point of failure. On the other hand, given that virtually every computing system out there is a house of cards, letting the experts focus on securing (and updating!) just a single one might be the best defense.
  - mywittyname 6y ago
    The cloud providers can afford to hire and train elite teams to handle security. I remember seeing a post about a guy trying to break out of the docker container used by Cloud SQL on GCP, and apparently the GCP admins made it known that he was being watched pretty early on. I believe the issue was patched fairly quickly too. It's possible that <Random F500 Co> has a great security team. But it's also possible that <Other F500 Co> doesn't.
    - brundolf 6y ago
      Really what we need is the ability to self-host reasonably secure systems without a team of experts working round the clock... but that doesn't appear to be the hand we've been dealt
      - belval 6y ago
        I might be biased because I work at AWS, but I really doubt that there are enough sys admins that know what they are doing and keep up to date let alone find vulnerabilities in the software they use to protect all companies. A Fortune 500 maybe, but at some point you simply can't afford someone who knows what he's doing and at that point you might as well have everything in the cloud so you can focus on your actual money making business.
        kaliszad 6y ago
        Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution to actually sit down and make simpler systems that a few skilled people can fit in their heads and actually understand?
- EvanAnderson 6y ago
  The vulnerabilities being exploited were all zero-day. Up-to-date installations were still vulnerable.
- kaliszad 6y ago
  The proper mitigation would be actually using much simpler, better quality software. Microsoft Exchange Server is quite famous for being an attack vector on corporate networks. At my previous job, the company was advised (by a very capable and expensive security consulting company) to keep Exchange as separate as possible from the corporate network - this of course is a bit counter intuitive, when you want to use e.g. Single Sign-On, contacts and more typically with Active Directory (AD). Thankfully my job wasn't to administer or develop any solutions for AD or Exchange so I just took a note. Obviously, no engineer can have even a sufficient overview of the full Exchange Server implementation not speaking of full understanding. In such a situation security, quality and user (or admin for that matter) experience always take a big hit. It doesn't help Exchange Server is most likely developed using programming languages and approaches that more or less demand complecting the solution with OOP-related ceremony. Supporting two decades or more of legacy features and protocols doesn't help. Some companies even want to connect AD and Exchange to SharePoint... which is at least as complex as Exchange. The problem companies don't understand is that you have to work on simplifying, which is very hard - much harder than adding features. If you don't, the interactions between components will overwhelm even the largest and best skilled team on the planet. The result is, we see breaches and security issues like this every day and realistically, nobody who can decide anything in the corporate environment gives a f** anymore because nobody pays the more or less laughable fines with their own money and nobody really goes to jail but the user data is lost, peoples lives are shattered.
  - EvanAnderson 6y ago
    Exchange product architecture was absolutely to blame for this. Very particualrly, the "/ECP" directory should have never been allowed to be Internet accessible. (I believe the upcoming version finally rectifies that in a "supported" way.) In general, though, Microsoft hasn't focused enough on making Exchange more compartmentalized. The servers' privileges in Active Directory are too high (though this is supposedly being addressed in the upcoming version too.)
    - kaliszad 6y ago
      Thank you for the insight. Certainly, "in the upcoming version" is a bit late for those affected and most of those other Exchange-related hacks in the past. The thinking around Exchange is still more or less left in the 20th century and it shows.
- fbelzile 6y ago
  > (or using cloud mail providers) Why? I don't see moving to a cloud solution being much better. The cloud service itself would be the single point of failure and would be just as vulnerable to a zero day. The organization would have even fewer risk mitigation options like NAT, firewalls, etc.
  - jasdine817 6y ago
    As someone who surveys different organizations networks day in and out, the amount of unpatched and out of date Exchange servers (and other internet facing services) I see is ridiculous. Most sysadmins don't have a tangible idea of the risk they take when they set this stuff up. At least Office 365 is patched and monitored on a regular basis and has actual security teams tasked with looking for potential exploits.
- jtdev 6y ago
  The patches went out Tuesday... after many organizations were already compromised.
- weare138 6y ago
  It was a 0-day exploit. The patch wasn't released until March 2nd but the vulnerability was being exploited at least since January.
brundolf 6y ago
It's almost like all of our institutions shouldn't use the exact same software vendors
- jgalt212 6y ago
  or Identity Providers.
- exporectomy 6y ago
  Really? Wouldn't multiple softwares be equally vulnerable overall but the hacks would be more distributed in time as they're discovered at different times? Is that the problem you'd hope to solve? That it all happened within a few days instead of at different institutions at different times?
  - brundolf 6y ago
    Yes, distributing the same number of hacks over a period of time would on its own make things a little bit less fragile. In general, having a single point of failure is bad for the stability of any large system. But more likely: imagine all these orgs were distributed across three or four providers. A bad actor comes up with a zero-day for one of them. They can now a) go ahead and use that, far fewer systems are compromised and awareness of the threat is raised, or b) wait a much longer time until they come up with vulns for all the other systems. Either of those is less bad than the current situation. These days it's starting to feel like China might get to a point where they could shut down an entire country, all at once, with the flip of a switch.
    - exporectomy 6y ago
      One hack happening doesn't raise awareness for the risk of different unknown vulnerabilities in different software. So the total number of institutions getting hacked would be the same. It's not really one system. It just looks like that because it's one news story. If instead, all school districts were hacked this year and all police departments next year, how is that any better than both together? If it was one system like one network, your idea is even worse because having more different software increases the attack surface so hacking any one of those compromises the whole system. Would you personally use uncommon software to avoid being part of a big hack like this? I don't think that's a valid way of protecting yourself. Your idea would make sense if many of these institutions were just providing services that were redundant with each other. Then if some of them are disabled, the others can take their place. But a police department's email server can't do the job of a school district's one. And if confidential information is taken in a hack, redundancy doesn't help at all.
bezelbuttons 6y ago
> Adair said he’s fielded dozens of calls today from state and local government agencies that have identified the backdoors in their Exchange servers and are pleading for help. I can imagine they are sending an email to support@microsoft.com pleading for help. A future attacker would be well served to deny email to be sent to any mailbox @microsoft.com EDIT: I'm now realizing that this follows the Microsoft-angle of the Solarwinds' attack. These customers are not going to be happy with $MS
- mschuster91 6y ago
  > EDIT: I'm now realizing that this follows the Microsoft-angle of the Solarwinds' attack. These customers are not going to be happy with $MS Won't hurt MS in the long run. There is no viable alternative to switch to, for any of their products: * OS: macOS runs only on expensive Apple hardware, Linux can't run business software, plus both have retraining costs for employees * Office software: Libreoffice just... doesn't cut it, let's be honest. Apple's stuff only runs on Macs. * Exchange: Lotus Notes is dead, and while there are open source solutions, there is no comprehensive single solution.
  - koprulusector 6y ago
    I’ve been using Linux for work and personal for 4 years. Almost everything used in the enterprise today has a web app/electron version or runs natively. Including MS products themselves. I laugh maniacally every time I’m working in Word online... muahahaha
    - Shared404 6y ago
      When Microsoft starts writing apps for Linux...
      - EvanAnderson 6y ago
        There's Teams... >snicker<
exporectomy 6y ago
I can't tell from the article, but was this vulnerability already being exploited but to a lesser extent or did the hackers apparently discover it as a result of the patch being released? If the latter, then maybe we need processes for patching faster than people can reverse engineer the patches.
- krebsonsecurity 6y ago
  Yes, it was being used to target specific organizations prior to Microsoft's patches this week. Since then, attackers have basically used tools like Shodan to find unpatched servers, and mass-backdoored them -- regardless of who the victim organization is.
  - deleted 6y ago
    [deleted]
  - achillean 6y ago
    Do you have any details you can share with us (support@shodan.io) about how attackers are using Shodan? We have a lot of mechanisms to prevent abuse (blocking anonymous access, limiting number of results/ searches, restricting certain search filters) and if there's more we can do please let me know. Btw Microsoft, CERTs and a bunch of other orgs are also using Shodan to find out who is exposed. We already had all the data to determine vulnerability before the announcement was made so enterprise customers could search their local Shodan database for affected systems. And we've been sending out notifications as well.
    - winthrowe 6y ago
      I don't think that's an accusation against you, but I have to imagine there's a Shodan inspired darkweb site somewhere that takes crypto in exchange for bypassing all those noble restrictions.
    - xbar 6y ago
      Keep it real, Shodan-bro. Thanks for the additional context. Lovin' my membership.
diskmuncher 6y ago
MSFT still outperformed SP500 index this week.
- panarky 6y ago
  Security vulns are a profit center for Microsoft. I have a client who was hit with ransomware that exploited holes in RDP. They paid Microsoft about 5% of their annual IT budget to upgrade. How much more license revenue and 365 subscriptions will this latest fuckup generate? And if vulns are this profitable, where's the incentive to prevent them in the first place?
  - johncessna 6y ago
    > And if vulns are this profitable, where's the incentive to prevent them in the first place? Prior to upgrading their software, where was the incentive for your client to keep everything up to date and put in the infrastructure needed to patch all of their systems minutes/hours/days of a new zero day? I can't speak for your customer (obviously), but do you think they would have invested 5% of their budget in upgrades for this particular hack? A ransomware attack shuts you down. This is blackmail/corporate espionage stuff. Very easy to ignore depending on what your company is saying in their email.
  - tester756 6y ago
    >about 5% of their annual IT budget so basically for free / at low cost?
- tinus_hn 6y ago
  People are still buying into the ‘nothing is secure, they can’t help it’ storyline.
annoyingnoob 6y ago
This is the kind of thing that keeps me up at night.
- MrMrtn 6y ago
  For those of us working incident response, it is exactly what is keeping us up at night these days
social_quotient 6y ago
I wish the title was a bit more clear from the original post. This feels a little bit vague on purpose. Microsoft Exchange server software , not to be confused with MS Outlook email software or the lesser Windows Mail software.
- klingon79 6y ago
  Exchange is often externally open in some way for OWA. One that server is hacked, you may be wide-open internally. I’d be at least as concerned about an Exchange vulnerability as I would be about Outlook, but probably more.
  - matwood 6y ago
    Maybe it's because I haven't dealt with MS products in awhile, but my first thought was who puts OWA on the open internet without requiring a VPN? That's just asking for trouble.
tehjoker 6y ago
They attribute the attack to a particular actor without providing any evidence to the public. A bug could exist that enables such an attack, but it's not proven any emails were ever even taken. They did find a tool left behind it seems. I am just increasingly skeptical of these hacking stories that have a nat sec angle on them after the previous ones have been shown to be mostly or entirely fraudulent years later.
- fouric 6y ago
  > the previous ones have been shown to be mostly or entirely fraudulent years later ...they said, while providing no evidence to the public.
  - tehjoker 6y ago
    I was referring to Russiagate and the allegations made over Wikileaks. Even if I was some kind of international troll (hint: I'm not), it wouldn't matter because it remains true that the attribution is an evidence free assertion about the current target for dirty tricks by the USG. It moved a few millimeters across the world map. "Bombshell: Crowdstrike admits ‘no evidence’ Russia stole emails from DNC server" https://thegrayzone.com/2020/05/11/bombshell-crowdstrike-admits-no-evidence-russia-stole-emails-from-dnc-server/ https://thegrayzone.com/2020/05/11/bombshell-crowdstrike-adm...
    - elefanten 6y ago
      Those examples are nothing like hack disclosures.
      - tehjoker 6y ago
        On the contrary, they literally were allegations of hacks in several cases. Remember "Russia is hacking our democracy!"?
    - withinboredom 6y ago
      Small map you have there
rhacker 6y ago
I remember this kind of thing happening all the time in the 90s and part of the 00s... It's just 10 to 1000 times worse now days since EVERYTHING is online now.
- yudlejoza 6y ago
  I wasn't aware Exchange Server was still this prevalent, and that its pwnage was still alive and kicking. Great job M$.
  - Triv888 6y ago
    which one is your favorite alternative?
    - CraigJPerry 6y ago
      Postfix. But that’s only an MTA i hear you cry, Exchange does both MTA & MDA! Bear with me. Postfix is software to learn from. It might be written in C but the architecture is the epitome of beautiful modular design. It’s not just the meticulous separation of concerns, the care and attention to detail, everything from string handling to memory management is pristinely handled. https://github.com/vdukhovni/postfix https://github.com/vdukhovni/postfix Even at runtime the beauty of the architecture allows for a sysadmin to choose (via master.cf) exactly how the components should be composed to fit their needs. The defaults are crafted for minimum fuss if you just need to get it running ASAP. The software is ergonomic in addition to being artfully crafted. So what does all this care and attention get you? Only 9 CVEs in 22 years, only 3 of which are code exec, only 2 of which are (maybe) remote code exec, only 1 of which is unauth user RCE - but very hard in practice to exploit. Maybe it’s just not that popular? It was 1/3 of all SMTP servers on the internet according to a 2019 scan. So it’s the best MTA ever to exist, but what about MDA? Well, that was the whole point. Compose well crafted components together to build a system. You especially don’t run part of your mailserver’s web interface in kernel space because, well i’m not sure why IIS/Exchange does that :-)
      - 6y ago
thepill 6y ago
Does anyone know how to check for malecious activity on exchange 2010? All the logs/tools explained in the articles do not exist befor exchange 2013
- EvanAnderson 6y ago
  With the caveat of "You know you shouldn't be running that anymore": Look for unauthenticated POST requests to "/ECP" URLs as a starting place. I haven't dug into 2010 (because I tried really hard to get rid of it in the run-up to end-of-support), and since there's no PoC available I can't try exploit code against one to see.
- throwaway366tt 6y ago
  If you have a Microsoft rep, hit them up for a patch, once you’ve switched off OWA. Exchange 2010 is out of support, but not super super out of support.
- WarOnPrivacy 6y ago
  Here's the patch: https://www.microsoft.com/en-us/download/details.aspx?id=102774 https://www.microsoft.com/en-us/download/details.aspx?id=102... Description: https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-server-2010-service-pack-3-march-2-2021-kb5000978-894f27bf-281e-44f8-b9ba-dad705534459 https://support.microsoft.com/en-us/topic/description-of-the...
networkimprov 6y ago
Yet another superb reason not to run your internal company comms on a publicly accessible email server. Or to replace email for internal use altogether. TMTP is a new protocol with that goal: https://mnmnotmail.org/ https://mnmnotmail.org/ https://twitter.com/mnmnotmail https://twitter.com/mnmnotmail
- closeparen 6y ago
  You can publicly expose an SMTP relay while keeping Exchange itself private, right?
  - EvanAnderson 6y ago
    Absolutely. The parts of Exchange that got exploited are exposed because people want ActiveSync on their phones and web-based email. If you did without on those, or ran them over a VPN, just having SMTP exposed didn't make you vulnerable to this. Hardly anybody does that, though.
  - Spooky23 6y ago
    Yes totally. Most of the people running exchange now with this issue are cheapskates or attorneys. If defense contractors keeping exchange on prem for security/compliance reasons are offering OWA on the internet, obviously there’s a deeper problem.
  - xbar 6y ago
    My Exchange server avoided this mess by living in this state.
- bjarneh 6y ago
  > As more sites adopt TMTP for their own reasons Isn't this the problem in replacing almost any technology that we know is "broken", it is often too ingrained to be replaced easily.
  - networkimprov 6y ago
    Consider the huge variety of messaging and discussion apps; it's relatively easy to embrace new communication tools. EDIT: you might never silence SMTP altogether, but a suitable protocol could supplant it for the great majority of its use cases.
j3th9n 6y ago
Slightly related, on BBC iPlayer is currently an interesting documentary series available called "China: A New World Order", which touches hacks like these a couple of times.
- te_chris 6y ago
  Can attest to this being very good
waynesoftware 6y ago
"This is the real deal," tweeted Christopher Krebs, the former CISA director. "If your organization runs an OWA server exposed to the internet, assume compromise between 02/26-03/03."
- imglorp 6y ago
  Chris acknowledged Brian as his "brother from another mother." :-) I was wondering...
mark-r 6y ago
So if you discover one of these hacked servers, how should you let them know - send them an email?
- vmception 6y ago
  1. Find approximate geographic location (whatismyip.com, traceroute, a couple pings to nearby datacenters) 2. Do a speedtest 3. Add location and speed to remote desktop access marketplaces on darknet 4. Collect passive income from renters looking for clearnet computers in certain areas to use. Often times all the known VPN IP addresses are polluted - even their "dedicated residential IPs" and this can ensure you have worse treatment on the internet, such as more captchas, outright bans, inability to use streaming services, and for actual criminals it means their stolen credit cards don't work. But with remote desktop marketplaces, you can find a computer near the postal code of the credit card you have and this ensures your online transactions go through. Obviously not "you" as you don't have to care what the people do on the other side of your tollbooth. Since you weren't the one compromising anything (computer, credit card, any actual spending) you'll be fine, but you're also going to do all this over Tor anyway. But because you'll be fine you don't have to worry about being detected due to some flaw in Tor because you won't have triggered a criminal investigation, the actual hackers and skimmers and thieves will have though, and incurred all the liability for themselves, people who will have paid an address on a darknet marketplace in Monero and gotten temporary access to a server.
  - 55555 6y ago
    This is the answer to a very different question.
    - vmception 6y ago
      haha you're right, I read "what would you do" not "how to contact" but I also figured that the spike in traffic or someone messing up their botnet's activity windows would alert the computer owner to something
lizardmancan 6y ago
Im still of the opinion that closed source should be illegal. How stupid is this idea?
technion 6y ago
We've got some information on the timeline (and a name) on one of the major exploits here: https://proxylogon.com/ https://proxylogon.com/ Some of the detail on where this is a mess - The relevant security update is only offered for the latest (-1) Cumulative Update for Exchange. So you can open Windows Update and it will say "fully updated and secured", but you're not. Complicating matters, Cumulative Updates for Exchange 2019 have to be done from the licensing portal, with a valid logon. So maybe you have a perfectly capable 24x7 tech team, but the guy who manages license acquisition is on leave today. This is how you may basically find yourself resorting to piracy to get this patched.
- bombcar 6y ago
  Reminiscent of Cisco IOS patches being stuck behind support contracts - and inaccessible to many until they pony up.
  - oasisbob 6y ago
    It's been a while since I've had to deal with Cisco IOS, but IIRC they were always good about releasing security fixes to anyone upon a TAC request. For used devices off support contract, security incidents were a great opportunity to get free updates.
    - sneak 6y ago
      IIRC you needed a contract login to open TAC cases. I always just got copies of the .bins from friends who worked at places that had contracts. They didn't gate updates at that time by which model you bought, once you had access you could get firmware for anything Cisco.
      - oasisbob 6y ago
        Phone or email would work too, no contract required. > As a special customer service, and to improve the overall security of the Internet, Cisco may offer customers free software updates to address high-severity security problems. The decision to provide free software updates is made on a case-by-case basis. Refer to the Cisco security publication for details. Free software updates will typically be limited to Critical and High severity Cisco Security Advisories. > If Cisco has offered a free software update to address a specific issue, noncontract customers who are eligible for the update may obtain it by contacting the Cisco TAC using any of the means described in the General Security-Related Queries section of this document. https://tools.cisco.com/security/center/resources/security_vulnerability_policy.html https://tools.cisco.com/security/center/resources/security_v...
a-dub 6y ago
the old wisdom used to be "don't expose microsoft stuff directly to the internet" apparently that's still true?
- technion 6y ago
  This take was always problematic in my view. It's the reason that I had years of inherited servers with a proxy sitting in front of exchange that consistently broke things everywhere it was done. In practice, this exploit is just another in the line of exchange issues that would get forwarded directly through a proxy to the backend unchallenged. Meanwhile the only place I've heard of pushback against applying this patch, is the "but we have a proxy we're secure" crowd.
  - a-dub 6y ago
    that's still a form of exposing microsoft stuff directly to the internet, in my view.
    - ryanlol 6y ago
      Are any other vendors better? I don’t think this is a MS issue.
      - robbyt 6y ago
        I've heard there is a few Redhat servers on the internet
        ryanlol 6y ago
        Which suffer just as many security issues if you try to do something complex with them.
      - a-dub 6y ago
        much of the software and design for MS stuff is from a period in personal computer history when people weren't worrying as much about public internet style security problems, so it has always seemed to have been at a disadvantage in the internet era which they have fought hard to try and overcome, but nonetheless a lot of code and culture remains. this affects not only the operating system and platform itself, but also major applications, development philosophies, major utilities and even the approaches used to operate it in production. it's actually an interesting question, while internet security problems largely outmoded old pc inspired designs and product-market fit (the diy part time sysadmin), will they outmode the personal operation of any software... that is, will computer security problems grow to the point to where everything must be actively managed and defended?
riffic 6y ago
lol - don't run services you can't competently manage. edit: this tweet restates this in a much nicer way: https://twitter.com/SwiftOnSecurity/status/1366867228914810880 https://twitter.com/SwiftOnSecurity/status/13668672289148108... > If you're not an F50 running your own Exchange Server is organizational clownery at this point.
WheelsAtLarge 6y ago
MS, Solarwinds, ... I suspect that the number of compromised software companies are much larger than these 2 companies. I'm almost certain that we will hear about others in the future. If you manage a software product I hope you are auditing the code regularly. You should also harden the security for it and who has access to the source code and its build no matter how unlikely you think you are a target.
- judge2020 6y ago
  > I suspect that the number of compromised software companies are much larger than these 2 companies. Given one of the CVEs is CVE-2021-26857, there has already been more that 26000 vulnerability submitted for an CVE ID this year so there are indeed countless other compromised systems - the two big hacks of recent are only in the news thanks to their large blast radius.
- xbar 6y ago
  Today is the day for all of the other victims to disclose under the umbrella of Microsoft attention.
jariel 6y ago
This needs to be considered an issue of national security and the US forces needs a 'Digital Force' more than they need a 'Space Force'.
- yborg 6y ago
  https://en.wikipedia.org/wiki/United_States_Cyber_Command https://en.wikipedia.org/wiki/United_States_Cyber_Command
- ThinkBeat 6y ago
  As another replier mentioned, the US has that and has had that in DOD as well and have had it for longer. However, NSA, has been around a long time. Dont forget about them. If by "applying digital force", do you mean attacking other countries? If so, does the same apply when the US destroys computer systems in other countries? If your country is attacked by the US what force are they reasonably allowed to use as a counter attack? You ruin one or more nuclear weapons facilities; they get to destroy a few nuclear installations of several types in the US? The US is not sitting around being the innocent victim. The US is engaged in offensive attacks on regular basis. (At the same time the US is engaged in massive real world war as well. Unike most of its counterparts. Oh "military conflicts" not war. Unless you are the country at the receiving end of "military conflict" in which case you will have to spend a lot of time trying to figure out why it is not war.
  - xbar 6y ago
    I see several straw men and an anti-war argument. None has the mission of broad national defense of civilian assets from cyber warfare by foreign nation states.
  - 1MachineElf 6y ago
    Your lack of close parenthesis might have been a typo, but I think it does well to symbolize that the point you're making is less an ancillary explanation and more a real world issue that we can't just ignore.
_robbywashere 6y ago
The United States Government should actively be trying to protect its businesses. They should create a three letter organization to do so. They should call it the National Security something or another.
- labster 6y ago
  That name is already taken by the department of hodling cyberattacks. They should have a National Vulnerability Agency that handles it.
  - jjeaff 6y ago
    Or maybe let's revisit the charter of the NSA and make some major tweaks.
    - gogopuppygogo 6y ago
      Step 1.) Let marijuana users be employed so you can attract talent. Step 2.) Pay above market rate for talent, even import it from Israel or other friendly nation states. We need a Wernher von Braun style approach to recruitment. Step 3.) ???? Profit ????
      - astrea 6y ago
        Step 1 sounds personal
        jcims 6y ago
        It sounds like it, and it might be, but testing for weed really does impact recruiting. I know of a very large US firm that has quietly stopped including it in their drug test over the past 3-4 years in part because of that. (I wouldn’t be surprised if folks start to push testing as a HIPAA issue.)
        Godel_unicode 6y ago
jtdev 6y ago
How does Microsoft bear no financial liability for the many major security flaws in their for profit software? I’m sure they have clauses in their legal agreements, but come on...
roody15 6y ago
The cynic in me thinks it’s not a coincidence that the cloud office 365 was not affected. Almost like a certain company would like to get its customers to migrate AD to Azure and Exchange to full office 365.
- Someone1234 6y ago
  One uses Outlook Web Access (where the first exploit exists), the other does not. If the hosted version lacks the component that has a security issue, it won't have that issue, it is technically misinformed to conclude anything nefarious.
WarOnPrivacy 6y ago
Exchange Server 2010 patch is here https://www.microsoft.com/en-us/download/details.aspx?id=102774 https://www.microsoft.com/en-us/download/details.aspx?id=102... Description: https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-server-2010-service-pack-3-march-2-2021-kb5000978-894f27bf-281e-44f8-b9ba-dad705534459 https://support.microsoft.com/en-us/topic/description-of-the...
Keverw 6y ago
Scary! my university uses Microsoft for email, but I think they use the cloud hosted version but wonder how much code is shared between the versions. When I added it to the mail app on my iPhone, it mentioned it could wipe my device. Guess that's a default with the implementation but that is a turn off. So I ended up just installing the Outlook app instead since couldn't find imap support. I feel like on desktop, just using the web version or even adding it to my home screen would be another use but partly was hoping to just have all my accounts together.
- ncann 6y ago
  If you're on Android there's an app to get around that device management requirement when adding an email account.
  - senectus1 6y ago
    Can you elaborate?
    - ncann 6y ago
      The app is named "Exchained": https://play.google.com/store/apps/details?id=mobi.biko.exchained&hl=en_CA&gl=US https://play.google.com/store/apps/details?id=mobi.biko.exch...
xyst 6y ago
my hate for anything micro$oft is justified, again
peter_retief 6y ago
Exchange has been a security problem since 1998. Surely there are open source solutions available that have better security? Seems obvious, have I missed something?
867-5309 6y ago
well it is Patch Friday after all
edrxty 6y ago
Bigger picture, what's the endgame here? It seems a lot of institutions handling sensitive work are considering air-gapping some or all of their networks at this point. Maybe that's even what has to happen. Is there a means of fending off these attacks on the political front? If this same level of espionage was happening in person, there would be a kinetic response but it seems everyone is happy to just turn the other cheek. These attacks have a very real impact. Copying others homework is a tried and true way to get a technological edge and in practical terms, it means a lot of research and development money is effectively wasted as it doesn't generate any returns. Mind, I don't think there should be a violent response, but it's odd that even the threat of sanctions isn't made whenever this happens.
- dcow 6y ago
  I’m curious to hear more about cases of large institutions seriously considering air-gapping. This is the first I’ve got wind of something like that.
  - lrem 6y ago
    Yeah, runs contrary to my perception too. Even things that one would reasonably expect to be air-gapped are online these days.
  - bob1029 6y ago
    Air-gapped systems really only make sense for the occasional need to access exceptionally sensitive materials. I.e. private keys for root CAs. For most businesses, air-gapping would mean we are back in the 20th century of business with filing cabinets and armies of people pushing paper between 2 rooms.
    - edrxty 6y ago
      It's not actually that bad. There's a lot of defense, security, and highly proprietary development that happens on isolated networks. You have to put significant effort into IT infrastructure but you'll end up with all your stuff hosted internally and most tools support custom package repo mirrors (linux distros, programming languages/build systems, docker). You'll also probably have a second system with internet access at your desk if not nearby for stackoverflow et al. Basically the idea is defense in depth. The valuable stuff (design files, schematics, code, documentation) lives in the air gapped network while communications live inside a VPN and detailed technical discussion is often discouraged.
AndyMcConachie 6y ago
I really wish the reports on hacks could treat attribution more seriously. Everytime a hack like this occurs it gets blamed on 'the Chinese', or 'the Russian', or 'the Iranians', without every showing any evidence to prove this. Attribution on the Internet is hard, like really hard. I want proof. And if you don't have proof, or can't show me the proof, then don't just blame Americas enemies. It's sloppy and dangerous.
- amanaplanacanal 6y ago
  I would guess that many times the evidence is from NSA signals intelligence and they can’t show their work because it’s classified. We end up just having to take their word for it.
  - baobabKoodaa 6y ago
    How convenient that the perpetrators always turn out to be political enemies.
- hntrader 6y ago
  I agree that they should show proof. But if they won't show proof yet it's nevertheless true and they have strong privately held evidence concluding it (perhaps from the NSA), that doesn't suddenly make it dangerous to blame the actual perpetrator. It's only dangerous if they're doing it incorrectly or presumptively or deceitfully (which you don't know to be the case).
gala8y 6y ago
Article focuses on US, but this is global. > “It’s massive. Absolutely massive,” one former national security official with knowledge of the investigation told WIRED. “We’re talking thousands of servers compromised per hour, globally.”
- annoyingnoob 6y ago
  It is really an epic hack, historic.
  - PenguinCoder 6y ago
    Just one more in the long line of 'historic' recent hacks.
chovybizzass 6y ago
I'm loving this. I hate microsoft. I hope they die and Bill Gates gets convicted of something heinous.
- dieortin 6y ago
  Bill Gates hasn’t been CEO for a long time now.
jimnotgym 6y ago
There will always be stolen emails. The problem is that the emails are in plain text on the server...
mjthompson 6y ago
I'm curious to know why this did not affect Office 365 / Exchange Online. I used to work for a law firm which ran on-premises Exchange, but had OWA running behind a VPN. I remember finding it extremely inconvenient at the time. But they're the ones laughing now.
- upofadown 6y ago
  Apparently because they are different code bases. So the answer could be: random chance.
mattowen_uk 6y ago
I patched my Exchange servers the morning this was announced, a few days ago. The patch takes about ten minutes per server, and does not require a reboot. If your server was a client facing one (CAS) users would have seen a brief outage in Outlook connectivity. The patches were single file downloads, one for each version of Exchange, yes you needed to be on the latest Cumulative Update for Exchange, so if you weren't you really have no right running a production mail system...
- deleted 6y ago
  [deleted]
- ocdtrekkie 6y ago
  The last few security patches have been available independent of the Cumulative Updates, so it was reasonable to be a few behind. But this one required the latest CU to install. Bear in mind after updating you still need to check if you were already hacked.
ManlyBread 6y ago
One can only ask what's the point of the forced automatic updates when this stuff is still happening at this scale.
- ocdtrekkie 6y ago
  There's no forced automatic updates of server software. In fact, Exchange CUs can only be installed manually by mounting an ISO file on the server.
ratiolat 6y ago
Wonder what has changed. It was standard practice 15 years ago not to expose Microsoft Exchange (nor any other Microsoft product) directly to internet.
- bearbawl 6y ago
  It was standard practice 15 years ago to never ever use Microsoft Exchange in the first place. Wonder what has changed.
r1ch 6y ago
What are the chances this was independently discovered and weaponized in the two months after the original report to MS? Can't help but wonder if the security researcher or MSRC were compromised or have a leak.
- deleted 6y ago
  [deleted]
- GraemeMeyer 6y ago
  CISA are indicating that the attacks go back to at least September: https://us-cert.cisa.gov/ncas/alerts/aa21-062a https://us-cert.cisa.gov/ncas/alerts/aa21-062a
porsager 6y ago
There's a powershell script to check your server here: https://github.com/cert-lv/exchange_webshell_detection https://github.com/cert-lv/exchange_webshell_detection
grandinj 6y ago
Isn't time to simply pull the plug on any and all internet access from China until they at least pretend to be discrete about their hacking? (In the old days, there was the Usenet Death Penalty for persistently bad actors)
bluedino 6y ago
Couldn’t you put these servers behind something like CloudFlare? Assuming they were knowledgeable of the attack and could block it.
Threeve303 6y ago
This will be the nail in the coffin for on premise email servers. Putting all of your eggs in one basket might be an even worse idea over time.
- ocdtrekkie 6y ago
  Eh, this could just as well have happened to everyone with 365 as well. On-premise servers allow you to manage the risk at your own level. For instance, you can decide not to expose a service to the Internet that 365 does.
bob1029 6y ago
We are seriously looking at strategies for clean room rebuild of our IT infrastructure, potentially on a recurring basis via automation. Obviously, you cant mitigate 0-day exploits in any situation where reasonable/expected network access is possible. But our concern, despite not being directly impacted by this, is that we may have accumulated malware over the past decade+ that has never been discovered. How many exploits exist in the wild which have never been documented or even noticed? Do we think it's at least one? The thinking we are getting into is - If we nuke-from-orbit and then reseed from trusted backups on a recurring basis, any malware that gets installed via some side-channel would not be able to persist for as long as it traditionally would. Keeping backups pure via deterministic cryptographic schemes is far easier to work with than running 100+ security suites across your IT stack in hopes you find something naughty. It is incredibly hard for malware to hide in a well-normalized SQL database without SP or other programmatic features. What if we built a new IT stack that was designed to be obliterated and reconstructed every 24 hours with latest patch builds each time? Surely many businesses could tolerate 1-2 hours of downtime overnight. It certainly works for the stock market. There really isn't a reason you need to give an attacker a well-managed private island to hide on for 10+ years at a time.
- iforgotpassword 6y ago
  We use netboot for most desktop computers and servers that are mostly stateless. Any changes are temporary ending up on a dedicated temp partition that gets wiped on boot, or in ram. Rebuilds are mostly automatic. Of course, netboot in itself opens new attack vectors where we're in early stages of exploring different approaches, even the painful secure boot crap. Honestly I think most of the security in our case right now comes from being an obscure in-house solution that you'd need to specifically target. Also, in case you do get pwned, a post mortem becomes mostly impossible since once you reboot a machine, everything is gone except stuff on network shares of course.
- mlac 6y ago
  I’ve thought a lot about this. I think from a tech standpoint and a security standpoint, my ideal approach would be to rotate out an A team and a B team. Every 2-3 years, the teams switch off. So year 1-3 A team is running the environment. B team is completely rebuilding and re-architecting the organization’s IT. The company is migrated to B teams infrastructure for 3 years. A team gets to re-build while B team is running and the cycle repeats. This has a few advantages, it keeps the org very current with tools and technology, everyone stays sharp on the latest tech, nothing is sacred, and teams get experience across the spectrum of design build implement and run. It also has good Disaster recovery properties if you idle the old environment so that you can fall back if some critical failure occurs in the new environment. This would be expensive, but please poke holes. I like your idea of clean rebuilds and can see a path to it with automation / terraform / cloud resources. And you don’t need the downtime if you stand up the second one in parallel and just fail over. There’s still persistent data that needs to carry through, so you’d need to figure out how to separate your persistent data from the elements that reset.
chovybizzass 6y ago
lots of pentest jobs available https://startworkingremotely.com/jobs?q=pentest https://startworkingremotely.com/jobs?q=pentest

1038 points by parsecs 6y ago

Wow. Patching (or using cloud mail providers) would have mitigated the risk for this one...and many others in the past (and the future). The cleanup from this is big for those who were hit. Launching attacks during major news events surely also helped the attackers stay under the radar for longer.

walrus01 6y ago
If I had to guess it's a huge laundry-list of organizations that for some legacy reason (Going back 10, 15, 20 years) are running on-premises Exchange, and don't have a full time person one of whose roles is to keep up on patches, security advisories and such.
- logifail 6y ago
  > to keep up on patches, security advisories and such Until you've personally experienced the full horror of attempting to keep on-premises Exchange patched, especially in the SME space where you may have few servers, it's hard to imagine how awful this is. Cumulative Updates are essentially "completely uninstall Exchange" and then "reinstall Exchange again". This is not what one might call a "patch". Then you get into dependencies on .Net and suddenly you need to upgrade the OS as well while you're in the middle of completely-uninstalling-and-reinstalling-Exchange. Last time I got sucked into this, I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system.
  - walrus01 6y ago
    Thankfully for my mental well being it has been 15+ years since I touched Exchange.
  - EvanAnderson 6y ago
    It's hardly a "full horror". I manage on-prem Exchange in the SME space, with single-server installations and multi-server installations (with and without high availability). The patching process is, arguably, inefficient (doing full installs over top of the existing installation) but, in terms of success rate, I've had good luck. I wouldn't put out any new on-prem Exchange today, but the ones I support have reasons to be on-prem or planned migration off-prem. Aside: I've been administering Exchange since version 4.0. I've never experienced "horrors" like so many people talk about. Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems that I've been privy to by way of friends, emergency engagements with non-Customers, etc.
    - logifail 6y ago
      > Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems I'm sure there are some SMEs who are happy to throw serious budget at doing on-prem Exchange "right". For everyone else, I'm not sure what they're supposed to do.
      - EvanAnderson 6y ago
        Everyone else pays for monthly Office 365 subscriptions and ends up spending more money. (Which is what I recommend now, but it galls me to no end.) I don't buy the "Exchange is expensive to support" argument. It's cheaper on-prem than paying for the subscription. We always saw break-even at around 16 - 20 months. I have billing records for a small business Customer w/ a single Exchange 2016 server for last year that amount to 6.5 hours for the entire year, including installing CU's 16 thru 18 (CU 19 fell in this year). Yes-- a piece of their overall Windows Update application budget applies to Exchange, as does the amortized cost of backup software, and server computer and support hardware. Even w/ the OS license, Exchange license, and CALs at 120x an Office 365 E3 monthly subscription they're still money ahead over the 4+ years they've been running Exchange.
        walrus01 6y ago
        However from the point of view of a medium sized business paying for office365, in terms of dollar per month per employee, they're getting much more than just exchange, they're getting onedrive, sharepoint, teams, and the office suite software itself as well.
  - Spooky23 6y ago
    I used to run a large on-prem exchange system with about 75k users. It’s literally the only product I’ve ever seen where the admins were the biggest, loudest advocates for outsourcing it to the predecessor to O365. It was more beastly back to run back then though. We did reduce our risk profile at the time by putting OWA behind a sslvpn and only allowing BlackBerry.
  - ocdtrekkie 6y ago
    It'd be nice if CUs were easier to install, but on-prem Exchange management isn't that much work once it's running smoothly. It'd be nice if they made it easier to firewall off more from your AD environment too. But most Exchange management I do is mailbox management, and you have to do that if it's in the cloud too.
    - EvanAnderson 6y ago
      This jibes with my experience. My Customers who have migrated to Office 365 have been using roughly the same labor as when they had on-prem Exchange. (If anything, they're using a little more.)
  - cutemonster 6y ago
    > I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system What did they reply?
- datavirtue 6y ago
  These people are always pulling double duty with Exchange administration being a bemoaned afterthought. I can barely get my head around the scope of this.
brundolf 6y ago
The cloud angle is interesting; on one hand, it creates an even-more-centralized single point of failure. On the other hand, given that virtually every computing system out there is a house of cards, letting the experts focus on securing (and updating!) just a single one might be the best defense.
- mywittyname 6y ago
  The cloud providers can afford to hire and train elite teams to handle security. I remember seeing a post about a guy trying to break out of the docker container used by Cloud SQL on GCP, and apparently the GCP admins made it known that he was being watched pretty early on. I believe the issue was patched fairly quickly too. It's possible that <Random F500 Co> has a great security team. But it's also possible that <Other F500 Co> doesn't.
  - brundolf 6y ago
    Really what we need is the ability to self-host reasonably secure systems without a team of experts working round the clock... but that doesn't appear to be the hand we've been dealt
    - belval 6y ago
      I might be biased because I work at AWS, but I really doubt that there are enough sys admins that know what they are doing and keep up to date let alone find vulnerabilities in the software they use to protect all companies. A Fortune 500 maybe, but at some point you simply can't afford someone who knows what he's doing and at that point you might as well have everything in the cloud so you can focus on your actual money making business.
      - kaliszad 6y ago
        Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution to actually sit down and make simpler systems that a few skilled people can fit in their heads and actually understand?
EvanAnderson 6y ago
The vulnerabilities being exploited were all zero-day. Up-to-date installations were still vulnerable.
kaliszad 6y ago
The proper mitigation would be actually using much simpler, better quality software. Microsoft Exchange Server is quite famous for being an attack vector on corporate networks. At my previous job, the company was advised (by a very capable and expensive security consulting company) to keep Exchange as separate as possible from the corporate network - this of course is a bit counter intuitive, when you want to use e.g. Single Sign-On, contacts and more typically with Active Directory (AD). Thankfully my job wasn't to administer or develop any solutions for AD or Exchange so I just took a note. Obviously, no engineer can have even a sufficient overview of the full Exchange Server implementation not speaking of full understanding. In such a situation security, quality and user (or admin for that matter) experience always take a big hit. It doesn't help Exchange Server is most likely developed using programming languages and approaches that more or less demand complecting the solution with OOP-related ceremony. Supporting two decades or more of legacy features and protocols doesn't help. Some companies even want to connect AD and Exchange to SharePoint... which is at least as complex as Exchange. The problem companies don't understand is that you have to work on simplifying, which is very hard - much harder than adding features. If you don't, the interactions between components will overwhelm even the largest and best skilled team on the planet. The result is, we see breaches and security issues like this every day and realistically, nobody who can decide anything in the corporate environment gives a f** anymore because nobody pays the more or less laughable fines with their own money and nobody really goes to jail but the user data is lost, peoples lives are shattered.
- EvanAnderson 6y ago
  Exchange product architecture was absolutely to blame for this. Very particualrly, the "/ECP" directory should have never been allowed to be Internet accessible. (I believe the upcoming version finally rectifies that in a "supported" way.) In general, though, Microsoft hasn't focused enough on making Exchange more compartmentalized. The servers' privileges in Active Directory are too high (though this is supposedly being addressed in the upcoming version too.)
  - kaliszad 6y ago
    Thank you for the insight. Certainly, "in the upcoming version" is a bit late for those affected and most of those other Exchange-related hacks in the past. The thinking around Exchange is still more or less left in the 20th century and it shows.
fbelzile 6y ago
> (or using cloud mail providers) Why? I don't see moving to a cloud solution being much better. The cloud service itself would be the single point of failure and would be just as vulnerable to a zero day. The organization would have even fewer risk mitigation options like NAT, firewalls, etc.
- jasdine817 6y ago
  As someone who surveys different organizations networks day in and out, the amount of unpatched and out of date Exchange servers (and other internet facing services) I see is ridiculous. Most sysadmins don't have a tangible idea of the risk they take when they set this stuff up. At least Office 365 is patched and monitored on a regular basis and has actual security teams tasked with looking for potential exploits.
jtdev 6y ago
The patches went out Tuesday... after many organizations were already compromised.
weare138 6y ago
It was a 0-day exploit. The patch wasn't released until March 2nd but the vulnerability was being exploited at least since January.

brundolf 6y ago

It's almost like all of our institutions shouldn't use the exact same software vendors

jgalt212 6y ago
or Identity Providers.
exporectomy 6y ago
Really? Wouldn't multiple softwares be equally vulnerable overall but the hacks would be more distributed in time as they're discovered at different times? Is that the problem you'd hope to solve? That it all happened within a few days instead of at different institutions at different times?
- brundolf 6y ago
  Yes, distributing the same number of hacks over a period of time would on its own make things a little bit less fragile. In general, having a single point of failure is bad for the stability of any large system. But more likely: imagine all these orgs were distributed across three or four providers. A bad actor comes up with a zero-day for one of them. They can now a) go ahead and use that, far fewer systems are compromised and awareness of the threat is raised, or b) wait a much longer time until they come up with vulns for all the other systems. Either of those is less bad than the current situation. These days it's starting to feel like China might get to a point where they could shut down an entire country, all at once, with the flip of a switch.
  - exporectomy 6y ago
    One hack happening doesn't raise awareness for the risk of different unknown vulnerabilities in different software. So the total number of institutions getting hacked would be the same. It's not really one system. It just looks like that because it's one news story. If instead, all school districts were hacked this year and all police departments next year, how is that any better than both together? If it was one system like one network, your idea is even worse because having more different software increases the attack surface so hacking any one of those compromises the whole system. Would you personally use uncommon software to avoid being part of a big hack like this? I don't think that's a valid way of protecting yourself. Your idea would make sense if many of these institutions were just providing services that were redundant with each other. Then if some of them are disabled, the others can take their place. But a police department's email server can't do the job of a school district's one. And if confidential information is taken in a hack, redundancy doesn't help at all.

bezelbuttons 6y ago

> Adair said he’s fielded dozens of calls today from state and local government agencies that have identified the backdoors in their Exchange servers and are pleading for help. I can imagine they are sending an email to support@microsoft.com pleading for help. A future attacker would be well served to deny email to be sent to any mailbox @microsoft.com EDIT: I'm now realizing that this follows the Microsoft-angle of the Solarwinds' attack. These customers are not going to be happy with $MS

mschuster91 6y ago
> EDIT: I'm now realizing that this follows the Microsoft-angle of the Solarwinds' attack. These customers are not going to be happy with $MS Won't hurt MS in the long run. There is no viable alternative to switch to, for any of their products: * OS: macOS runs only on expensive Apple hardware, Linux can't run business software, plus both have retraining costs for employees * Office software: Libreoffice just... doesn't cut it, let's be honest. Apple's stuff only runs on Macs. * Exchange: Lotus Notes is dead, and while there are open source solutions, there is no comprehensive single solution.
- koprulusector 6y ago
  I’ve been using Linux for work and personal for 4 years. Almost everything used in the enterprise today has a web app/electron version or runs natively. Including MS products themselves. I laugh maniacally every time I’m working in Word online... muahahaha
  - Shared404 6y ago
    When Microsoft starts writing apps for Linux...
    - EvanAnderson 6y ago
      There's Teams... >snicker<

exporectomy 6y ago

I can't tell from the article, but was this vulnerability already being exploited but to a lesser extent or did the hackers apparently discover it as a result of the patch being released? If the latter, then maybe we need processes for patching faster than people can reverse engineer the patches.

krebsonsecurity 6y ago
Yes, it was being used to target specific organizations prior to Microsoft's patches this week. Since then, attackers have basically used tools like Shodan to find unpatched servers, and mass-backdoored them -- regardless of who the victim organization is.
- deleted 6y ago
  [deleted]
- achillean 6y ago
  Do you have any details you can share with us (support@shodan.io) about how attackers are using Shodan? We have a lot of mechanisms to prevent abuse (blocking anonymous access, limiting number of results/ searches, restricting certain search filters) and if there's more we can do please let me know. Btw Microsoft, CERTs and a bunch of other orgs are also using Shodan to find out who is exposed. We already had all the data to determine vulnerability before the announcement was made so enterprise customers could search their local Shodan database for affected systems. And we've been sending out notifications as well.
  - winthrowe 6y ago
    I don't think that's an accusation against you, but I have to imagine there's a Shodan inspired darkweb site somewhere that takes crypto in exchange for bypassing all those noble restrictions.
  - xbar 6y ago
    Keep it real, Shodan-bro. Thanks for the additional context. Lovin' my membership.

diskmuncher 6y ago

MSFT still outperformed SP500 index this week.

panarky 6y ago
Security vulns are a profit center for Microsoft. I have a client who was hit with ransomware that exploited holes in RDP. They paid Microsoft about 5% of their annual IT budget to upgrade. How much more license revenue and 365 subscriptions will this latest fuckup generate? And if vulns are this profitable, where's the incentive to prevent them in the first place?
- johncessna 6y ago
  > And if vulns are this profitable, where's the incentive to prevent them in the first place? Prior to upgrading their software, where was the incentive for your client to keep everything up to date and put in the infrastructure needed to patch all of their systems minutes/hours/days of a new zero day? I can't speak for your customer (obviously), but do you think they would have invested 5% of their budget in upgrades for this particular hack? A ransomware attack shuts you down. This is blackmail/corporate espionage stuff. Very easy to ignore depending on what your company is saying in their email.
- tester756 6y ago
  >about 5% of their annual IT budget so basically for free / at low cost?
tinus_hn 6y ago
People are still buying into the ‘nothing is secure, they can’t help it’ storyline.

annoyingnoob 6y ago

This is the kind of thing that keeps me up at night.

MrMrtn 6y ago
For those of us working incident response, it is exactly what is keeping us up at night these days

social_quotient 6y ago

I wish the title was a bit more clear from the original post. This feels a little bit vague on purpose. Microsoft Exchange server software , not to be confused with MS Outlook email software or the lesser Windows Mail software.

klingon79 6y ago
Exchange is often externally open in some way for OWA. One that server is hacked, you may be wide-open internally. I’d be at least as concerned about an Exchange vulnerability as I would be about Outlook, but probably more.
- matwood 6y ago
  Maybe it's because I haven't dealt with MS products in awhile, but my first thought was who puts OWA on the open internet without requiring a VPN? That's just asking for trouble.

tehjoker 6y ago

They attribute the attack to a particular actor without providing any evidence to the public. A bug could exist that enables such an attack, but it's not proven any emails were ever even taken. They did find a tool left behind it seems. I am just increasingly skeptical of these hacking stories that have a nat sec angle on them after the previous ones have been shown to be mostly or entirely fraudulent years later.

fouric 6y ago
> the previous ones have been shown to be mostly or entirely fraudulent years later ...they said, while providing no evidence to the public.
- tehjoker 6y ago
  I was referring to Russiagate and the allegations made over Wikileaks. Even if I was some kind of international troll (hint: I'm not), it wouldn't matter because it remains true that the attribution is an evidence free assertion about the current target for dirty tricks by the USG. It moved a few millimeters across the world map. "Bombshell: Crowdstrike admits ‘no evidence’ Russia stole emails from DNC server" https://thegrayzone.com/2020/05/11/bombshell-crowdstrike-admits-no-evidence-russia-stole-emails-from-dnc-server/ https://thegrayzone.com/2020/05/11/bombshell-crowdstrike-adm...
  - elefanten 6y ago
    Those examples are nothing like hack disclosures.
    - tehjoker 6y ago
      On the contrary, they literally were allegations of hacks in several cases. Remember "Russia is hacking our democracy!"?
  - withinboredom 6y ago
    Small map you have there

rhacker 6y ago

I remember this kind of thing happening all the time in the 90s and part of the 00s... It's just 10 to 1000 times worse now days since EVERYTHING is online now.

yudlejoza 6y ago
I wasn't aware Exchange Server was still this prevalent, and that its pwnage was still alive and kicking. Great job M$.
- Triv888 6y ago
  which one is your favorite alternative?
  - CraigJPerry 6y ago
    Postfix. But that’s only an MTA i hear you cry, Exchange does both MTA & MDA! Bear with me. Postfix is software to learn from. It might be written in C but the architecture is the epitome of beautiful modular design. It’s not just the meticulous separation of concerns, the care and attention to detail, everything from string handling to memory management is pristinely handled. https://github.com/vdukhovni/postfix https://github.com/vdukhovni/postfix Even at runtime the beauty of the architecture allows for a sysadmin to choose (via master.cf) exactly how the components should be composed to fit their needs. The defaults are crafted for minimum fuss if you just need to get it running ASAP. The software is ergonomic in addition to being artfully crafted. So what does all this care and attention get you? Only 9 CVEs in 22 years, only 3 of which are code exec, only 2 of which are (maybe) remote code exec, only 1 of which is unauth user RCE - but very hard in practice to exploit. Maybe it’s just not that popular? It was 1/3 of all SMTP servers on the internet according to a 2019 scan. So it’s the best MTA ever to exist, but what about MDA? Well, that was the whole point. Compose well crafted components together to build a system. You especially don’t run part of your mailserver’s web interface in kernel space because, well i’m not sure why IIS/Exchange does that :-)
    - 6y ago

thepill 6y ago

Does anyone know how to check for malecious activity on exchange 2010? All the logs/tools explained in the articles do not exist befor exchange 2013

EvanAnderson 6y ago
With the caveat of "You know you shouldn't be running that anymore": Look for unauthenticated POST requests to "/ECP" URLs as a starting place. I haven't dug into 2010 (because I tried really hard to get rid of it in the run-up to end-of-support), and since there's no PoC available I can't try exploit code against one to see.
throwaway366tt 6y ago
If you have a Microsoft rep, hit them up for a patch, once you’ve switched off OWA. Exchange 2010 is out of support, but not super super out of support.
WarOnPrivacy 6y ago
Here's the patch: https://www.microsoft.com/en-us/download/details.aspx?id=102774 https://www.microsoft.com/en-us/download/details.aspx?id=102... Description: https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-server-2010-service-pack-3-march-2-2021-kb5000978-894f27bf-281e-44f8-b9ba-dad705534459 https://support.microsoft.com/en-us/topic/description-of-the...

networkimprov 6y ago

Yet another superb reason not to run your internal company comms on a publicly accessible email server. Or to replace email for internal use altogether. TMTP is a new protocol with that goal: https://mnmnotmail.org/ https://mnmnotmail.org/ https://twitter.com/mnmnotmail https://twitter.com/mnmnotmail

closeparen 6y ago
You can publicly expose an SMTP relay while keeping Exchange itself private, right?
- EvanAnderson 6y ago
  Absolutely. The parts of Exchange that got exploited are exposed because people want ActiveSync on their phones and web-based email. If you did without on those, or ran them over a VPN, just having SMTP exposed didn't make you vulnerable to this. Hardly anybody does that, though.
- Spooky23 6y ago
  Yes totally. Most of the people running exchange now with this issue are cheapskates or attorneys. If defense contractors keeping exchange on prem for security/compliance reasons are offering OWA on the internet, obviously there’s a deeper problem.
- xbar 6y ago
  My Exchange server avoided this mess by living in this state.
bjarneh 6y ago
> As more sites adopt TMTP for their own reasons Isn't this the problem in replacing almost any technology that we know is "broken", it is often too ingrained to be replaced easily.
- networkimprov 6y ago
  Consider the huge variety of messaging and discussion apps; it's relatively easy to embrace new communication tools. EDIT: you might never silence SMTP altogether, but a suitable protocol could supplant it for the great majority of its use cases.

j3th9n 6y ago

Slightly related, on BBC iPlayer is currently an interesting documentary series available called "China: A New World Order", which touches hacks like these a couple of times.

te_chris 6y ago
Can attest to this being very good

waynesoftware 6y ago

"This is the real deal," tweeted Christopher Krebs, the former CISA director. "If your organization runs an OWA server exposed to the internet, assume compromise between 02/26-03/03."

imglorp 6y ago
Chris acknowledged Brian as his "brother from another mother." :-) I was wondering...

mark-r 6y ago

So if you discover one of these hacked servers, how should you let them know - send them an email?

vmception 6y ago
1. Find approximate geographic location (whatismyip.com, traceroute, a couple pings to nearby datacenters) 2. Do a speedtest 3. Add location and speed to remote desktop access marketplaces on darknet 4. Collect passive income from renters looking for clearnet computers in certain areas to use. Often times all the known VPN IP addresses are polluted - even their "dedicated residential IPs" and this can ensure you have worse treatment on the internet, such as more captchas, outright bans, inability to use streaming services, and for actual criminals it means their stolen credit cards don't work. But with remote desktop marketplaces, you can find a computer near the postal code of the credit card you have and this ensures your online transactions go through. Obviously not "you" as you don't have to care what the people do on the other side of your tollbooth. Since you weren't the one compromising anything (computer, credit card, any actual spending) you'll be fine, but you're also going to do all this over Tor anyway. But because you'll be fine you don't have to worry about being detected due to some flaw in Tor because you won't have triggered a criminal investigation, the actual hackers and skimmers and thieves will have though, and incurred all the liability for themselves, people who will have paid an address on a darknet marketplace in Monero and gotten temporary access to a server.
- 55555 6y ago
  This is the answer to a very different question.
  - vmception 6y ago
    haha you're right, I read "what would you do" not "how to contact" but I also figured that the spike in traffic or someone messing up their botnet's activity windows would alert the computer owner to something

lizardmancan 6y ago

Im still of the opinion that closed source should be illegal. How stupid is this idea?

technion 6y ago

We've got some information on the timeline (and a name) on one of the major exploits here: https://proxylogon.com/ https://proxylogon.com/ Some of the detail on where this is a mess - The relevant security update is only offered for the latest (-1) Cumulative Update for Exchange. So you can open Windows Update and it will say "fully updated and secured", but you're not. Complicating matters, Cumulative Updates for Exchange 2019 have to be done from the licensing portal, with a valid logon. So maybe you have a perfectly capable 24x7 tech team, but the guy who manages license acquisition is on leave today. This is how you may basically find yourself resorting to piracy to get this patched.

bombcar 6y ago
Reminiscent of Cisco IOS patches being stuck behind support contracts - and inaccessible to many until they pony up.
- oasisbob 6y ago
  It's been a while since I've had to deal with Cisco IOS, but IIRC they were always good about releasing security fixes to anyone upon a TAC request. For used devices off support contract, security incidents were a great opportunity to get free updates.
  - sneak 6y ago
    IIRC you needed a contract login to open TAC cases. I always just got copies of the .bins from friends who worked at places that had contracts. They didn't gate updates at that time by which model you bought, once you had access you could get firmware for anything Cisco.
    - oasisbob 6y ago
      Phone or email would work too, no contract required. > As a special customer service, and to improve the overall security of the Internet, Cisco may offer customers free software updates to address high-severity security problems. The decision to provide free software updates is made on a case-by-case basis. Refer to the Cisco security publication for details. Free software updates will typically be limited to Critical and High severity Cisco Security Advisories. > If Cisco has offered a free software update to address a specific issue, noncontract customers who are eligible for the update may obtain it by contacting the Cisco TAC using any of the means described in the General Security-Related Queries section of this document. https://tools.cisco.com/security/center/resources/security_vulnerability_policy.html https://tools.cisco.com/security/center/resources/security_v...

a-dub 6y ago

the old wisdom used to be "don't expose microsoft stuff directly to the internet" apparently that's still true?

technion 6y ago
This take was always problematic in my view. It's the reason that I had years of inherited servers with a proxy sitting in front of exchange that consistently broke things everywhere it was done. In practice, this exploit is just another in the line of exchange issues that would get forwarded directly through a proxy to the backend unchallenged. Meanwhile the only place I've heard of pushback against applying this patch, is the "but we have a proxy we're secure" crowd.
- a-dub 6y ago
  that's still a form of exposing microsoft stuff directly to the internet, in my view.
  - ryanlol 6y ago
    Are any other vendors better? I don’t think this is a MS issue.
    - robbyt 6y ago
      I've heard there is a few Redhat servers on the internet
      - ryanlol 6y ago
        Which suffer just as many security issues if you try to do something complex with them.
    - a-dub 6y ago
      much of the software and design for MS stuff is from a period in personal computer history when people weren't worrying as much about public internet style security problems, so it has always seemed to have been at a disadvantage in the internet era which they have fought hard to try and overcome, but nonetheless a lot of code and culture remains. this affects not only the operating system and platform itself, but also major applications, development philosophies, major utilities and even the approaches used to operate it in production. it's actually an interesting question, while internet security problems largely outmoded old pc inspired designs and product-market fit (the diy part time sysadmin), will they outmode the personal operation of any software... that is, will computer security problems grow to the point to where everything must be actively managed and defended?

riffic 6y ago

lol - don't run services you can't competently manage. edit: this tweet restates this in a much nicer way: https://twitter.com/SwiftOnSecurity/status/1366867228914810880 https://twitter.com/SwiftOnSecurity/status/13668672289148108... > If you're not an F50 running your own Exchange Server is organizational clownery at this point.

WheelsAtLarge 6y ago

MS, Solarwinds, ... I suspect that the number of compromised software companies are much larger than these 2 companies. I'm almost certain that we will hear about others in the future. If you manage a software product I hope you are auditing the code regularly. You should also harden the security for it and who has access to the source code and its build no matter how unlikely you think you are a target.

judge2020 6y ago
> I suspect that the number of compromised software companies are much larger than these 2 companies. Given one of the CVEs is CVE-2021-26857, there has already been more that 26000 vulnerability submitted for an CVE ID this year so there are indeed countless other compromised systems - the two big hacks of recent are only in the news thanks to their large blast radius.
xbar 6y ago
Today is the day for all of the other victims to disclose under the umbrella of Microsoft attention.

jariel 6y ago

This needs to be considered an issue of national security and the US forces needs a 'Digital Force' more than they need a 'Space Force'.

yborg 6y ago
https://en.wikipedia.org/wiki/United_States_Cyber_Command https://en.wikipedia.org/wiki/United_States_Cyber_Command
ThinkBeat 6y ago
As another replier mentioned, the US has that and has had that in DOD as well and have had it for longer. However, NSA, has been around a long time. Dont forget about them. If by "applying digital force", do you mean attacking other countries? If so, does the same apply when the US destroys computer systems in other countries? If your country is attacked by the US what force are they reasonably allowed to use as a counter attack? You ruin one or more nuclear weapons facilities; they get to destroy a few nuclear installations of several types in the US? The US is not sitting around being the innocent victim. The US is engaged in offensive attacks on regular basis. (At the same time the US is engaged in massive real world war as well. Unike most of its counterparts. Oh "military conflicts" not war. Unless you are the country at the receiving end of "military conflict" in which case you will have to spend a lot of time trying to figure out why it is not war.
- xbar 6y ago
  I see several straw men and an anti-war argument. None has the mission of broad national defense of civilian assets from cyber warfare by foreign nation states.
- 1MachineElf 6y ago
  Your lack of close parenthesis might have been a typo, but I think it does well to symbolize that the point you're making is less an ancillary explanation and more a real world issue that we can't just ignore.

_robbywashere 6y ago

The United States Government should actively be trying to protect its businesses. They should create a three letter organization to do so. They should call it the National Security something or another.

labster 6y ago
That name is already taken by the department of hodling cyberattacks. They should have a National Vulnerability Agency that handles it.
- jjeaff 6y ago
  Or maybe let's revisit the charter of the NSA and make some major tweaks.
  - gogopuppygogo 6y ago
    Step 1.) Let marijuana users be employed so you can attract talent. Step 2.) Pay above market rate for talent, even import it from Israel or other friendly nation states. We need a Wernher von Braun style approach to recruitment. Step 3.) ???? Profit ????
    - astrea 6y ago
      Step 1 sounds personal
      - jcims 6y ago
        It sounds like it, and it might be, but testing for weed really does impact recruiting. I know of a very large US firm that has quietly stopped including it in their drug test over the past 3-4 years in part because of that. (I wouldn’t be surprised if folks start to push testing as a HIPAA issue.)
        Godel_unicode 6y ago

jtdev 6y ago

How does Microsoft bear no financial liability for the many major security flaws in their for profit software? I’m sure they have clauses in their legal agreements, but come on...

roody15 6y ago

The cynic in me thinks it’s not a coincidence that the cloud office 365 was not affected. Almost like a certain company would like to get its customers to migrate AD to Azure and Exchange to full office 365.

Someone1234 6y ago
One uses Outlook Web Access (where the first exploit exists), the other does not. If the hosted version lacks the component that has a security issue, it won't have that issue, it is technically misinformed to conclude anything nefarious.

WarOnPrivacy 6y ago

Exchange Server 2010 patch is here https://www.microsoft.com/en-us/download/details.aspx?id=102774 https://www.microsoft.com/en-us/download/details.aspx?id=102... Description: https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-server-2010-service-pack-3-march-2-2021-kb5000978-894f27bf-281e-44f8-b9ba-dad705534459 https://support.microsoft.com/en-us/topic/description-of-the...

Keverw 6y ago

Scary! my university uses Microsoft for email, but I think they use the cloud hosted version but wonder how much code is shared between the versions. When I added it to the mail app on my iPhone, it mentioned it could wipe my device. Guess that's a default with the implementation but that is a turn off. So I ended up just installing the Outlook app instead since couldn't find imap support. I feel like on desktop, just using the web version or even adding it to my home screen would be another use but partly was hoping to just have all my accounts together.

ncann 6y ago
If you're on Android there's an app to get around that device management requirement when adding an email account.
- senectus1 6y ago
  Can you elaborate?
  - ncann 6y ago
    The app is named "Exchained": https://play.google.com/store/apps/details?id=mobi.biko.exchained&hl=en_CA&gl=US https://play.google.com/store/apps/details?id=mobi.biko.exch...

xyst 6y ago

my hate for anything micro$oft is justified, again

peter_retief 6y ago

Exchange has been a security problem since 1998. Surely there are open source solutions available that have better security? Seems obvious, have I missed something?

867-5309 6y ago

well it is Patch Friday after all

edrxty 6y ago

Bigger picture, what's the endgame here? It seems a lot of institutions handling sensitive work are considering air-gapping some or all of their networks at this point. Maybe that's even what has to happen. Is there a means of fending off these attacks on the political front? If this same level of espionage was happening in person, there would be a kinetic response but it seems everyone is happy to just turn the other cheek. These attacks have a very real impact. Copying others homework is a tried and true way to get a technological edge and in practical terms, it means a lot of research and development money is effectively wasted as it doesn't generate any returns. Mind, I don't think there should be a violent response, but it's odd that even the threat of sanctions isn't made whenever this happens.

dcow 6y ago
I’m curious to hear more about cases of large institutions seriously considering air-gapping. This is the first I’ve got wind of something like that.
- lrem 6y ago
  Yeah, runs contrary to my perception too. Even things that one would reasonably expect to be air-gapped are online these days.
- bob1029 6y ago
  Air-gapped systems really only make sense for the occasional need to access exceptionally sensitive materials. I.e. private keys for root CAs. For most businesses, air-gapping would mean we are back in the 20th century of business with filing cabinets and armies of people pushing paper between 2 rooms.
  - edrxty 6y ago
    It's not actually that bad. There's a lot of defense, security, and highly proprietary development that happens on isolated networks. You have to put significant effort into IT infrastructure but you'll end up with all your stuff hosted internally and most tools support custom package repo mirrors (linux distros, programming languages/build systems, docker). You'll also probably have a second system with internet access at your desk if not nearby for stackoverflow et al. Basically the idea is defense in depth. The valuable stuff (design files, schematics, code, documentation) lives in the air gapped network while communications live inside a VPN and detailed technical discussion is often discouraged.

AndyMcConachie 6y ago

I really wish the reports on hacks could treat attribution more seriously. Everytime a hack like this occurs it gets blamed on 'the Chinese', or 'the Russian', or 'the Iranians', without every showing any evidence to prove this. Attribution on the Internet is hard, like really hard. I want proof. And if you don't have proof, or can't show me the proof, then don't just blame Americas enemies. It's sloppy and dangerous.

amanaplanacanal 6y ago
I would guess that many times the evidence is from NSA signals intelligence and they can’t show their work because it’s classified. We end up just having to take their word for it.
- baobabKoodaa 6y ago
  How convenient that the perpetrators always turn out to be political enemies.
hntrader 6y ago
I agree that they should show proof. But if they won't show proof yet it's nevertheless true and they have strong privately held evidence concluding it (perhaps from the NSA), that doesn't suddenly make it dangerous to blame the actual perpetrator. It's only dangerous if they're doing it incorrectly or presumptively or deceitfully (which you don't know to be the case).

gala8y 6y ago

Article focuses on US, but this is global. > “It’s massive. Absolutely massive,” one former national security official with knowledge of the investigation told WIRED. “We’re talking thousands of servers compromised per hour, globally.”

annoyingnoob 6y ago
It is really an epic hack, historic.
- PenguinCoder 6y ago
  Just one more in the long line of 'historic' recent hacks.

chovybizzass 6y ago

I'm loving this. I hate microsoft. I hope they die and Bill Gates gets convicted of something heinous.

dieortin 6y ago
Bill Gates hasn’t been CEO for a long time now.

jimnotgym 6y ago

There will always be stolen emails. The problem is that the emails are in plain text on the server...

mjthompson 6y ago

I'm curious to know why this did not affect Office 365 / Exchange Online. I used to work for a law firm which ran on-premises Exchange, but had OWA running behind a VPN. I remember finding it extremely inconvenient at the time. But they're the ones laughing now.

upofadown 6y ago
Apparently because they are different code bases. So the answer could be: random chance.

mattowen_uk 6y ago

I patched my Exchange servers the morning this was announced, a few days ago. The patch takes about ten minutes per server, and does not require a reboot. If your server was a client facing one (CAS) users would have seen a brief outage in Outlook connectivity. The patches were single file downloads, one for each version of Exchange, yes you needed to be on the latest Cumulative Update for Exchange, so if you weren't you really have no right running a production mail system...

deleted 6y ago
[deleted]
ocdtrekkie 6y ago
The last few security patches have been available independent of the Cumulative Updates, so it was reasonable to be a few behind. But this one required the latest CU to install. Bear in mind after updating you still need to check if you were already hacked.

ManlyBread 6y ago

One can only ask what's the point of the forced automatic updates when this stuff is still happening at this scale.

ocdtrekkie 6y ago
There's no forced automatic updates of server software. In fact, Exchange CUs can only be installed manually by mounting an ISO file on the server.

ratiolat 6y ago

Wonder what has changed. It was standard practice 15 years ago not to expose Microsoft Exchange (nor any other Microsoft product) directly to internet.

bearbawl 6y ago
It was standard practice 15 years ago to never ever use Microsoft Exchange in the first place. Wonder what has changed.

r1ch 6y ago

What are the chances this was independently discovered and weaponized in the two months after the original report to MS? Can't help but wonder if the security researcher or MSRC were compromised or have a leak.

deleted 6y ago
[deleted]
GraemeMeyer 6y ago
CISA are indicating that the attacks go back to at least September: https://us-cert.cisa.gov/ncas/alerts/aa21-062a https://us-cert.cisa.gov/ncas/alerts/aa21-062a

porsager 6y ago

There's a powershell script to check your server here: https://github.com/cert-lv/exchange_webshell_detection https://github.com/cert-lv/exchange_webshell_detection

grandinj 6y ago

Isn't time to simply pull the plug on any and all internet access from China until they at least pretend to be discrete about their hacking? (In the old days, there was the Usenet Death Penalty for persistently bad actors)

bluedino 6y ago

Couldn’t you put these servers behind something like CloudFlare? Assuming they were knowledgeable of the attack and could block it.

Threeve303 6y ago

This will be the nail in the coffin for on premise email servers. Putting all of your eggs in one basket might be an even worse idea over time.

ocdtrekkie 6y ago
Eh, this could just as well have happened to everyone with 365 as well. On-premise servers allow you to manage the risk at your own level. For instance, you can decide not to expose a service to the Internet that 365 does.

bob1029 6y ago

We are seriously looking at strategies for clean room rebuild of our IT infrastructure, potentially on a recurring basis via automation. Obviously, you cant mitigate 0-day exploits in any situation where reasonable/expected network access is possible. But our concern, despite not being directly impacted by this, is that we may have accumulated malware over the past decade+ that has never been discovered. How many exploits exist in the wild which have never been documented or even noticed? Do we think it's at least one? The thinking we are getting into is - If we nuke-from-orbit and then reseed from trusted backups on a recurring basis, any malware that gets installed via some side-channel would not be able to persist for as long as it traditionally would. Keeping backups pure via deterministic cryptographic schemes is far easier to work with than running 100+ security suites across your IT stack in hopes you find something naughty. It is incredibly hard for malware to hide in a well-normalized SQL database without SP or other programmatic features. What if we built a new IT stack that was designed to be obliterated and reconstructed every 24 hours with latest patch builds each time? Surely many businesses could tolerate 1-2 hours of downtime overnight. It certainly works for the stock market. There really isn't a reason you need to give an attacker a well-managed private island to hide on for 10+ years at a time.

iforgotpassword 6y ago
We use netboot for most desktop computers and servers that are mostly stateless. Any changes are temporary ending up on a dedicated temp partition that gets wiped on boot, or in ram. Rebuilds are mostly automatic. Of course, netboot in itself opens new attack vectors where we're in early stages of exploring different approaches, even the painful secure boot crap. Honestly I think most of the security in our case right now comes from being an obscure in-house solution that you'd need to specifically target. Also, in case you do get pwned, a post mortem becomes mostly impossible since once you reboot a machine, everything is gone except stuff on network shares of course.
mlac 6y ago
I’ve thought a lot about this. I think from a tech standpoint and a security standpoint, my ideal approach would be to rotate out an A team and a B team. Every 2-3 years, the teams switch off. So year 1-3 A team is running the environment. B team is completely rebuilding and re-architecting the organization’s IT. The company is migrated to B teams infrastructure for 3 years. A team gets to re-build while B team is running and the cycle repeats. This has a few advantages, it keeps the org very current with tools and technology, everyone stays sharp on the latest tech, nothing is sacred, and teams get experience across the spectrum of design build implement and run. It also has good Disaster recovery properties if you idle the old environment so that you can fall back if some critical failure occurs in the new environment. This would be expensive, but please poke holes. I like your idea of clean rebuilds and can see a path to it with automation / terraform / cloud resources. And you don’t need the downtime if you stand up the second one in parallel and just fail over. There’s still persistent data that needs to carry through, so you’d need to figure out how to separate your persistent data from the elements that reset.

chovybizzass 6y ago

lots of pentest jobs available https://startworkingremotely.com/jobs?q=pentest https://startworkingremotely.com/jobs?q=pentest