26 ms·

Thanksgiving 2023 security incident

643 points by nomaxx117 3y ago

orenlindsey 3y ago
Cloudflare being compromised would be enormous. Something between 5 and 25% of all sites use CF in some fashion. An attacker could literally hold the internet hostage.
nullpointer00 3y ago
I am trying to learn and understand the attack. Can someone please help me brainstorm some possibilities? (please note, I do not intend to question Cloudflare's business practices or security program; my intention is to learn and understand). -- Blog: The threat actor (TA) accessed Okta’s customer support system and viewed files uploaded by Cloudflare (CF) as support cases. Why was the session token part of the support files uploaded to Okta support? Does Okta require it for troubleshooting? TA hijacked a session token of a CF employee from a support ticket. Blog: Using the token extracted from Okta, the TA accessed Cloudflare’s Okta and compromised two separate Cloudflare employee accounts within the Okta platform. How did this happen? Was the stolen token privileged? Also, why only 2 employee accounts? Were these different employees, or was one the same one whose token was compromised? What does Okta employee account compromise mean - did TA reset the password and MFA, and how or was there no MFA? Blog: TA used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code. Did the stolen employee credentials only have access to Atlassian? Blog: TA gained access to a set of credentials Does this mean multiple credentials got uploaded to the Okta support system? Blog: The Okta compromise was in October, but the threat actor only began targeting CF systems using stolen credentials from the Okta compromise in mid-November. Does this mean the compromised token was long-lived? Blog: We failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise. I didn’t get this. Does this mean that over time, CF employees had uploaded support info for 1000s of apps managed by Okta? Which credentials did CF rotate initially after the Okta compromise? Leaked Credentials: 1. Moveworks service token that granted remote access into our Atlassian system. Is this service token a bearer token? And without expiry. Is this like an API key? TA accessed Atlassian Jira and Confluence using the Moveworks service token to authenticate through the gateway. 2. A service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance, So here, the Smartsheet Saas was given access to the on-prem Atlassian Jira instance? What kind of trust is it? Is this as well managed through Okta? And how does support case filing include a service account? Here, does the Service account mean again some kind of hardcoded API key without expiry TA used the Smartsheet service account to gain access to the Atlassian suite. They used Smartsheet credentials to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment. Since the Smartsheet service account had administrative access to Atlassian Jira, the TA was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for the Jira plugin. This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access, the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. The access was denied, and they could not access any global networks. 3. A Bitbucket service account, which was used to access our source code management system 4. AWS environment that had no access to the global network and no customer or sensitive data. Were these AWS access keys? Also, it looks like these keys did provide access to the AWS account. That means the access key didn’t require MFA. The only production system the TA could access using the stolen credentials was our Atlassian environment. Mitigations: Blog: We decided a huge effort was needed to further harden our security protocols to prevent the threat actor from being able to get that foothold had we overlooked something from our log files. What does hardening security protocol mean here? Is it techniques for D&R or something else Blog: We undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials) I believe this means forced resets on employees (Okta users), right?
mmvasq 3y ago
Really good write up
jshier 3y ago
Fascinating and thorough analysis! I guess if you think an account is unused, just delete it!
- phyzome 3y ago
  Probably safer to rotate the credentials and then schedule it for deletion later. Then if you discover it wasn't unused after all, you have an easier recovery... :-)
sebmellen 3y ago
The most surprising part of this is that Cloudflare uses BitBucket.
- deleted 3y ago
  [deleted]
- Cthulhu_ 3y ago
  How so? It integrates well with the other Atlassian products they use.
- infecto 3y ago
  Maybe but maybe not. I don't like Bitbucket but there are a number of large companies where they worry about using services owned by competitors in one of their verticals.
  - kccqzy 3y ago
    Bitbucket doesn't have to be a service. It can be an old-fashioned downloaded software that you install on your own machines. Not everything is SaaS.
    - infecto 3y ago
      Not sure what you mean? If you are alluding to the OP that said it was surprising...I don't think he found it suprising they they use Bitbucket over Mercurial. I think its safe to assume he meant bitbucket over a Github. In the git universe there is a pretty short list of services, locally or hosted that you would probably use as an entity as large as cloud flare.
- toyg 3y ago
  Integrates with Jira and the rest of Atlassian's stuff, and it's just another git server at the end of the day.
- gempir 3y ago
  A lot of very big companies use Bitbucket, it's just a lot more cost effective than Gitlab/Github.
sevg 3y ago
> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian products (Jira, Confluence, and Bitbucket). > The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.
- readyplayernull 3y ago
  > The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. Aha, the old replace-your-trusted-hardware trick.
  - zitterbewegung 3y ago
    Manufacturers have had security vulnerabilities for hardware to the point that the firmware on device couldn’t be trusted to be replaced so they said to get new hardware so it’s not a bad strategy.
  - AzzyHN 3y ago
    In a corporate environment, standard procedure when an employee's computer gets infected is to re-image it. Even if it was a stupid virus that was immediately caught, the potential risk of undetected malware running amuck is just too high. Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.
- syncsynchalt 3y ago
  Honestly I wish we'd had an excuse/reason to do an org-wide prod creds refresh like this at some places I've been. You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.
marcinzm 3y ago
> we were (for the second time) the victim of a compromise of Okta’s systems I'm curious if they're rethinking being on Okta.
- Icathian 3y ago
  The challenge being, who else could possibly handle Cloudflare's requirements? I imagine the next step is to build their own, and that's obviously not an easy pill to swallow.
  - amluto 3y ago
    Why not? Cloudflare already operates a system that can help customers to require SSO for access to their services — why not try to capture more of that vertical by becoming an IdP?
  - whalesalad 3y ago
    They already run their own zero trust infrastructure for customers, kinda surprised they are not dogfooding it. https://www.cloudflare.com/plans/zero-trust-services/ https://www.cloudflare.com/plans/zero-trust-services/
    - mikey_p 3y ago
      Did you read the article? They are using zero trust and explained that it's why the scope of the security incident was extremely limited.
    - jgrahamc 3y ago
      We use our Zero Trust stuff extensively. In fact, we built it for ourselves initially.
    - tomschlick 3y ago
      They are, but they don't have management for user accounts, 2fa, etc. You setup a connection to something like Okta, Google Apps, O365, SAML, etc to be your persistent user db and cloudflare just enforces it. I wouldn't be surprised if they are working on first party IAM user support though.
jrockway 3y ago
Which "nation state" do we think this was?
- meowface 3y ago
  For these kinds of attacks it's nearly always China, Russia, US, or sometimes Iran. 95% chance it's either China or Russia, here.
  - 2OEH8eoCRo0 3y ago
    When has it been the US?
    - AzzyHN 3y ago
      We do a lot of hacking
      - 2OEH8eoCRo0 3y ago
        I'm sure we do. I don't agree that we attack private civilian enterprise.
        vikramkr 3y ago
        If the usa does I don't understand why you expect we'd know about it. Also the us totally does and we do know about it - the nsa buys zero days - it's not exactly a secret lol
        2OEH8eoCRo0 3y ago
        The same way Cloudflare can report on foreign state hackers, other countries can discover and report on ours, no?
        vikramkr 3y ago
        Right- and they probably do, and they don't know it was us just like cloudflare doesn't know what country it was. For all we know that cloudflare attack was the US. I don't know why China/Russia/Iran/nk would be able to carry out the cloudflare attack without cloudflare being able to pin who exactly did it while the US is supposed to be so incompetent that we would be immediately identified and called out?
belltaco 3y ago
Great write up. > Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories > They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages). Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though. > Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin. > This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. Ouch. Full access to a server OS is always scary.
- Aeolun 3y ago
  > Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though. I think my org has on the order of 3 repositories per dev? They seem to have 3200 employees, with what I assume to be a slightly higher rate of devs, so you’d expect around 6-7 thousand? 2M Jira tickets is probably easily achieved if you create tickets using any automated process.
  - trollied 3y ago
    They might create a JIRA ticket for each customer support interaction. Would make sense.
- ummonk 3y ago
  The number of repositories sounds really high. The number of tickets doesn't.
- spenczar5 3y ago
  12k git repos can happen if the team uses github enterprise with forking internally. It can also happen in franken-build systems which encourage decoupling by making separate repos: one repo that defines a service’s API, containing just proto (for example). A second repo that contains generated client code, a third with generated server code, a fourth for implementation, a fifth which supplies integration test harnesses, etc… Sound insane? It is! But its also how an awful lot of stuff worked at AWS, just as an example.
Aeolun 3y ago
Reading this 2 months after the fact feels a bit late, but I guess it’s better for your stock price if these revelations happen with remediation already in hand? Since they didn’t really have reason to believe my data was accessed, maybe that’s ok. I know from firsthand experience how hard rotating all your credentials across the whole org is.
- Cthulhu_ 3y ago
  The final security report was only released yesterday, and the amount of work they did to make sure all of their systems were secure after the incident was A Lot; two months is pretty quick for a project of that scale IMO.
  - Aeolun 3y ago
    Yes, but if after two months they’d found out that customer data had been compromised, that would be a little late for me to do anything about it.
    - nemothekid 3y ago
      What do you expect them to do? It sounds like you are complaining that they weren't able to instantly ascertain if customer data had been compromised.
      - Aeolun 3y ago
        No, I’m saying that if they’d found out after the fact that it had, it could have been bad for me. What I’d expect is a note on the 24th that they’d kicked a threat actor of their Jira system, and that they had no reason to believe the rest of their systems were compromised, but that they were taking action to prevent it from happening again and starting a full investigation. I get that the business might not want to do that if they are not certain there is any cause for alarm. Uncertainty might be even worse for some.
    - eastdakota 3y ago
BytesAndGears 3y ago
Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business. Yes, they aren’t perfect. They do some things that I disagree with. But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this. Thank you for the blog post!
- el-dude-arino 3y ago
  [flagged]
  - _heimdall 3y ago
    What are they now, if not an engineering company?
    - el-dude-arino 3y ago
      [flagged]
      - dang 3y ago
        Their CTO restores vintage laptops, still runs a Minitel, tests prime numbers for fun, programmed a KIM-1 with a hex keypad as a kid, and fixes hard drives with woodworking tools: Restoration of an IBM Thinkpad 701C Butterfly-keyboard laptop - https://news.ycombinator.com/item?id=39128387 https://news.ycombinator.com/item?id=39128387 - Jan 2024 (42 comments) Using my Minitel 1B over the phone network in 2023 - https://news.ycombinator.com/item?id=38291493 https://news.ycombinator.com/item?id=38291493 - Nov 2023 (0 comments) My primality testing code is faster than Sir Roger Penrose's - https://news.ycombinator.com/item?id=38274642 https://news.ycombinator.com/item?id=38274642 - Nov 2023 (25 comments) My 1976 KIM-1 - https://news.ycombinator.com/item?id=38161617 https://news.ycombinator.com/item?id=38161617 - Nov 2023 (31 comments) Retrieving 1TB of data from a faulty drive with the help of woodworking tools - https://news.ycombinator.com/item?id=37160783 https://news.ycombinator.com/item?id=37160783 - Aug 2023 (186 comments) ... and that's just a recent random sample! There's a reason why his site has been posted to Hacker News over 600 times. I don't know if you could pick a worse example.
throwaway67743 3y ago
[flagged]
- tomschlick 3y ago
  Switching an entire orgs authentication to a new provider (or something built in house) isn't something you can (or should) do quickly.
OJFord 3y ago
> The one service token and three accounts were not rotated because mistakenly it was believed they were unused. Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?
- stepupmakeup 3y ago
  Rotating could have been manual and the person in charge wanted to save time. Stress could be a factor too.
- htrp 3y ago
  blameless post mortem most likely Great call out too > Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.
  - OJFord 3y ago
    It can still be blameless though? The 'because' makes it sound like that's a correct reason to leave it; that the only error was thinking they were unused. (i.e. that it's fine to leave them if unused, only a problem if they're used) i.e. instead of 'because they were mistakenly thought to be unused' you can say 'because they were mistakenly thought to be ok to leave as unused' (or something less awkward depending on exactly what the scenario was) and there's no more blame there? And if you really want to emphasise blamelessness you can say how your processes and training failed to sufficiently encourage least privilege, etc.
- phyzome 3y ago
  Betting they have a new item in their compromise runbook. :-)
  - OJFord 3y ago
    No I don't think so, I do think something's just difficult to say because of what they can't say, or they just neglected to say/didn't word it well, or something. i.e. a bug in the writing, not the post mortem itself. Because if you take it exactly as it's written it's just too weird, I'm not a security expert with something to teach Cloudflare about err maybe don't leave secrets lying around that aren't actually needed for anything, that's not news to many people, and they surely have many actual security people for whom that would not even be a fizzbuzz interview question reviewing any kind of secret storage or revocation policy/procedure. And also the mentioned third-party audit.
londons_explore 3y ago
Am I the only one who just sees a totally blank page? Viewing the HTML shows it's got an empty body tag, and a single script in the <head> with a URL of https://static.cloudflareinsights.com/beacon.min.js/v84a3a4012de94ce1a686ba8c167c359c1696973893317 https://static.cloudflareinsights.com/beacon.min.js/v84a3a40...
- chankstein38 3y ago
  No, that's also what I see. I'm not sure why you're getting downvoted. EDIT: re-opened the link a few minutes later and now I see the post
- overstay8930 3y ago
  Happened to me on my iPhone too
TABrazil 3y ago
[flagged]
- eastdakota 3y ago
  I think that’s unlikely. São Paulo was a new, not fully provisioned facility that didn’t have all our security hardening in place. That’s why the threat actor likely targeted it. That it was in Brazil is most likely incidental.
londons_explore 3y ago
So after the Okta incident they rotated the leaked credentials... But I think they should have put honeypots on them, and then waited to see what attackers did. Honeypots discourage the attackers from continuing for fear of being discovered too.
fierro 3y ago
>The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This odd to me - unused credentials should probably be deleted, not rotated.
- pbhjpbhj 3y ago
  This smells weird, surely? I'd be looking at who chose not to rotate those particular credentials. 1: "what are these accounts?" 2: "oh they're unused, they don't even appear in the logs" 1: "we should rotate them" 2: "no, let's keep those rando accounts with the old credentials, the ones we think might be compromised ... y' know, for reasons" ?
  - pphysch 3y ago
    More likely: "no one has any idea what these old credentials do, so let's not touch them and potentially break everything"
    - sodality2 3y ago
      Sounds like the perfect time to revoke the credentials and find out what uses them, so we can find why they weren't registered as credentials in use. Personally I'd rather do that, have a team ready, and break production for x minutes in order to properly register auth keys. I'd definitely consider a "silent" credential - a credential not registered centrally - to be a huge red flag. Either it could get stolen, or break and no one knows how to regenerate it. And it's pretty easy as devs to quickly generate an auth key that ends up being used permanently, without any documentation.
      - pphysch 3y ago
        > Personally I'd rather do that, have a team ready, and break production for x minutes in order to properly register auth keys. Sure, but you aren't going to do all that when your team is juggling N other priorities. At least, it will be very difficult getting mgmt and others on board. Unless it's explicitly in the context of a recent breach.
deleted 3y ago
[deleted]
muzso 3y ago
> The threat actor searched the wiki for things like remote access, secret, client-secret, openconnect, cloudflared, and token. They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages). In Atlassian's Confluence even the built-in Apache Lucene search engine can leak sensitive information and this kind of access (to the info by the attacker) can be very hard to track/identify. They don't have to open a Confluence page if the sensitive information is already shown on the search results page.
kccqzy 3y ago
> Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold. For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have the person send back "information about the architecture, security, and management" of the target company. Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.
- toyg 3y ago
  Not if such citizens are sanctioned. Code Red. Hint hint.
  - elashri 3y ago
    I think this probably was a name after the famous Code Red worm [1], not a reference to China. [1] https://en.wikipedia.org/wiki/Code_Red_(computer_worm) https://en.wikipedia.org/wiki/Code_Red_(computer_worm)
    - curiousgal 3y ago
      It's the tech scene on the Internet, everything is a reference to the CCP! /s
    - duskwuff 3y ago
      Or after the flavor of Mountain Dew which was that worm's namesake. Not all names have to make sense. :)
  - eep_social 3y ago
    > we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”. Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat. The MO screams China to me but I wouldn’t read anything into the name “code red” which would have been selected before they identified the specific threat actor anyway.
htrp 3y ago
>They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023. All threat actor access and connections were terminated on November 24 and CrowdStrike has confirmed that the last evidence of threat activity was on November 24 at 10:44. Okta hitting everywhere
wepple 3y ago
They mention Zero Trust, yet you can gain access to applications with just a single bearer token? Am I missing something here? There’s no machine cert used? AuthN tokens aren’t cryptographically bound? This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”
- _cenw 3y ago
  these were service accounts used by third parties to provide jira integrations, not a user account
  - Bluecobra 3y ago
    If they are using Active Directory, wouldn’t a service account be no different than a regular employee account? Both a Jira service account and the CEO of Cloudflare are still Domain Users in AD. Granted, a service account should be way more locked down and have the least amount of access possible.
    - _cenw 3y ago
      https://developers.cloudflare.com/cloudflare-one/identity/service-tokens/ https://developers.cloudflare.com/cloudflare-one/identity/se...
- Bluecobra 3y ago
  Yeah it seems odd to me that their internal wiki, code repo, and Jira is exposed directly to the internet and arbitrary IPs could connect to it. Atlassian had a rash of vulnerabilities recently, who knows how many undiscovered ones still exist. If they had a VPN in place secured with machine certs, that would be yet another layer for an attacker to defeat.
- prg20 3y ago
  You're not. The article makes no sense. They claim robust security controls but apparely lacked a proper accounting of service accounts with external access, especially with admin access to freakin' Jira.
mmaunder 3y ago
Thing about a data breach is once the data is out there - source code in this case - it’s out there for good and you have absolutely no control over who gets it. You can do as much post incident hardening as you want, and talk about it as much as you want, but the thing you’re trying to protect against, and blogging about how good you’re getting at preventing, has already happened. Can’t unscramble those eggs.
- arp242 3y ago
  The source code next year is not the same as source code this year. The customer data next year is not the same as the customer data this year.
- burnished 3y ago
  Whats your point?
  - mmaunder 3y ago
    That this is messaged and received as a net win. It’s not.
    - malwrar 3y ago
      Are they just supposed to be invincible? Next best thing is an incident response with this level of quality and transparency. Thats definitely a win in my book, I want to know the provider of a core part of my infra is able to competently and maturely respond to a security incident and this post strongly communicates that.
- BandButcher 3y ago
  agreed, to me this is a big deal for CF. especially coupled with confluence documentation which most likely includes future plans and designs, org charts, meeting minutes... you could also find other easter eggs in any legacy code, almost all companies have undocumented backdoors obviously a customer data breach would be worse but this is really no bueno
jbverschoor 3y ago
It's almost valentine
wowmuchhack 3y ago
Such a beautiful report and beautiful ownage. Whenever some shitty Australian telco gets owned, people are angry and call them incompetent and idiots; it's nice to see Cloudflare gets owned in style with much more class and expertise. Like the rest of the HN crowd, this incident has only increased my trust in Cloudflare.
culopatin 3y ago
Im growing more and more annoyed at Cloudflare and their stupid “are you a human” crap.
joshbetz 3y ago
What makes them think it's a nation state?
j-rom 3y ago
> To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. The thoroughness is pretty amazing
alam2000 3y ago
[dead]
lopkeny12ko 3y ago
> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. This seems incredibly wasteful. Replacing an entire datacenter is effectively tossing tens of millions of dollars of compute hardware.
- rjzzleep 3y ago
  It is, but for most of these components there is no other choice since there is no way to guarantee that nothing was changed. lvrick would say that's what why want to attest everything. Anyway, I really hope that the hardware isn't just tossed into the recycling, but provided to schools and other places that could put them to good use.
- perlgeek 3y ago
  The sentence before... > To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. It doesn't say all equipment, and that would have been very helpful. But if it's just two or three access devices sitting on the border, it's not so bad. Also, the manufacturer likely just sold the hardware to a different customer, sounds like it was pretty new and unused anyway. Just flash the firmware and you're good.
this_steve_j 3y ago
This is an excellent report, and congratulations are due to the security teams at CS for a quick detection, response and investigation. It also highlights the need for a faster move in the entire industry away from long-lived service account credentials (access tokens) and toward federated workload identity systems like OpenId connect in the software supply chain. These tokens too often provide elevated privileges in devops tools while bypassing MFA, and in many cases are rotated yearly. Github [1], Gitlab, and AZDO now support OIDC, so update your service connections now! Note: I’m not familiar with this incident and don’t know whether that is precisely what happened here or if OIdC would have prevented the attack. Devsecops and Zero Trust are often-abused buzzwords, but the principles are mature and can significantly reduce blast radius. [1] https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect https://docs.github.com/en/actions/deployment/security-harde...
JoyRivers 3y ago
[dead]
prg20 3y ago
> Then, from November 27, we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”. Why didn't they start this effort BEFORE there was an incident? > we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials Bearer credentials should already be rotated on a regular basis. Why did they wait until an incident to do this? > To ensure these systems are 100% secure Nothing is 100% secure. Not being to see and acknowledge that is a huge red flag. > Nothing was found, but we replaced the hardware anyway. Well that is just plain stupid and wasteful. > We also looked for software packages that hadn’t been updated Why weren't you looking for that prior to the incident? > we were (for the second time) the victim of a compromise of Okta’s systems which resulted in a threat actor gaining access to a set of credentials. And yet they continue using Okta. The jokes just write themselves. > The one service token and three accounts were not rotated because mistakenly it was believed they were unused. Wait, wait, wait. You KNEW the accounts with remote access to your systems were UNUSED and yet they continue to be active? Hahahahaha. > The wiki searches and pages accessed suggest the threat actor was very interested in all aspects of access to our systems: password resets, remote access, configuration, our use of Salt, but they did not target customer data or customer configurations. Totally makes sense, I'm sure the attacker was just a connoisseur of credentials and definitely did not want them to target customer data.
zelon88 3y ago
What I don't understand is how they got access to Jira yet you still insist there was no compromise. The very nature of Jira and Confluence (both terrible products, btw) is to collect documentation. I'm assuming it was an internal Jira/Confluence for engineering teams, but still. There have got to be addresses, passwords, service account info, all kinds of info. If it was a tech support server then it's impossible to assert that you didn't lose customer data. So we have this double standard where you pay for this product that is designed to house your deepest secrets and most cherished organizational information, that's so important to you that you run on premises servers to keep it safe, but it's not important enough to constitute a real "beach". You're lying. Either the server contained junk of no value in which case it wouldn't have existed in the first place, or you actually did lose something of value that you won't identify to us. Nobody sets up on-prem Jira just to leave it empty and never put secrets in it.

643 points by nomaxx117 3y ago

Cloudflare being compromised would be enormous. Something between 5 and 25% of all sites use CF in some fashion. An attacker could literally hold the internet hostage.

nullpointer00 3y ago

I am trying to learn and understand the attack. Can someone please help me brainstorm some possibilities? (please note, I do not intend to question Cloudflare's business practices or security program; my intention is to learn and understand). -- Blog: The threat actor (TA) accessed Okta’s customer support system and viewed files uploaded by Cloudflare (CF) as support cases. Why was the session token part of the support files uploaded to Okta support? Does Okta require it for troubleshooting? TA hijacked a session token of a CF employee from a support ticket. Blog: Using the token extracted from Okta, the TA accessed Cloudflare’s Okta and compromised two separate Cloudflare employee accounts within the Okta platform. How did this happen? Was the stolen token privileged? Also, why only 2 employee accounts? Were these different employees, or was one the same one whose token was compromised? What does Okta employee account compromise mean - did TA reset the password and MFA, and how or was there no MFA? Blog: TA used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code. Did the stolen employee credentials only have access to Atlassian? Blog: TA gained access to a set of credentials Does this mean multiple credentials got uploaded to the Okta support system? Blog: The Okta compromise was in October, but the threat actor only began targeting CF systems using stolen credentials from the Okta compromise in mid-November. Does this mean the compromised token was long-lived? Blog: We failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise. I didn’t get this. Does this mean that over time, CF employees had uploaded support info for 1000s of apps managed by Okta? Which credentials did CF rotate initially after the Okta compromise? Leaked Credentials: 1. Moveworks service token that granted remote access into our Atlassian system. Is this service token a bearer token? And without expiry. Is this like an API key? TA accessed Atlassian Jira and Confluence using the Moveworks service token to authenticate through the gateway. 2. A service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance, So here, the Smartsheet Saas was given access to the on-prem Atlassian Jira instance? What kind of trust is it? Is this as well managed through Okta? And how does support case filing include a service account? Here, does the Service account mean again some kind of hardcoded API key without expiry TA used the Smartsheet service account to gain access to the Atlassian suite. They used Smartsheet credentials to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment. Since the Smartsheet service account had administrative access to Atlassian Jira, the TA was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for the Jira plugin. This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access, the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. The access was denied, and they could not access any global networks. 3. A Bitbucket service account, which was used to access our source code management system 4. AWS environment that had no access to the global network and no customer or sensitive data. Were these AWS access keys? Also, it looks like these keys did provide access to the AWS account. That means the access key didn’t require MFA. The only production system the TA could access using the stolen credentials was our Atlassian environment. Mitigations: Blog: We decided a huge effort was needed to further harden our security protocols to prevent the threat actor from being able to get that foothold had we overlooked something from our log files. What does hardening security protocol mean here? Is it techniques for D&R or something else Blog: We undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials) I believe this means forced resets on employees (Okta users), right?

mmvasq 3y ago

Really good write up

jshier 3y ago

Fascinating and thorough analysis! I guess if you think an account is unused, just delete it!

phyzome 3y ago
Probably safer to rotate the credentials and then schedule it for deletion later. Then if you discover it wasn't unused after all, you have an easier recovery... :-)

sebmellen 3y ago

The most surprising part of this is that Cloudflare uses BitBucket.

deleted 3y ago
[deleted]
Cthulhu_ 3y ago
How so? It integrates well with the other Atlassian products they use.
infecto 3y ago
Maybe but maybe not. I don't like Bitbucket but there are a number of large companies where they worry about using services owned by competitors in one of their verticals.
- kccqzy 3y ago
  Bitbucket doesn't have to be a service. It can be an old-fashioned downloaded software that you install on your own machines. Not everything is SaaS.
  - infecto 3y ago
    Not sure what you mean? If you are alluding to the OP that said it was surprising...I don't think he found it suprising they they use Bitbucket over Mercurial. I think its safe to assume he meant bitbucket over a Github. In the git universe there is a pretty short list of services, locally or hosted that you would probably use as an entity as large as cloud flare.
toyg 3y ago
Integrates with Jira and the rest of Atlassian's stuff, and it's just another git server at the end of the day.
gempir 3y ago
A lot of very big companies use Bitbucket, it's just a lot more cost effective than Gitlab/Github.

sevg 3y ago

> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian products (Jira, Confluence, and Bitbucket). > The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.

readyplayernull 3y ago
> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. Aha, the old replace-your-trusted-hardware trick.
- zitterbewegung 3y ago
  Manufacturers have had security vulnerabilities for hardware to the point that the firmware on device couldn’t be trusted to be replaced so they said to get new hardware so it’s not a bad strategy.
- AzzyHN 3y ago
  In a corporate environment, standard procedure when an employee's computer gets infected is to re-image it. Even if it was a stupid virus that was immediately caught, the potential risk of undetected malware running amuck is just too high. Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.
syncsynchalt 3y ago
Honestly I wish we'd had an excuse/reason to do an org-wide prod creds refresh like this at some places I've been. You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.

marcinzm 3y ago

> we were (for the second time) the victim of a compromise of Okta’s systems I'm curious if they're rethinking being on Okta.

Icathian 3y ago
The challenge being, who else could possibly handle Cloudflare's requirements? I imagine the next step is to build their own, and that's obviously not an easy pill to swallow.
- amluto 3y ago
  Why not? Cloudflare already operates a system that can help customers to require SSO for access to their services — why not try to capture more of that vertical by becoming an IdP?
- whalesalad 3y ago
  They already run their own zero trust infrastructure for customers, kinda surprised they are not dogfooding it. https://www.cloudflare.com/plans/zero-trust-services/ https://www.cloudflare.com/plans/zero-trust-services/
  - mikey_p 3y ago
    Did you read the article? They are using zero trust and explained that it's why the scope of the security incident was extremely limited.
  - jgrahamc 3y ago
    We use our Zero Trust stuff extensively. In fact, we built it for ourselves initially.
  - tomschlick 3y ago
    They are, but they don't have management for user accounts, 2fa, etc. You setup a connection to something like Okta, Google Apps, O365, SAML, etc to be your persistent user db and cloudflare just enforces it. I wouldn't be surprised if they are working on first party IAM user support though.

jrockway 3y ago

Which "nation state" do we think this was?

meowface 3y ago
For these kinds of attacks it's nearly always China, Russia, US, or sometimes Iran. 95% chance it's either China or Russia, here.
- 2OEH8eoCRo0 3y ago
  When has it been the US?
  - AzzyHN 3y ago
    We do a lot of hacking
    - 2OEH8eoCRo0 3y ago
      I'm sure we do. I don't agree that we attack private civilian enterprise.
      - vikramkr 3y ago
        If the usa does I don't understand why you expect we'd know about it. Also the us totally does and we do know about it - the nsa buys zero days - it's not exactly a secret lol
        2OEH8eoCRo0 3y ago
        The same way Cloudflare can report on foreign state hackers, other countries can discover and report on ours, no?
        vikramkr 3y ago
        Right- and they probably do, and they don't know it was us just like cloudflare doesn't know what country it was. For all we know that cloudflare attack was the US. I don't know why China/Russia/Iran/nk would be able to carry out the cloudflare attack without cloudflare being able to pin who exactly did it while the US is supposed to be so incompetent that we would be immediately identified and called out?

belltaco 3y ago

Great write up. > Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories > They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages). Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though. > Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin. > This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. Ouch. Full access to a server OS is always scary.

Aeolun 3y ago
> Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though. I think my org has on the order of 3 repositories per dev? They seem to have 3200 employees, with what I assume to be a slightly higher rate of devs, so you’d expect around 6-7 thousand? 2M Jira tickets is probably easily achieved if you create tickets using any automated process.
- trollied 3y ago
  They might create a JIRA ticket for each customer support interaction. Would make sense.
ummonk 3y ago
The number of repositories sounds really high. The number of tickets doesn't.
spenczar5 3y ago
12k git repos can happen if the team uses github enterprise with forking internally. It can also happen in franken-build systems which encourage decoupling by making separate repos: one repo that defines a service’s API, containing just proto (for example). A second repo that contains generated client code, a third with generated server code, a fourth for implementation, a fifth which supplies integration test harnesses, etc… Sound insane? It is! But its also how an awful lot of stuff worked at AWS, just as an example.

Aeolun 3y ago

Reading this 2 months after the fact feels a bit late, but I guess it’s better for your stock price if these revelations happen with remediation already in hand? Since they didn’t really have reason to believe my data was accessed, maybe that’s ok. I know from firsthand experience how hard rotating all your credentials across the whole org is.

Cthulhu_ 3y ago
The final security report was only released yesterday, and the amount of work they did to make sure all of their systems were secure after the incident was A Lot; two months is pretty quick for a project of that scale IMO.
- Aeolun 3y ago
  Yes, but if after two months they’d found out that customer data had been compromised, that would be a little late for me to do anything about it.
  - nemothekid 3y ago
    What do you expect them to do? It sounds like you are complaining that they weren't able to instantly ascertain if customer data had been compromised.
    - Aeolun 3y ago
      No, I’m saying that if they’d found out after the fact that it had, it could have been bad for me. What I’d expect is a note on the 24th that they’d kicked a threat actor of their Jira system, and that they had no reason to believe the rest of their systems were compromised, but that they were taking action to prevent it from happening again and starting a full investigation. I get that the business might not want to do that if they are not certain there is any cause for alarm. Uncertainty might be even worse for some.
  - eastdakota 3y ago

BytesAndGears 3y ago

Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business. Yes, they aren’t perfect. They do some things that I disagree with. But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this. Thank you for the blog post!

el-dude-arino 3y ago
[flagged]
- _heimdall 3y ago
  What are they now, if not an engineering company?
  - el-dude-arino 3y ago
    [flagged]
    - dang 3y ago
      Their CTO restores vintage laptops, still runs a Minitel, tests prime numbers for fun, programmed a KIM-1 with a hex keypad as a kid, and fixes hard drives with woodworking tools: Restoration of an IBM Thinkpad 701C Butterfly-keyboard laptop - https://news.ycombinator.com/item?id=39128387 https://news.ycombinator.com/item?id=39128387 - Jan 2024 (42 comments) Using my Minitel 1B over the phone network in 2023 - https://news.ycombinator.com/item?id=38291493 https://news.ycombinator.com/item?id=38291493 - Nov 2023 (0 comments) My primality testing code is faster than Sir Roger Penrose's - https://news.ycombinator.com/item?id=38274642 https://news.ycombinator.com/item?id=38274642 - Nov 2023 (25 comments) My 1976 KIM-1 - https://news.ycombinator.com/item?id=38161617 https://news.ycombinator.com/item?id=38161617 - Nov 2023 (31 comments) Retrieving 1TB of data from a faulty drive with the help of woodworking tools - https://news.ycombinator.com/item?id=37160783 https://news.ycombinator.com/item?id=37160783 - Aug 2023 (186 comments) ... and that's just a recent random sample! There's a reason why his site has been posted to Hacker News over 600 times. I don't know if you could pick a worse example.

throwaway67743 3y ago

[flagged]

tomschlick 3y ago
Switching an entire orgs authentication to a new provider (or something built in house) isn't something you can (or should) do quickly.

OJFord 3y ago

> The one service token and three accounts were not rotated because mistakenly it was believed they were unused. Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?

stepupmakeup 3y ago
Rotating could have been manual and the person in charge wanted to save time. Stress could be a factor too.
htrp 3y ago
blameless post mortem most likely Great call out too > Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.
- OJFord 3y ago
  It can still be blameless though? The 'because' makes it sound like that's a correct reason to leave it; that the only error was thinking they were unused. (i.e. that it's fine to leave them if unused, only a problem if they're used) i.e. instead of 'because they were mistakenly thought to be unused' you can say 'because they were mistakenly thought to be ok to leave as unused' (or something less awkward depending on exactly what the scenario was) and there's no more blame there? And if you really want to emphasise blamelessness you can say how your processes and training failed to sufficiently encourage least privilege, etc.
phyzome 3y ago
Betting they have a new item in their compromise runbook. :-)
- OJFord 3y ago
  No I don't think so, I do think something's just difficult to say because of what they can't say, or they just neglected to say/didn't word it well, or something. i.e. a bug in the writing, not the post mortem itself. Because if you take it exactly as it's written it's just too weird, I'm not a security expert with something to teach Cloudflare about err maybe don't leave secrets lying around that aren't actually needed for anything, that's not news to many people, and they surely have many actual security people for whom that would not even be a fizzbuzz interview question reviewing any kind of secret storage or revocation policy/procedure. And also the mentioned third-party audit.

londons_explore 3y ago

Am I the only one who just sees a totally blank page? Viewing the HTML shows it's got an empty body tag, and a single script in the <head> with a URL of https://static.cloudflareinsights.com/beacon.min.js/v84a3a4012de94ce1a686ba8c167c359c1696973893317 https://static.cloudflareinsights.com/beacon.min.js/v84a3a40...

chankstein38 3y ago
No, that's also what I see. I'm not sure why you're getting downvoted. EDIT: re-opened the link a few minutes later and now I see the post
overstay8930 3y ago
Happened to me on my iPhone too

TABrazil 3y ago

[flagged]

eastdakota 3y ago
I think that’s unlikely. São Paulo was a new, not fully provisioned facility that didn’t have all our security hardening in place. That’s why the threat actor likely targeted it. That it was in Brazil is most likely incidental.

londons_explore 3y ago

So after the Okta incident they rotated the leaked credentials... But I think they should have put honeypots on them, and then waited to see what attackers did. Honeypots discourage the attackers from continuing for fear of being discovered too.

fierro 3y ago

>The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This odd to me - unused credentials should probably be deleted, not rotated.

pbhjpbhj 3y ago
This smells weird, surely? I'd be looking at who chose not to rotate those particular credentials. 1: "what are these accounts?" 2: "oh they're unused, they don't even appear in the logs" 1: "we should rotate them" 2: "no, let's keep those rando accounts with the old credentials, the ones we think might be compromised ... y' know, for reasons" ?
- pphysch 3y ago
  More likely: "no one has any idea what these old credentials do, so let's not touch them and potentially break everything"
  - sodality2 3y ago
    Sounds like the perfect time to revoke the credentials and find out what uses them, so we can find why they weren't registered as credentials in use. Personally I'd rather do that, have a team ready, and break production for x minutes in order to properly register auth keys. I'd definitely consider a "silent" credential - a credential not registered centrally - to be a huge red flag. Either it could get stolen, or break and no one knows how to regenerate it. And it's pretty easy as devs to quickly generate an auth key that ends up being used permanently, without any documentation.
    - pphysch 3y ago
      > Personally I'd rather do that, have a team ready, and break production for x minutes in order to properly register auth keys. Sure, but you aren't going to do all that when your team is juggling N other priorities. At least, it will be very difficult getting mgmt and others on board. Unless it's explicitly in the context of a recent breach.

deleted 3y ago

[deleted]

muzso 3y ago

> The threat actor searched the wiki for things like remote access, secret, client-secret, openconnect, cloudflared, and token. They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages). In Atlassian's Confluence even the built-in Apache Lucene search engine can leak sensitive information and this kind of access (to the info by the attacker) can be very hard to track/identify. They don't have to open a Confluence page if the sensitive information is already shown on the search results page.

kccqzy 3y ago

> Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold. For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have the person send back "information about the architecture, security, and management" of the target company. Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.

toyg 3y ago
Not if such citizens are sanctioned. Code Red. Hint hint.
- elashri 3y ago
  I think this probably was a name after the famous Code Red worm [1], not a reference to China. [1] https://en.wikipedia.org/wiki/Code_Red_(computer_worm) https://en.wikipedia.org/wiki/Code_Red_(computer_worm)
  - curiousgal 3y ago
    It's the tech scene on the Internet, everything is a reference to the CCP! /s
  - duskwuff 3y ago
    Or after the flavor of Mountain Dew which was that worm's namesake. Not all names have to make sense. :)
- eep_social 3y ago
  > we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”. Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat. The MO screams China to me but I wouldn’t read anything into the name “code red” which would have been selected before they identified the specific threat actor anyway.

htrp 3y ago

>They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023. All threat actor access and connections were terminated on November 24 and CrowdStrike has confirmed that the last evidence of threat activity was on November 24 at 10:44. Okta hitting everywhere

wepple 3y ago

They mention Zero Trust, yet you can gain access to applications with just a single bearer token? Am I missing something here? There’s no machine cert used? AuthN tokens aren’t cryptographically bound? This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”

_cenw 3y ago
these were service accounts used by third parties to provide jira integrations, not a user account
- Bluecobra 3y ago
  If they are using Active Directory, wouldn’t a service account be no different than a regular employee account? Both a Jira service account and the CEO of Cloudflare are still Domain Users in AD. Granted, a service account should be way more locked down and have the least amount of access possible.
  - _cenw 3y ago
    https://developers.cloudflare.com/cloudflare-one/identity/service-tokens/ https://developers.cloudflare.com/cloudflare-one/identity/se...
Bluecobra 3y ago
Yeah it seems odd to me that their internal wiki, code repo, and Jira is exposed directly to the internet and arbitrary IPs could connect to it. Atlassian had a rash of vulnerabilities recently, who knows how many undiscovered ones still exist. If they had a VPN in place secured with machine certs, that would be yet another layer for an attacker to defeat.
prg20 3y ago
You're not. The article makes no sense. They claim robust security controls but apparely lacked a proper accounting of service accounts with external access, especially with admin access to freakin' Jira.

mmaunder 3y ago

Thing about a data breach is once the data is out there - source code in this case - it’s out there for good and you have absolutely no control over who gets it. You can do as much post incident hardening as you want, and talk about it as much as you want, but the thing you’re trying to protect against, and blogging about how good you’re getting at preventing, has already happened. Can’t unscramble those eggs.

arp242 3y ago
The source code next year is not the same as source code this year. The customer data next year is not the same as the customer data this year.
burnished 3y ago
Whats your point?
- mmaunder 3y ago
  That this is messaged and received as a net win. It’s not.
  - malwrar 3y ago
    Are they just supposed to be invincible? Next best thing is an incident response with this level of quality and transparency. Thats definitely a win in my book, I want to know the provider of a core part of my infra is able to competently and maturely respond to a security incident and this post strongly communicates that.
BandButcher 3y ago
agreed, to me this is a big deal for CF. especially coupled with confluence documentation which most likely includes future plans and designs, org charts, meeting minutes... you could also find other easter eggs in any legacy code, almost all companies have undocumented backdoors obviously a customer data breach would be worse but this is really no bueno

jbverschoor 3y ago

It's almost valentine

wowmuchhack 3y ago

Such a beautiful report and beautiful ownage. Whenever some shitty Australian telco gets owned, people are angry and call them incompetent and idiots; it's nice to see Cloudflare gets owned in style with much more class and expertise. Like the rest of the HN crowd, this incident has only increased my trust in Cloudflare.

culopatin 3y ago

Im growing more and more annoyed at Cloudflare and their stupid “are you a human” crap.

joshbetz 3y ago

What makes them think it's a nation state?

j-rom 3y ago

> To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. The thoroughness is pretty amazing

alam2000 3y ago

[dead]

lopkeny12ko 3y ago

> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. This seems incredibly wasteful. Replacing an entire datacenter is effectively tossing tens of millions of dollars of compute hardware.

rjzzleep 3y ago
It is, but for most of these components there is no other choice since there is no way to guarantee that nothing was changed. lvrick would say that's what why want to attest everything. Anyway, I really hope that the hardware isn't just tossed into the recycling, but provided to schools and other places that could put them to good use.
perlgeek 3y ago
The sentence before... > To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. It doesn't say all equipment, and that would have been very helpful. But if it's just two or three access devices sitting on the border, it's not so bad. Also, the manufacturer likely just sold the hardware to a different customer, sounds like it was pretty new and unused anyway. Just flash the firmware and you're good.

this_steve_j 3y ago

This is an excellent report, and congratulations are due to the security teams at CS for a quick detection, response and investigation. It also highlights the need for a faster move in the entire industry away from long-lived service account credentials (access tokens) and toward federated workload identity systems like OpenId connect in the software supply chain. These tokens too often provide elevated privileges in devops tools while bypassing MFA, and in many cases are rotated yearly. Github [1], Gitlab, and AZDO now support OIDC, so update your service connections now! Note: I’m not familiar with this incident and don’t know whether that is precisely what happened here or if OIdC would have prevented the attack. Devsecops and Zero Trust are often-abused buzzwords, but the principles are mature and can significantly reduce blast radius. [1] https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect https://docs.github.com/en/actions/deployment/security-harde...

JoyRivers 3y ago

[dead]

prg20 3y ago

> Then, from November 27, we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”. Why didn't they start this effort BEFORE there was an incident? > we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials Bearer credentials should already be rotated on a regular basis. Why did they wait until an incident to do this? > To ensure these systems are 100% secure Nothing is 100% secure. Not being to see and acknowledge that is a huge red flag. > Nothing was found, but we replaced the hardware anyway. Well that is just plain stupid and wasteful. > We also looked for software packages that hadn’t been updated Why weren't you looking for that prior to the incident? > we were (for the second time) the victim of a compromise of Okta’s systems which resulted in a threat actor gaining access to a set of credentials. And yet they continue using Okta. The jokes just write themselves. > The one service token and three accounts were not rotated because mistakenly it was believed they were unused. Wait, wait, wait. You KNEW the accounts with remote access to your systems were UNUSED and yet they continue to be active? Hahahahaha. > The wiki searches and pages accessed suggest the threat actor was very interested in all aspects of access to our systems: password resets, remote access, configuration, our use of Salt, but they did not target customer data or customer configurations. Totally makes sense, I'm sure the attacker was just a connoisseur of credentials and definitely did not want them to target customer data.

zelon88 3y ago

What I don't understand is how they got access to Jira yet you still insist there was no compromise. The very nature of Jira and Confluence (both terrible products, btw) is to collect documentation. I'm assuming it was an internal Jira/Confluence for engineering teams, but still. There have got to be addresses, passwords, service account info, all kinds of info. If it was a tech support server then it's impossible to assert that you didn't lose customer data. So we have this double standard where you pay for this product that is designed to house your deepest secrets and most cherished organizational information, that's so important to you that you run on premises servers to keep it safe, but it's not important enough to constitute a real "beach". You're lying. Either the server contained junk of no value in which case it wouldn't have existed in the first place, or you actually did lose something of value that you won't identify to us. Nobody sets up on-prem Jira just to leave it empty and never put secrets in it.