13 ms·
CircleCI says hackers stole encryption keys and customers’ source code
- heartbreak 4y agoThere’s nothing in this article that says customer source code was accessed or stolen. Is that an error with the title?
- stuartd 4y agoAs a CI company, their ‘customer data’ is source code!
- heartbreak 4y agoThe story, and the CircleCI post, say the customer data was environment variables and other configuration data. Not source code.
- throwaway892238 4y agoCircleCI has listed literally every kind of credential used by its users as vulnerable. This includes deploy keys that are used to download source code. Anyone who has access to customer data, and a deploy key, can just check out the source code, instantly. The extent to which CircleCI has gone to eliminate all threats is... scary. They've gotten GitHub to invalidate any GitHub access tokens used by a CircleCI customer. They've gotten AWS to e-mail AWS customers if one of their access keys was stored in CircleCI. It's a complete and total compromise of literally every customer secret in CircleCI. I expect this will be the biggest hack of 2023... and it's still January. So, yeah, I'm pretty sure customer source code is up for grabs.
- taspeotis 4y agoWhat do you store in GitHub repos... > Review GitHub audit log files for unexpected commands such as ... repo.download_zip https://circleci.com/blog/jan-4-2023-incident-report/ https://circleci.com/blog/jan-4-2023-incident-report/
- charcircuit 4y agoThe article is referring to how the attacker could use the stolen github tokens to download someone's source code. The source code isn't coming from CircleCI
- taspeotis 4y ago> On December 29, 2022, we were alerted to suspicious GitHub OAuth activity by one of our customers. This notification kicked off a deeper review by CircleCI’s security team with GitHub. > ... > Review GitHub audit log files for unexpected commands such as ... repo.download_zip I don't know what you'd get from GitHub other than source code but the CircleCI blog post explicitly describes the attacker downloading entire repos as a .zip https://circleci.com/blog/jan-4-2023-incident-report/ https://circleci.com/blog/jan-4-2023-incident-report/
- throwaway892238 4y agoGitHub OAuth access credentials could be used to compromise OAuth-authenticated apps. And GitHub personal access tokens with too broad of permissions could download pretty much anything, including any secrets stored in GitHub Actions. But getting the source code is pretty bad by itself. Not from an intellectual property standpoint, but because I've never seen a company whose developers didn't commit live credentials into their source code.
- alfiedotwtf 4y agoIf they had access to GitHub Actions creds, I'm assuming the attackers could also push releases. I wonder how much malware is now in the wild because of this breach
- sangnoir 4y ago> I've never seen a company whose developers didn't commit live credentials into their source code Said company ought to immediately rotate the credentials and force rewrite the repo history to nuke the commit for good measure.
- 29athrowaway 4y agoIt can do everything the user can do.
- ChrisMarshallNY 4y ago> I've never seen a company whose developers didn't commit live credentials into their source code. I don’t do that, but I’m pretty pathological about stuff like that. I learned it from the company I used to work for, who were paranoid to the point of lunacy, about Chinese hackers (they were breached once, and took the lesson to extremes). I don’t think they are an outlier. I’ll bet lots of companies are just as tinfoil. It’s a downright unbearable development environment, though. I’ve heard banks can be even worse.
- moogly 4y agoHeadline is now changed. > Updated headline to better reflect the customer data that was taken. Probably should be updated on HN too.
- pcblues 4y ago"some" means "all" from Australia's recent intrusions (Optus, Telstra). They can say "some" when the people at the top responsible for reporting only want a sample. It's PR.
- TobyTheDog123 4y agoI honestly can't think of a worse result of a hack of a CI/CD service than source code of companies being stolen. In my mind, this is akin to the Okta breach a while back, a ton of companies being hit hard through no fault of their own all at once. I can appreciate the want to diversify services so that secrets/env are separate from code, but I think I would honestly trust the behemoth that is Github with both. That being said, my company still uses Okta, so freebies and mulligans are certainly still tolerated when it comes to data breaches.
- BarryMilo 4y agoI mean, it was preventable, you just had to not use the massive single point of failure that is a cloud CI app. But everything is "so much easier" and "so much cheaper" with the cloud I guess!
- zdragnar 4y agoHaving experienced cloud CI on several platforms as well as in-house installs of both Travis and Jenkins, I think the ratio of downtime between cloud and local makes the cloud a far, far better pick. That said, I am currently very glad I'm not running anything on circle ci...
- BarryMilo 4y agoHindsight is 20/20 for sure, but for this use case confidentiality seems more important than availability.It is something of an impossible choice, but I've come to think that every service big or small gets hacked sooner or later. I'm thinking being a smaller target is better in that case.
- chii 4y agoBut a smaller target might mean less funds for security, as it's basically a constant overhead.
- 4y ago
- marsupialtail_2 4y agoIf you make everything open source...
- ferminaut 4y agoThere shouldn't be any coming back from this. There are failures on multiple levels here & CircleCI demonstrated no one should keep any sensitive data with them. > Zuber said that while customer data was encrypted, the cybercriminals also obtained the encryption keys able to decrypt customer data. ...what? why did this engineer have access to everything? Does CircleCI know what minimum access policies are for?
- bdcravens 4y agoFrom the outside, it looks like many key infrastructure operations are running things like they are still startups, moving fast and breaking things. No access controls, engineers with machines they can take anywhere and install anything on, with full access to everything. Can't get in the way of moving fast, after all.
- kodah 4y agoDeveloper laptops are the next logical vector from zero trust networks backed by SSO. Reading between the lines it seems like the session was an SSO token. I think you're right about giving too broad of access. A lot of companies I've been at assign you to LDAP groups over time and if you've been there a while you'll belong to a lot of LDAP groups. There is no ritual to minifying access that I've ever been made aware of. We definitely need some developer oriented antivirus software, because moving to a life where every bit of software needs to be approved would be a literal nightmare.
- lemon_bottle 4y agoUnpopular opinion but I think whenever these 'hack' incidents happen, there should be a full disclosure and the company should be made to tell us what exactly was hacked right down to the nuts and blots, sql queries and server processes level. There should be full transparency on this and all the open source eyeballs should be able to study and scrutinize this. If not, anyone will be able to get away with data theft tomorrow saying, "my server got hacked". What will cause them to not do it except some sense of personal ethics which is rapidly degrading these days?
- foota 4y agoWould hardware security keys protect from this? If you already have a session token on a site (and that site doesn't somehow restrict the session token to only being used on the machine that generated it? Which afaik isn't possible) then it's too late, yes?
- T3OU-736 4y ago> Would hardware security keys protect from this? If you already have a session token on a site (and that site doesn't somehow restrict the session token to only being used on the machine that generated it? Which afaik isn't possible) then it's too late, yes? Sort of. There are ways to verify both the device being used to access a resource, and the account. This is a good use for TPM. One of the features is attestation, which can be used to do this sort of a thing. For things like crypto keys, this is why HSM (hardware security module) exist. - they make it very difficult to get access to private key (basically, private key doesn't leave the HSM itself, but other cryptographic artifacts based on that key do).
- ec109685 4y agoWith WebAuthN it should be possible to require that credentials are frequently refreshed, which can only be done on the original device. mTLS has similar properties. You can also require an authentication loop for each major service, so only services the developer is authenticated to would be vulnerable to credential stealing. And for critical infrastructure network security would help (e.g. a vpn). With hardware tokens required to initiate the connection, the attacker wouldn’t be able to spin up their own connection. Finally, access to user secrets shouldn’t be a default level of access for engineers. Instead, that should require a break glass escalation of privileges, reducing the risk of pilfering.
- FreakLegion 4y agoCorrect, the hardware token adds nothing over TOTP in this case. You can use a platform authenticator instead of a roaming authenticator, but that only stops you from authenticating on a different machine; it doesn't stop a stolen session from being used on a different machine. The step-up authentication CircleCI mentions is probably more of a sudo model, with a long-lived baseline level of privilege that can only be elevated for short bursts. This is orthogonal to MFA, but it's at least less of a nuisance with a hardware token than with other options. > and that site doesn't somehow restrict the session token to only being used on the machine that generated it One simple practice for security-critical systems is to bind sessions to the client's IP address during authentication. It's not bulletproof given the assumption of malware an attacker can still tunnel through, but neither is locking the session to the device. You could, but the malware is also on the device and can do anything the user can (this is why it's preferable to test user presence outside the device by e.g. tapping a USB key).
- fexecve 4y agoThe sad part is, the damage this does to companies won't be felt for years (which is how long it'll take someone to take the stolen source code, analyze it, and make a convincingly-distinct clone), so companies will think that nothing came of this, and they'll keep using CircleCI (and other similar platforms which put everyone's eggs in the same basket, how appealing to hackers that must be).
- scary-size 4y agoI suspect the intruders would use the code to find vulnerabilities in other companies. Which they can use for easy ransomware access or straight up stealing customer data. Or both. Much less trouble than analysing and running someone else’s software.
- alfalfasprout 4y agoClone? More like hackers will use this to test vulns offline and make even more elaborate hacks against customers of circleci.
- fexecve 4y agoWeb software is incredibly simple. Having someone's ruby code doesn't really help you much. Hackers just try a few SQL injections, stuff script tags into all of the form fields, and call it a day. More advanced hacks on webapps are virtually unheard of. Certainly custom-designing a hack for some company isn't going to happen. It's a waste of time, because chances are you won't find some complicated hack that wasn't already exposed through the spamming techniques I mentioned above. And most larger, "worthwhile" companies tend to run their own in-house CI and not use Circle.
- tpmx 4y agoIn this incident, the unauthorized actor exfiltrated customer information on December 22, 2022, which included environment variables, keys, and tokens for third-party systems. A lot of production AWS/GCP keys are likely stolen for those that deploy from CI/CD.
- llIIllIIllIIl 4y agoOh man, I feel so lucky that I've switched exclusively to Github actions late 2020, no good news from CircleCI since then.
- 29athrowaway 4y agoWhat if they just soft-deleted your info and you still might be exposed somehow?
- Dunedan 4y agoWhich is in fact what they did (or even worse they didn't delete them at all): https://discuss.circleci.com/t/circleci-security-alert-rotate-any-secrets-stored-in-circleci/46479/172 https://discuss.circleci.com/t/circleci-security-alert-rotat...
- Dowwie 4y agoYou're giving Github way too much unearned credit about its security practices
- paulmendoza 4y agoHopefully this is a wake up call for GitHub
- jay-barronville 4y agoI hate to be that guy but this news highlights some blatantly incompetent security protocols (especially key management) by a company that we should expect better from. Even something as simple as a Vault (HashiCorp) cluster with decentralized key shares would’ve prevented this. I’m really disappointed in CircleCI. There’s no way I’d trust them after this.
- akdor1154 4y agoWould it? A vault token could be stolen in the same way.
- komuW 4y agoNote that dumping the Vault's process memory is beyond hashicorp/Vault's threat model. See: https://github.com/hashicorp/vault/issues/1446#issuecomment-221625921 https://github.com/hashicorp/vault/issues/1446#issuecomment-...
- avereveard 4y agoSo from exploit to reaching the news it took almost a month. That's a large window of opportunity.
- m00x 4y agoCustomers got an email from Circle immediately after they were able to "resolve" the issue. The full post-mortem was released recently. We got the email on Jan 4 or Jan 5.
- avereveard 4y agoSo "only" three week
- Dunedan 4y agoSome customers got such an email, but not all of them (also see https://news.ycombinator.com/item?id=34255721 https://news.ycombinator.com/item?id=34255721). The only email received by people in my org was the email pointing to the post-mortem, which went out on 2022-01-13.
- m00x 4y agoI wonder why they didn't include everyone. Would be nice to know why.
- alfalfasprout 4y agoLaptop security aside (this is a hard problem and good solutions can often be detrimental in other ways) there should have been way, way more auditing around access to customer repos. The fact that it took so long to both mitigate further access and to understand the rough scope of the hack is concerning. More broadly... it shouldn't be that easy to get encryption keys to everyone's secret env variables used for CI jobs.
- blntechie 4y agoThis is a super critical hack, looks really bad for CircleCI. But I don't see any negative news much elsewhere. They will move on. Also, maybe that localised build system running on an old server for each team seem to be not a bad idea to reduce the blast radius when eventually a hack happens. These providers are supposed to be the gatekeepers and experts who one leaves the tedious and critical work to. If they are just being a leaky cauldron, maybe not bad to cook in my old pot at home.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- StopHammoTime 4y agoHow could a stolen session token even be useful. I have to log into tools every day, if I change IPs at all I have to re-authenticate, and all prod access needs to be approved and has a finite lifespan. How could a CI company be that negligent. They should be leading this stuff from a best practice point of view.
- m00x 4y agoThe employee had malware on his computer, and the hackers might've used a VPN that's located in a "safe" zone. Security is pretty lax at most companies, and the more employees you have, the worse it is.
- shamiln 4y agoMost places I’ve worked, has the concepts from above. Security is lax at most companies but even having worked as an engineer on support rotation, customer anc production access was the last thing I could do and I needed to have exhausted all other options. I don’t feel retaining production access to generate tokens should be a thing.
- m00x 4y agoIt depends on the eng. If you're a senior eng with on-call responsibilities, you're getting full time prod access.
- hveragerdi 4y agoNo need for a VPN if you have access to the laptop, you just use that as a proxy. Look at any c2, open source or commercial, and they'll have that as a feature.
- deleted 4y ago[deleted]
- eurasiantiger 4y agoHard agree. A stolen token was still valid a week later! There is no excuse.
- debarshri 4y agoWhat is interesting here is CircleCI is SOC2 Type 2 compliant. The whole narrative changes if CircleCI was only a self hosted solution and the hack would have happened by one of the customer employees. I'm sure no one would have blamed CircleCI. I dont know if this employee had remote access to all the self hosted enterprise customers too, then that's true lapse on CircleCIs part.
- wrldos 4y agoSOC Type 2 compliance only suggests they have a sit in audit once a year. Basically the produced a lot of paperwork and it doesn't mean that they evidenced stuff honestly to the auditors.
- debarshri 4y agoAuditors audit for one year. They make sure all the processes and controls are hit and used as expected. I am wondering why the audit firm is not made accountable when these hacks happen.
- scandox 4y agoCompliance and real security are two different things.
- thraxil 4y agoSOC2 is relatively minimal. You describe your policies and controls, the auditor verifies that they meet some baseline requirements of generic "best practices" appropriate to your risk profile, then they collect data during the observation window to verify that you actually adhere to them. In this case, I imagine that CircleCI's controls involved developer workstations having anti-malware software, logs and audit trails for access to production systems, and some kind of intrusion detection system around those, and some SLAs and policies on how they react to alerts from those systems. It sounds like they had all of those things in place but the malware wasn't detected and their IDS didn't pick up the external access. Unfortunate, but both of those are entirely possible in many environments. SOC2 can't guarantee that your anti-malware systems or IDS are flawless and can't be bypassed by a clever attacker. Otherwise, since they've been able to identify what the attackers gained access to, it sounds like they did have logs and audit trails in place. SOC2 auditors typically only have a limited understanding of security and technology themselves (the higher end ones might have more resources available to them) and are only able to sample a small amount of data during the observation window. If you're trusting your sensitive data and credentials to a 3rd party, you should definitely require that they have SOC2 or better, but you still have to own your own security and consider how you need to protect yourself if/when they are compromised.
- komuW 4y agoFrom the circleCI blogpost[1]: "Our investigation indicates that the malware was able to execute session cookie theft, enabling them to impersonate the targeted employee in a remote location" I haven't seen much discussion on how this specific attacker entrypoint can be mitigated. So I'm going to make a naive attempt in this comment. How about storing the client's IP address in the session cookie. Then whenever the server recieves the cookie, it compares the client's IP address against the one stored in the session cookie. The server denies the login if there's a mismatch. The cookie would of-course have to be signed(hmac etc) so that it is tamper proof. One problem with this is that client IP addresses are easily spoofed[2]. So, instead of storing the client's IP address; how about we instead store the clients' SSL fingerprints[3][4]. I haven't looked much into the literature, but I think those fingerprints are hard to spoof. 1. https://circleci.com/blog/jan-4-2023-incident-report/ https://circleci.com/blog/jan-4-2023-incident-report/ 2. https://adam-p.ca/blog/2022/03/x-forwarded-for/ https://adam-p.ca/blog/2022/03/x-forwarded-for/ 3. https://github.com/salesforce/hassh https://github.com/salesforce/hassh 4. https://github.com/salesforce/ja3 https://github.com/salesforce/ja3
- AYBABTME 4y agoThen the malware will proxy via your machine.
- komuW 4y agoyeah this just kicks the can down the road.
- mschuster91 4y ago> How about storing the client's IP address in the session cookie. Then whenever the server recieves the cookie, it compares the client's IP address against the one stored in the session cookie. The server denies the login if there's a mismatch. That doesn't work in environments with multiple NAT origin IPs in place, or when they're using crap like Netskope/some other "security"/"privacy"/"VPN" software, as IPs tend to randomly change with these. It would generate way too many false-positive reports. > One problem with this is that client IP addresses are easily spoofed[2]. Only if the backend servers are badly set up. For me, I always run haproxy as the frontend and forcibly delete incoming headers (X-Forwarded-*, Forwarded), and as an added precaution the backend software is configured to only trust the haproxy origin IPs - so even in the case an attacker manages somehow to directly access the backend servers directly, they cannot get a spoofed IP past the system. > So, instead of storing the client's IP address; how about we instead store the clients' SSL fingerprints That requires client-side SSL authentication, which is theoretically supported by all major browsers, but very rarely used and the UI support is... clunky at best.
- athul_jayaram 4y agoA hack of a CI/CD (Continuous Integration/Continuous Deployment) service can have a significant impact on the companies that use it. In this scenario, an attacker would gain unauthorized access to the CI/CD service's servers, potentially stealing sensitive information such as source code for various companies' software projects. This type of incident is similar to the Okta breach
- iLoveOncall 4y agoThank you ChatGPT.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- athul_jayaram 4y agoit is like saying thank you google
- jwilk 4y agohttps://archive.today/uvvVx https://archive.today/uvvVx
- oxfordmale 4y agoI worked for an anti-virus company. There are tools that check if your malware can avoid detection by the major virus scanners. As such, the recommendation is never to rely on a virus scanners alone to protect critical assets.
- srazzaque 4y agoWhilst slightly off-topic, curious if they published, or if anyone knows, what OS the compromised employee machine was running?
- aeyes 4y agoFiles and path names listed in the "Malicious files" section in the blog post https://circleci.com/blog/jan-4-2023-incident-report/ https://circleci.com/blog/jan-4-2023-incident-report/ lead me to believe that it is MacOS.
- portoal 4y agoLooks like MacOS Malicious files to search for and remove: /private/tmp/.svx856.log /private/tmp/.ptslog PTX-Player.dmg (SHA256: 8913e38592228adc067d82f66c150d87004ec946e579d4a00c53b61444ff35bf) PTX.app
- hjuutilainen 4y agoIt would be so interesting to get more details about the initial compromise. What was the engineer trying to do that ended up with downloading PTX-Player.dmg and (probably) the PTX.app installed in /Applications? Was it targeted directly at CircleCI or is this some generic info stealer? What AV / endpoint security solution were they using? Did it pass the built-in macOS protections (gatekeeper, xprotect, etc)? Public VirusTotal seems to know nothing about that hash.
- portoal 4y agoWhat operating system is running on that malware-d laptop ? 90% chance it's Windows 10 ?
- portoal 4y agoLooks like MacOS
- c3534l 4y agoSeems like everyone gets hacked eventually. Like, I'm sure CircleCI had security experts they hired. I don't doubt that they took things seriously and made sure they followed best practices. But that's not good enough. You will still get hacked. What do we do about this?
- debarshri 4y agoYou are absolutely right. The sophistication of the hack implies, you will never know how the attackers will get you. What an organisation can do is, firstly do better disclosures. Secondly, let's the end users know what is risk they have when they add keys, credentials etc. To what extent certain systems have compromised and what steps have been take to mitigate this. Thirdly, organisations should run drills, better access management systems, better auditing systems, have an XDR system in place. You will get hacked, security will be compromised, it is the ability to reduce the contagion risk is what the organisation should measure. Protecting their customers. You cannot shutdown access to resources as developer need them for productivity. Secondly, your competitor is most likely going to take risk of hack and move faster in bringing features while you reduce productivity.
- fareesh 4y agoJoke's on them my source code is terrible
- bamboozled 4y agoI think the CTO has played it pretty well, his recent blog post is kind of "transparent" enough to sound like they care, but very quick to rush everyone back to normality with a kind of "nothing to see here" attitude. "Thanks customers for the support" is almost a patronizing thing to say IMO. They should at least offer compensation financially for this and as others have said, his recent update has left more questions unanswered for me. The way I see it, I'm done as a customer, just need the time to migrate away.