10 ms·
CircleCI security alert: Rotate any secrets stored in CircleCI
- arkadiyt 4y agoGreat reminder for folks to switch any AWS actions you perform from CI/CD to use OIDC role assumption instead of static IAM user credentials. Then even if an attacker stole all your secrets they can't do anything in your AWS account.
- ollien 4y agoCan you elaborate, as someone with little AWS experience? Are OIDC based creds just more scoped? What makes them special?
- woodruffw 4y agoI assume what they meant is having AWS accept short-lived OIDC tokens from Circle's OIDC provider, which in turn would generate them on demand when the CI is actually run. There'd be no secrets at rest and the attack surface would be (in principle) smaller.
- arkadiyt 4y agoTo assume a role with OIDC you'd need to do it from the context of a specific CircleCI job run - getting access to the secrets of a particular CircleCI account alone would not be enough to authenticate to AWS (unlike when you use IAM user credentials). Even if the attacker had access to env vars from running jobs (which includes the signed token needed to do an OIDC role assumption), those tokens have a short expiry time, and even if an attacker stole that token and performed a role assumption then that session can only be valid for a maximum of 12 hours in AWS, and then you know the attacker is out of your account. It just significantly reduces, practically nullifies, anything that an attacker can do in your AWS account.
- rcme 4y agoThis only applies if the stolen credentials can’t create roles and can’t modify existing roles.
- thinkmassive 4y ago...and don't leave a payload behind, to maintain persistent access (unless I'm missing something?)
- Corrado 4y agoThis is a good reminder to always follow least-permission best practices.
- danw1979 4y agoI’d add drift detection on everything IAM / SCP / Org to this list too. A session token with only a few minutes validity can be enough for someone to make their access permanent.
- shitlord 4y agoI recently did this for one of my GitHub repos which runs several test suites (cumulatively taking >1h). If your actions are slow, pay attention to the IAM role session duration. The maximum duration with role chaining is 1 hour. In the end your credentials need to outlive your CI/CD actions.
- ary 4y agoI believe the max duration of an assumed role session is 12 hours, but this can be changed per-role.
- arkadiyt 4y agoFor assuming one role it can be up to 12 hours. If you're doing role chaining like the parent mentioned (where the 1st assumed role then assumes a 2nd role) then the maximum session duration is 1 hour. AWS has this documented here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_te... > Role chaining limits your AWS CLI or AWS API role session to a maximum of one hour.
- deleted 4y ago[deleted]
- bazookaworkable 4y agoThrowaway for reasons: From experience, be careful and ensure you properly scope your OIDC connection. It’s very easy to allow ANY GitHub repo with proper OIDC connection bits (SA email, connector pool, etc) to get an OIDC token, rather than what you expect, whether that’s any repo in your private org or a specific single repository. As always, RTFM
- unfunco 4y agoA bit of a shameless plug for a relevant Terraform module I made (specific to GitHub in this case): https://github.com/unfunco/terraform-aws-oidc-github https://github.com/unfunco/terraform-aws-oidc-github
- varunsharma07 4y agoWhile OIDC is a good option, at StepSecurity, we are building an open-source project that allows using your MFA tokens for deployments in CI/ CD. So far, it is implemented for GitHub Actions - https://github.com/step-security/wait-for-secrets https://github.com/step-security/wait-for-secrets. In this method, you get a link in the build log, click the link, and can enter credentials at run time, which then gets used in the next step in the pipeline for deployment. So there are no persistent secrets stored in the CI/ CD pipeline and no need for managing/ rotating separate deployment credentials.
- woodruffw 4y agoPerhaps just unfortunate timing, but of note: this comes approximately a month after CircleCI reduced their staff by about 17%[1]. [1]: https://circleci.com/blog/ceo-jim-rose-email-to-circleci-employees/ https://circleci.com/blog/ceo-jim-rose-email-to-circleci-emp...
- gurchik 4y agoNot even a month, only 14 days between your linked post (Dec 7) and when secrets might have been leaked ("starting from December 21, 2022 through today, January 4, 2023")
- rektide 4y agoOur hodgepodge of microservices- developed over more than a decade- never got coordinated env variables, so now we've got to go through like ~50 services & libraries, one by one, updating secrets. Yuck. If you do your shit right, you can just dump most of your secrets into some Contexts- containers of env variables- and apply them. Then when this stuff roles around, it's easy to update everything centrally; change the context & everyone sees it. We, alas, can't easily do that, since we have so many differing env var names. New Year, new fun!
- whitexn--g28h 4y agoSwitch to contexts and add the same secret under multiple names
- latchkey 4y agoA decade of chances to fix this?
- Eduard 4y ago> Then when this stuff roles around, it's easy to update everything centrally; change the context & everyone sees it. But one still has to update their credentials on any downstream service, e.g. Third-party API keys. In general, this is highly individual for each service, and can mostly only be doneanially.
- namaria 4y agoSuch is the tragedy of for-profit software engineering. The trade-offs we see today lead to choices that tie our hands when facing trade-offs we didn't foresee. Also why experience comes at such a premium. Seeing further down the line and knowing how to argue about it prevents whole classes of problems.
- theogravity 4y agoDoes this also include deploy SSH keys?
- preetamjinka 4y agoBetter to be cautious and rotate those too, right?
- richbell 4y agoGiven the wording of "any and all secrets" I would not take any chances.
- chrisntb 4y ago"Immediately rotate any and all secrets stored in CircleCI. These may be stored in project environment variables or in contexts." The blog post calls out "environment variables" and "contexts"
- richbell 4y agoEmphasis on may be; not to mention, they are actively investigating the breach and do not have all the information at this time.
- DylanL 4y agoLooks like yes: https://discuss.circleci.com/t/circleci-security-alert-rotate-any-secrets-stored-in-circleci/46479/8 https://discuss.circleci.com/t/circleci-security-alert-rotat...
- theogravity 4y agoThanks. Sigh, I have ~20 OSS repos with them that I'll need to generate new keys for then.
- chrisntb 4y agoWe contacted CircleCI support and they clarified their blog post statement with the following info: "Thank you for contacting CircleCI Support. This does also apply to SSH Keys, as such we do recommend to rotate SSH Keys as well as to take extra caution. If you have any other concerns please reach out."
- chubs 4y agoSomeone please correct me if i'm wrong... but there was a kerfuffle in 2017 about Circle using third-party JS which could be an attack vector: https://news.ycombinator.com/item?id=15442636 https://news.ycombinator.com/item?id=15442636 To give credence to this, a gitlabber spoke up in that thread, said it was a serious thing and they deliberately had no third-party stuff on their site for that reason. And I just logged into Circle today, and use the Safari network inspector to see what JS it loads... and it's still plenty of third party stuff that I can see: * Amplitude * Segment * cci-growth-utils * Statuspage * DataDog * HotJar * Pusher Not sure if this is an issue, but it doesn't make me comfortable.
- herpderperator 4y agoNo email? I found out about this from a random HN post?
- anderber 4y agoI received an email in my spam folder.
- ajorgensen 4y agoI received an email to my inbox but a few of my colleague did not.
- ithkuil 4y agoWe received an email
- imroot 4y agoNobody at my org did, either. Fun night when you need to reroll your credentials...at least it's nice to have a list in the CircleCI UI, but sucks when you need to make sure that you have all of the scopes available to you.
- wlonkly 4y agoBased on who received and didn't receive it at my workplace, we concluded that it was only sent to users who had not unsubscribed from marketing mail.
- herpderperator 4y agoThat would be it. It should have been sent as a forced account-related email. How strange of them to do that.
- nixgeek 4y agoWhat's tricky is this is not the first interesting recent post from Rob, he previously posted on "An Update on CirclCI Reliability" (Dec '22) [1] and "CircleCI remains secure; be vigilant and aware of phishing attempts for your credentials" (Nov '22) [2]. Overall, CircleCI has had a rough run of it lately. [1] https://circleci.com/blog/an-update-on-circleci-reliability/ https://circleci.com/blog/an-update-on-circleci-reliability/ [2] https://circleci.com/blog/circleci-security-update/ https://circleci.com/blog/circleci-security-update/
- nixgeek 4y agoAnother thread which may need merging to this: https://news.ycombinator.com/item?id=34255189 https://news.ycombinator.com/item?id=34255189
- p-e-w 4y agoI want the following option in my account settings for all critical services: [X] In case of a "security incident", lock down my account until I take action. I understand why they can't do that by default, but it's crazy that every time this happens, I have to run in order to secure my assets when in many cases, I'd be perfectly fine with things just shutting down until I have time to take care of them. Better yet, also give me a button that does this even when there's no official incident reported. That means disabling all access tokens, resetting the password, halting any scheduled jobs, and revoking access for any connected OAuth services until I manually re-enable them.
- VWWHFSfQ 4y agoI don't think locking down the account will do anything. It sounds like secrets were already stolen. GitHub access tokens, etc. Locking the account won't unsteal that stuff.
- __MatrixMan__ 4y agoRight. You'd need lock-down-all-AWS-controlled-by-the-foo-key because CircleCI got hacked and it had the foo-key. Sounds like a separate product (something about breaches and blast radii) and not a CircleCI feature.
- p-e-w 4y agoThe product already exists, it's called OAuth. All you need is an additional role that you can authorize: CircleCI would like to: - Upload build artifacts - Report security incidents Then in GitHub (or wherever), you have the aforementioned checkbox. So when CircleCI reports the incident, the GitHub account is locked down.
- namaria 4y agoYou mean hanging whole sections of our value chain on other companies' assets was not the best idea?
- p-e-w 4y ago
- sickmate 4y agohttps://twitter.com/sanitybit/status/1610829345676996609 https://twitter.com/sanitybit/status/1610829345676996609 >I've been investigating the use of a @ThinkstCanary AWS token that was improperly accessed on December 27th and suspected as much.
- ab-dm 4y agoWhy on earth haven't I received an email from Circle about this?? I guess the answer is, why on earth am I still using Circle CI.... Thankfully all of my secrets/env variables are just dummy data for tests, and already using OIDC
- robertoandred 4y agoI have. Maybe PEBKAC
- ryanisnan 4y agoCheck your personal Github account email.
- ab-dm 4y agoyeah I did. None anywhere
- atymic 4y agoHad one legacy app still on CircleCI and figured may as well move it over to GH actions if we're already rotating tokens anyway. Really hard to recommend anything else these days.
- Corrado 4y agoI'm kinda in the same boat. We've just started to experiment with GH Actions and I'm really liking it. This is just going to move the needle faster.
- bamboozled 4y agoPeople on my team are talking about it, I'd say this incident is the end of our trust in Circle CI going forwards. On the other hand, I'm becoming increasingly weary of putting all my eggs in the Microsoft basket if move our source code, build system, dev environments (codespaces) to GitHib, is it just me ?
- jgaa 4y agoI really don't understand why you use someones else's computer to compile and test your stuff. When their computers are compromised, by internal or external crooks, the crooks have full access to your code, and - in some cases - your data. If they wanted, they could inject their own shit into your binaries, totally ruining your reputation. As a bonus, you get to pay a premium! I still compile and test my code on my own machines, in my own network. It's much faster than CircleCI, cheaper, and it's ∞ safer.
- bamboozled 4y agoIt's nice you can do that, it doesn't work for large distributed teams.
- lrvick 4y agoSure it does. Do engineers not compile their code locally constantly as a part of the process of writing it? Store deterministic hashes of expected binaries with signed commits in PRs. Then untrusted CI merely needs to generate and sign -matching- hashes and now we are good as long as the engineer and CI system are not compromised at the same time.
- hacb 4y agoWhat about testing? In my company, before any code goes to production it has to go through hundreds if not thousands of unit tests. This can't be done on a dev laptop (see XKCD #303)
- lrvick 4y agoTesting is a separate concern than supply chain security. Testing should also never require any secrets useful to an adversary, so third party hosted CI is low risk here.
- mnahkies 4y agoYou're talking about creating reproducible builds - which is a good idea, but in most cases you will still need to deliver that binary somewhere. That typically requires authentication, whether you're deploying to kubernetes or copying the files somewhere using scp, etc So either your laptop or the ci system needs some level of secrets present to put the artifact in the correct place
- ryanisnan 4y agoI legitimately don't understand how the ranking on HN works sometimes. How is it that there are older, less-commented posts ranking higher than this story? @dang? edit: I sincerely think this should be bumped, given how many folks don't seem to be getting the news here in a timely fashion.
- bamboozled 4y ago> We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well. Is anyone else a little annoyed by the messaging here, I read it as, "We think something bad happened to your ultra secret data, but we don't know, so we're asking teams to spend potentially hours or days fixing things while we aren't really able to tell you if your stuff was actually compromised"? What I find more troubling is, if they don't quite know what happened, or aren't telling us, and we do the work to change everything, how do they know it won't just happen again in the next day or so and people are still accessing our systems, where is the details? > At this point, we are confident that there are no unauthorized actors active in our systems. Confident isn't really a good enough word to use here in my opinion. We've just blocked Circle CI from all our systems for now until we hear more, likely start to move to another build system. I know accidents happen but this is likely the beginning of the end for our teams relationship with Circle CI. Trust has been broken.
- csomar 4y agoI can see you are bamboozled. But you should have seen the writing on the wall for CircleCI for quite some time now.
- Felipebury 4y ago[flagged]
- mjmasn 4y agoPSA: Seems like deleting deploy keys on the CircleCI end doesn't actually delete them from Github, so you need to do it on both ends.
- shdh 4y agoSubpar product, never enjoyed using it. Constant downtime and incidents.
- throwaway892238 4y ago@dang this is currently #198 off the front page, yet this is basically an emergency (literally every customer's secrets are exposed?)... either circleci has no more customers, or people are very calm about this... we need to rotate: - secrets in context environment variables - secrets in project environment variables - project deploy keys - circleci api tokens then we have to go back and look at all audit logs for... basically everything... and try to find something that looks weird. :/
- bbu 4y agothis is such a clusterfuck... and the circleci api doesn't even allow to automate most of the steps. and the ones that should work, error with "internal server error". of course, support is completely unresponsive
- rupert-m-a 4y agoI've created a tool due to this incident to help you find your secrets in CircleCi. https://github.com/rupert-madden-abbott/circleci-audit https://github.com/rupert-madden-abbott/circleci-audit It can: * List env vars attached to your repos and contexts * List SSH keys attached to your repos * List which repos are configured with Jira (a secret that might need rotating)
- starwatch 4y agoThanks for taking the initiative! Circle CI have also released something similar [0] linked to near the bottom of their blog post[1]. [0]: https://github.com/CircleCI-Public/CircleCI-Env-Inspector https://github.com/CircleCI-Public/CircleCI-Env-Inspector [1]: https://circleci.com/blog/january-4-2023-security-alert/ https://circleci.com/blog/january-4-2023-security-alert/
- benced 4y agoYou have to trust a CI provider almost as much as your production host. Circle has not earned the same trust as organizations like AWS.