8 ms·
What happens when you leak AWS credentials and how AWS minimizes the damage
- tiborhercz 4y agoEver wondered what would happen when you leak AWS credentials to GitHub? I was, so I leaked AWS credentials to GitHub to see what would happen. Both AWS and malicious actors respond very quickly and start taking actions on these leaked credentials.
- deleted 4y ago[deleted]
- pmx 4y agoI can understand how AWS can react quickly using the secrets scanning service but how do malicious actors do it without having access to that service?
- Gordonjcp 4y agoLuck, I guess? Maybe they have a bot watching some "latest thing on github" API, that scans everything coming in for credentials.
- blibble 4y agothere's a public firehose feed of everything on github https://api.github.com/events https://api.github.com/events (not sure why really)
- andy_ppp 4y agoIt says in the docs this API is delayed 5 minutes though. So no idea how they did things this quickly unless you can add a webhook to every repository somehow as it is created? It's probably just polling and luck to be honest.
- chpatrick 4y agoMaybe the 5 minutes delay is so partners like AWS get a chance to scan first. https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning#about-secret-scanning-alerts-for-partners https://docs.github.com/en/code-security/secret-scanning/abo...
- bradreaves2 4y agoReally interesting experiment! To answer "how do the bad guys find them so quickly" the answer is "judicious queries to the GitHub API." My research group describes how this works, and what we could see with it, in this paper: https://bradreaves.net/mmr19.pdf https://bradreaves.net/mmr19.pdf We're currently working on follow up work to make secret management better (with NSF funding[1]), and we're always looking for research collaborators[2]. [1] https://nsf.gov/awardsearch/showAward?AWD_ID=2055554&HistoricalAwards=false https://nsf.gov/awardsearch/showAward?AWD_ID=2055554&Histori... [2] https://bradreaves.net/collaborate/ https://bradreaves.net/collaborate/
- BillinghamJ 4y agoTailing this presumably: https://api.github.com/events https://api.github.com/events
- macintux 4y agoApparently not; there’s reportedly a 5 minute delay on that feed.
- orf 4y agoI set up a project[1] to automatically leak AWS secrets published to the Python package index, which then triggers the secret scanning process and quarantines the keys[2] 1. https://github.com/pypi-data/pypi-aws-secrets https://github.com/pypi-data/pypi-aws-secrets 2. https://github.com/pypi-data/pypi-aws-secrets/blob/main/keys/pypi/GreenMatterAI/GreenMatterAI-0.0.18.tar.gz.md https://github.com/pypi-data/pypi-aws-secrets/blob/main/keys...
- djbusby 4y agoChaotic Good?
- remram 4y agoIt is really silly that the only easy way to report a leaked secret from most services is to push it to GitHub. I don't understand how the industry failed to standardize any secret-reporting scheme.
- orf 4y agoHow would this work in practice? In order to effectively filter for tokens you need a set of pretty specific regular expressions. These specific expressions may contain non-public information about the structure of the tokens. But imagine we live in a world where companies are OK with exposing this information publicly and investing time in keeping the expressions up to date. Now what? How do you report it to services whilst stopping abuse? Like, if the code isn’t public then all this is an endpoint to let anyone to disable another users (or companies) key without any explanation or recourse. Pushing to GitHub is actually quite an elegant solution to these issues I think, even if I do think it is a bit too centralized.
- mcspiff 4y agoI mean yes — that’s the point. If untrusted parties have access to your keys, it’s already game over. You’ve lost. Disabling them is the nicest thing an attacker can do for you.
- pritambarhate 4y agoI think it will be great if AWS allowed us to use the AWS keys to be used from certain IPs only. It will add additional layer of security which can help to prevent a lot of misuse. Just like AWS best practices of putting DB instances in a private VPC can prevent a lot of attack vectors, ability to use AWS Keys only from certain IPs or IP ranges, can also prevent from misuse, even if one accidentally leaks certain keys.
- circular_logic 4y agoThis looks to be possible via this guide https://repost.aws/knowledge-center/iam-restrict-calls-ip-addresses https://repost.aws/knowledge-center/iam-restrict-calls-ip-ad...
- aeyes 4y agoThis example is for a role but for IAM users it works the same way.
- bpodgursky 4y agoI don't think AWS wants to imply in any way that you need to be on a VPN to interact with AWS resources. It's not the direction they are going (Zero trust / RBA https://aws.amazon.com/security/zero-trust/ https://aws.amazon.com/security/zero-trust/).
- Godel_unicode 4y agoZero trust doesn’t mean no VPNs, it means not just VPNs. In other words you can put your control plane behind ACLs (and you should!) but you should still require strong auth from the private network.
- Hikikomori 4y agoIt's already possible, but on a iam action level.
- wmfiv 4y agoSource IP is an IAM global condition so it's available for all requests. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p...
- lordnacho 4y agoHas anyone got a script or git hook that will suggest you not to push anything that looks like a credential? Could be really useful. It would need to identify strings that look like a key/password, and I guess also it would need an override, perhaps a comment like "// Demo password".
- Jeremy1026 4y agoGitHub has some protections that you can enable to help prevent this. https://github.blog/2022-04-04-push-protection-github-advanced-security/ https://github.blog/2022-04-04-push-protection-github-advanc...
- Xylakant 4y agoNote: paid add-on, and I believe it requires an enterprise subscription to start with. There exists some open source tooling in the same space.
- aeyes 4y agoThere are several tools which can be used with pre-commit, for example trufflehog.
- jonfw 4y agoI've used sync-secrets before, which has worked well- https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
- jrochkind1 4y agoI use one a git pre-commit hook that according to my notes i copied from here (at some point in the past, it may have changed since then): https://gist.github.com/czardoz/b8bb58ad10f4063209bd https://gist.github.com/czardoz/b8bb58ad10f4063209bd Then adapted with a better regex for secret keys I got from here: https://github.com/awslabs/git-secrets/blob/80230afa8c8bdeac766a0fece36f95ffaa0be778/git-secrets#L233-L242# https://github.com/awslabs/git-secrets/blob/80230afa8c8bdeac... (I think maybe I was getting false positives before)
- 4y ago
- nikanj 4y agoIt’s interesting how well cryptos have adapted to monetizing stolen aws credentials.
- belter 4y ago"The Anatomy of an AWS Key Leak to a Public Code Repository" (2019) - https://blog.technodrone.cloud/2019/03/the-anatomy-of-aws-key-leak-to-public.html https://blog.technodrone.cloud/2019/03/the-anatomy-of-aws-ke...
- flerchin 4y agoIt's not clear to me that we have proof that it was a malicious actor. Was it perhaps AWS automation testing out the validity of the credentials before applying the leaked credential policy?
- ceejayoz 4y agoThe AWS process is hopefully "does this match a secret we know? invalidate!" and nothing more. Relying on "can this key do x?" would be a problem, because it might be able to do x in the future, and it doesn't appear they're testing all possible capabilities either.
- javajosh 4y ago12:33:12 – Pushed the credentials to GitHub 12:34:19 – The AWSCompromisedKeyQuarantineV2 policy is attached to the IAM user test-user by AWS Ah, so my exploit must detect and execute within about a minute. Seems doable.
- prepend 4y agoThis is the equivalent of the crime movie where they start the heist and then start the stopwatch. “Ok, secret is pushed, we have 77 seconds until Amazon shuts us off. Let’s go!!!!”
- dylan604 4y agoWhere are you getting the additional 10 seconds to pull off this heist?
- prepend 4y agoHa, the magic of my bad arithmetic. It should be 67 seconds.
- rzzzt 4y agoHarold Faltermeyer's tune kicks into high gear: https://www.youtube.com/watch?v=XfYgW3Uj9g4&t=107s https://www.youtube.com/watch?v=XfYgW3Uj9g4&t=107s
- sebzim4500 4y agoDoable for an automated system. Anything that requires human intervention would be way harder.
- yencabulator 4y agoPlus the time it takes for IAM eventual consistency to catch up. These still went through, there was just nothing interesting available: 12:34:32 – Various List and Describe calls are made using the leaked credentials
- LelouBil 4y agoDiscord does a similar thing, if a bot toke is discovered online it automatically gets disabled and you get an email.
- cloudginger 4y agoInteresting blog, thanks for sharing!
- cloudginger 4y agoI really liked this blog, thanks for sharing!
- jrochkind1 4y ago> In the email they also give guidance on how to secure the account. They do this in four steps: 1. Rotate and delete the exposed AWS Access Key... I work at a very small shop (at a non-profit institution). In the past, when we've accidentally leaked a credential, AWS has required us to rotate all of our AWS Access Keys in the account, not just the exposed one. Which I know ideally rotation would be easy/automated and this wouldn't matter, but at the time it was disruptive for us. And a bit frustrating when we had intentionally created separate AWS credentials with least privs needed for the service, etc. But thinking on it, I guess any given credential might give permissions to create other credentials or other devious things, rotating all of them might be necessary to be safe. It's been a couple years since we had a leaked credential now (hooray). I wonder if the policy has changed, or if I misunderstood, or if in fact you still need to rotate all credentials in the account, maybe OP just only had one credential in the account they were intentionally doing insecure things so the compromised key was in fact all keys. (I don't specifically remember the `AWSCompromisedKeyQuarantineV2` policy, but probably am just not remembering it, I had other things to worry about at the times!)
- Hikikomori 4y agoCan also avoid having any static credentials at all if possible.
- GauntletWizard 4y agoThis used to not be possible. AWS only added support for OIDC Credentials in 2014, which is 10 years ago but well after AWS launched. Convenient support in client libraries was only added in 2019[1][2]. [1] https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-f... [2] https://github.com/aws/aws-cli/releases/tag/1.16.232 https://github.com/aws/aws-cli/releases/tag/1.16.232
- nerdjon 4y agoThis is one of those things that I was somewhat aware existed since we had seen various reports of it, but I was never fully aware of the speed (and obviously automated) of it. Is this actually documented by aws anywhere? I assume other cloud providers must be doing something similar.
- bdcravens 4y agoA few years ago this wasn't in place. I goofed and leaked a credential via Github, and woke up to $50k and climbing in AWS fees (Monero miners). AWS worked with us, and once we proved we had adequately locked our account down and added additional fail-safes, they took it off of our bill.
- janikvonrotz 4y agoOpenAi does the same. Can this somehow used to quarantine other credentials? F.e. is it checking the key and the secret? Or is exposing the key sufficient?
- belter 4y ago"What to Do If You Inadvertently Expose an AWS Access Key" - https://aws.amazon.com/blogs/security/what-to-do-if-you-inadvertently-expose-an-aws-access-key/ https://aws.amazon.com/blogs/security/what-to-do-if-you-inad... "What do I do if I notice unauthorized activity in my AWS account?" - https://repost.aws/knowledge-center/potential-account-compromise https://repost.aws/knowledge-center/potential-account-compro...
- mcdwayne 4y agoBasically, rotate as soon as you can* and start looking through your AWS logs and setting to see if any services you don't recognize have been spun up. Is you think you have been attacked or see stuff you did not spin up, contact AWS support ASAP! *Do NOT just revoke keys if it is in a production system where other people are working or are depending on. Talk to your team and figure out what the remediation process is internally and follow that! If you are working by yourself and no one is relying on services this key is associated with, then yeah, just revoke and replace ASAP. Here are a couple blogs that cover what you are looking for as well https://blog.gitguardian.com/leaking-secrets-on-github-what-to-do/ https://blog.gitguardian.com/leaking-secrets-on-github-what-... https://blog.gitguardian.com/thinking-like-a-hacker-aws-keys-in-private-repos/ https://blog.gitguardian.com/thinking-like-a-hacker-aws-keys...
- scarface74 4y agoOB: There is never a reason to have access keys anywhere in your source code or a config file in your source control directory. All of the AWS SDKs will get your credentials from - hardcoded credentials in your code -> don’t do this - environment variables - you get this when running locally from your control tower interface that most companies access via SSO via AD/Okta etc - your global configuration file. When running on AWS, your credentials are retrieved via the attached IAM Role for Lambda, ECS, EC2, etc. I have done one third party integration where I used Azure DevOps to integrate with AWS using the AWS tools for Azure DevOps. Even then you store your credentials in an Azure vault separate from your source code.
- jimt1234 4y agoPlease be aware of leaking AWS credentials in any log output, especially Jenkins console output. That's usually protected by some layer of auth, but if I get some chump's account info, and something is leaking AWS creds to Jenkins console output, it's probably gonna end up with full account take-over.