9 ms·
Ask HN: Our AWS account got compromised after their outage
Could there be any link between the two events?
Here is what happened:
Some 600 instances were spawned within 3 hours before AWS flagged it off and sent us a health event. There were numerous domains verified and we could see SES quota increase request was made.
We are still investigating the vulnerability at our end. our initial suspect list has 2 suspects. api key or console access where MFA wasn’t enabled.
- bdcravens 1y agoAny chance you did something crazy while troubleshooting downtime (before you knew it was an AWS issue)? I've had to deal with a similar situation, and in my case, I was lazy and pushed a key to a public repo. (Not saying you are, just saying in my case it was a leaked API key)
- klysm 1y agoSounds like a coincidence to me
- yfiapo 1y agoHighly likely to be coincidence. Typically an exposed access key. Exposed password for non-MFA protected console access happens but is less common.
- ThreatSystems 1y agoCloudtrail events should be able to demonstrate WHAT created the EC2s. Off the top of my head I think it's the runinstance event.
- sylens 1y agoRunInstances
- ThreatSystems 1y agoI'm officially off of AWS so don't have any consoles to check against, but back on a laptop. Based on docs and some of the concerns about this happening to someone else, I would probably start with the following: 1. Check who/what created those EC2s[0] using the console to query: eventSource:ec2.amazonaws.com eventName:RunInstances 2. Based on the userIdentity field, query the following actions. 3. Check if someone manually logged into Console (identity dependent) [1]: eventSource:signin.amazonaws.com userIdentity.type:[Root/IAMUser/AssumedRole/FederatedUser/AWSLambda] eventName:ConsoleLogin 4. Check if someone authenticated against Security Token Service (STS) [2]: eventSource:sts.amazonaws.com eventName:GetSessionToken 5. Check if someone used a valid STS Session to AssumeRole: eventSource:sts.amazonaws.com eventName:AssumeRole userIdentity.arn (or other identifier) 6. Check for any new IAM Roles/Accounts made for persistence: eventSource:iam.amazonaws.com (eventName:CreateUser OR eventName:DeleteUser) 7. Check if any already vulnerable IAM Roles/Accounts modified to be more permissive [3]: eventSource:iam.amazonaws.com (eventName:CreateRole OR eventName:DeleteRole OR eventName:AttachRolePolicy OR eventName:DetachRolePolicy) 8. Check for any access keys made [4][5]: eventSource:iam.amazonaws.com (eventName:CreateAccessKey OR eventName:DeleteAccessKey) 9. Check if any production / persistent EC2s have had their IAMInstanceProfile changed, to allow for a backdoor using EC2 permissions from a webshell/backdoor they could have placed on your public facing infra. [6] etc. etc. But if you have had a compromise based on initial investigations, probably worth while getting professional support to do a thorough audit of your environment. [0] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html#cloudtrail-log-file-examples-ec2 https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c... [1] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-aws-console-sign-in-events.html https://docs.aws.amazon.com/awscloudtrail/latest/userguide/c... [2] https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-... [3] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/security_iam_id-based-policy-examples.html https://docs.aws.amazon.com/awscloudtrail/latest/userguide/s... [4] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... [5] https://research.splunk.com/sources/0460f7da-3254-4d90-b8c0-2ca657d0cea0/ https://research.splunk.com/sources/0460f7da-3254-4d90-b8c0-... [6] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ReplaceIamInstanceProfileAssociation.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_R...
- jinen83 1y agothis is helpful. i will look for the logs. Also some more observations below: 1) some 20 organisations were created within our Root all with email id with same domain (co.jp) 2) attacker had created multiple fargate templates 3) they created resources in 16-17 AWS regions 4) they requested to raise SES,WS Fargate Resource Rate Quota Change was requested, sage maker Notebook maintenance - we have no need of using these instances (recd an email from aws for all of this) 5) in some of the emails i started seeing a new name added (random name @outlook.com)
- ThreatSystems 1y agoIt does sound like you've been compromised by an outfit that has got automation to run these types of activities across compromised accounts. A Reddit post[0] from 3 years ago seems to indicate similar activities. Do what you can to triage and see what's happened. But I would strongly recommend getting a professional outfit in ASAP to remediate (if you have insurance notify them of the incident as well - as often they'll be able to offer services to support in remediating), as well as, notify AWS that an incident has occurred. [0] https://www.reddit.com/r/aws/comments/119admy/300k_bill_after_aws_account_hacked/ https://www.reddit.com/r/aws/comments/119admy/300k_bill_afte...
- sousastep 1y agocouple folks on reddit said while they were refreshing during the outage, they were briefly logged in as a whole different user
- afandian 1y agoGot references? This is crazy.
- deleted 1y ago[deleted]
- blast 1y agoI saw a link to https://old.reddit.com/r/webdev/comments/1obtbmg/aws_site_returned_wrong_users_session_token/ https://old.reddit.com/r/webdev/comments/1obtbmg/aws_site_re... at one point but then it was deleted
- duk3luk3 1y agoThis isn't about an aws account, this is about the auth inside the project that user is running.
- perpil 1y agoThis is not about the AWS Console. It is talking about the customer's site hosted on CloudFront. It is possible to cross wires with user sessions when using CloudFront if you haven't set caching granular enough to be specific to an end user. This scenario is customer error, not AWS.
- fulafel 1y agoI'd argue it's a classic footgun and a flaw of CloudFront (they should at least warn about it much more).
- CodesInChaos 1y agoelectricity_is_life's comment on reddit seems to explain it: > Not sure if this is what happened to you, but one thing I ran into a while back is that even if you return Cache-Control: no-store it's still possible for a response to be reused by CloudFront. This is because of something called a "collapse hit" where two requests that occur at the same time and are identical (according to your cache key) get merged together into a single origin request. CloudFront isn't "storing" anything, but the effect is still that a user gets a copy of a response that was already returned to a different user. > https://stackoverflow.com/a/69455222 https://stackoverflow.com/a/69455222 > If your app authenticates based on cookies or some other header, and that header isn't part of the cache key, it's possible for one user to get a response intended for a different user. To fix it you have to make sure any headers that affect the server response are in the cache key, even if the server always returns no-store. --- Though the AWS docs seem to imply that no-store is effective: > If you want to prevent request collapsing for specific objects, you can set the minimum TTL for the cache behavior to 0 and configure the origin to send Cache-Control: private, Cache-Control: no-store, Cache-Control: no-cache, Cache-Control: max-age=0, or Cache-Control: s-maxage=0. https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-traffic-spikes https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
- itsnowandnever 1y agoi cant imagine it's related. if it is related, hello Bloomberg News or whoever will be reading this thread because that would be a catastrophic breach of customer trust that would likely never fully return
- jddj 1y agoYou say that, but azure and okta have had a handful of these and life over there has more or less gone on. Inertia is a hell of a drug
- testfrequency 1y agoSimilarly, everyone is back to using CS and their stock is just fine
- timdev2 1y agoI would normally say that "That must be a coincidence", but I had a client account compromise as well. And it was very strange: Client was a small org, and two very old IAM accounts had suddenly had recent (yesterday) console log ins and password changes. I'm investigating the extent of the compromise, but so far it seems all they did was open a ticket to turn on SES production access and increase the daily email limit to 50k. These were basically dormant IAM users from more than 5 years ago, and it's certainly odd timing that they'd suddenly pop on this particular day.
- tcdent 1y agoSmells like a phishing attack to me. Receive an email that says AWS is experiencing an outage. Log into your console to view the status, authenticate through a malicious wrapper, and compromise your account security.
- SoftTalker 1y agoGood point. Phishers would certainly take advantage of a widely reported outage to send emails related to "recovering your services." Even cautious people are more vulnerable to phishing when the message aligns with their expectations and they are under pressure because services are down. Always, always log in through bookmarked links or typing them manually. Never use a link in an email unless it's in direct response to something you initiated and even then examine it carefully.
- roblabla 1y agoYou can also use phishing-resistant login/2FA like passkeys/FIDO keys, where it is available (and I'm pretty sure amazon supports it), to minimize the risk of accidentally login into a phishing website while under pressure.
- SoftTalker 1y agoThey probably support it but how many accounts have not configured it? I'd bet it's a lot.
- CaptainOfCoit 1y agoIs it possible that people who already managed to get access (that they confirmed) has been waiting for any hiccups in AWS infrastructure in order to hide among the chaos when it happens? So maybe the access token was exposed weeks/months ago, but instead of going ahead directly, idle until there is something big going on. Certainly feels like an strategy I'd explore if I was on that side of the aisle.
- jinen83 1y agoI am from the same team & i can concur with what you are saying. I did see a warning about the same key that was used in todays exploit about 2 years ago from some random person in an email. but there was no exploutation till yesterday.
- LeonardoTolstoy 1y agoThis is it. I had the same thing happen to me a year ago and there was a month between the original access to our system and the attack. And similarly they waited until a perceived lull in what might be org diligence (just prior to thanksgiving) to attack.
- iainctduncan 1y agoAbsolutely. I'm in diligence and we are hearing about attackers even laying the ground work and then waiting for company sales. The sophisticated ones are for sure smart enough to take advantage of this kind of thing and to even be prepping in advance and waiting for golden opportunities.
- shadowpho 1y agoWouldn’t this be a terrible time because everyone is looking/logging into AWS? If my company used AWS I would be hyper aware about anything that it’s doing right now
- LorenPechtel 1y agoI think the idea is that after an outage you would expect unusual patterns and thus not be sensitive to them.
- AtNightWeCode 1y agoNot uncommon that machines get exposed during trouble-shooting. Just look at the Crowdstrike incident just the other year. People enabled RDP on a lot machines to "implement the fix" and now many of these machines are more vulnerable than if if they never installed that garbage security software in the first place.
- geor9e 1y agoIf I was a burgler holding a stolen key to a house, waiting to pick a good day, a city-wide blackout would probably feel like a good day.
- brador 1y agoLot of keys and passwords being panic entered on insecure laptops yesterday. Do not discount the possibility of regular malware.
- tylergetsay 1y agoOr the keys were long compromised and yesterday someone opened permissions on them in order to mitigate
- NedF 1y ago[dead]
- uoflcards22 1y agohttps://www.reddit.com/r/webdev/comments/1obtbmg/aws_site_returned_wrong_users_session_token/ https://www.reddit.com/r/webdev/comments/1obtbmg/aws_site_re...
- deleted 1y ago
- ohdeardear 1y ago[dead]
- kondro 1y agous-east-1 is unimaginably large. The last public info I saw said it had 159 datacenters. I wouldn't be surprised if many millions of accounts are primarily located there. While this could possibly be related to the downtime, I think this is probably an unfortunate case of coincidence.
- Scramblejams 1y ago159! Staggering. Got a source?
- kondro 1y agoSorry, 158: https://baxtel.com/data-center/aws-us-east-n-virginia https://baxtel.com/data-center/aws-us-east-n-virginia
- temptemptemp111 1y ago[dead]
- didip 1y agoDuring time of panic, that’s when people are most vulnerable to phishing attacks. Total password reset and tell your AWS representative. They usually let it slide on good faith.
- unit149 1y ago[dead]
- deleted 1y ago[deleted]
- mr_windfrog 1y agoConsidering AWS’s position as the No.1 cloud provider worldwide, their operational standards are extremely high. If something like this happened right after an outage, coincidence is the most plausible explanation rather than incompetence.
- jmward01 1y agoIf I were an attacker I would choose when to attack and a major disruption happening leaving your logging is in chaos seems like it could be a good time. Is it possible you had been compromised for a while and they took that moment to take advantage of it? Or, similarly, they took that moment to use your resources for a different attack that was spurred by the outage?
- defraudbah 1y agoweird, can you send me your API key so I can verify it's not in the list of compromised credentials?
- darkamaul 1y agoI know this is just a playful joke, but I wanted to gently flag something important. Even in humor, we should never casually discuss sharing API keys or credentials. You never know when or if someone might misinterpret a message like this.
- wiether 1y agoNow that we have people browsing with an "AI browser", it could become quite interesting though
- 1oooqooq 1y agowin-win
- bigDinosaur 1y agoIt's not our responsibility to avoid jokes because some people are awful at their jobs and/or idiots. How on earth would people who would send an API key in response to a joke fare against a genuinely malicious social engineering attempt...?
- nashashmi 1y agoIt is not my job so stuff like this is helpful to know.
- defraudbah 1y agono worries my friend, it's all good, we have a team of professionals to run security checks on your AWS keys. Since many businesses were affected by an awful, irresponsible AWS incident, we understand it might be challenging times for software business, which is why our team runs free security checks for all tokens we receive, limited offer, only today, send us your credentials and get your report in less than 24 hours. we already received more than 100 API keys from people with a referral from hackernews, there are only 50 seats left
- Traubenfuchs 1y agoIt makes me very uncomfortable to know I got my CC in GCP, AWS and oracle cloud and that I have access to 3 corporate AWS accounts with bills on the level of 10's of millions per month. Why don't cloud providers offer IP restrictions? I can only access GitHub from my corporate account if I am in the VPN and it should be like that for every of those services with the capability to destroy lives.
- WesleyJohnson 1y agoOur Alexa had a random person "drop in" yesterday. We could hear a child talking on the other end, but no idea who it was. It may just be a coincidence, but it's never happened before so it's easy to imagine it might be related to the AWS issues.
- mrktf 1y agoMore on technical side I'm interesting what is plausible explanation for this type "glitches"?: it inconsistent backend router state between processing nodes, processing application restart and screw up in shared memory segment (i can imagine to decrease load times - use "persistent" shared memory block for outstanding data), or just plain hash table collision and lack of empty slots (i mean: https://en.wikipedia.org/wiki/Hash_collision https://en.wikipedia.org/wiki/Hash_collision).
- whoknew1122 1y agoThe AWS issue related to DNS entries. And IAM doesn't use Dynamo DB. It wasn't related, other than an outage gives a good way to obfuscate TTPs.
- more_corn 1y agoHow can you not know what credentials were used. A simple cloud trail search on the affected infrastructure will tell you.