7 ms·
AWS Experiencing Intermittent DNS Resolution Errors
- itamarst 7y agoAnd of course Route 53 still has a green checkmark.
- johanneswu 7y agoNow on https://status.aws.amazon.com/ https://status.aws.amazon.com/ Intermittent DNS Resolution Errors We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution.
- mlyle 7y agoYes, that's the original link. There's still a green checkmark next to route 53, though.
- bowmessage 7y agoLikely because Route53 doesn’t have any degraded service at the moment. Would be more apt for a yellow status next to S3, given context in threads above.
- deleted 7y ago[deleted]
- banana_giraffe 7y agoIt's not just S3. I'm seeing failures of DNS resolution of well known, and very much still working, non-AWS sites from within AWS now. It's really hit or miss, though. Most of the big stuff works.
- turdnagel 7y ago...and even if they changed it - are you going to subscribe to 100+ RSS feeds to find out?
- skyraider 7y agoNothing listed on AWS Personal Health Dashboard related to this incident. Also nothing abnormal listed on the AWS status page under S3, which is seemingly having DNS resolution issues for buckets with the `[bucket_name].s3.amazonaws.com` pattern.
- pgroudas 7y agoWe've observed the same... we started seeing this at ~9am et this morning so this has now been going on for over 8 hours.
- tyingq 7y agoThere's almost a business model for a 3rd party to run a AWS service status dashboard that's more accurate.
- loneranger_11x 7y agoWe are facing this right now for us-east-1 :/ thankfully production is unaffected.
- svacko 7y agoIn our case the issue is affecting DNS resolution of 'only' S3 related hostnames (my-bucket.s3.amazonaws.com)
- smudgymcscmudge 7y agoThis comment from the other thread may be useful in mitigating your issue. https://news.ycombinator.com/item?id=21328285 https://news.ycombinator.com/item?id=21328285
- ric2b 7y agoYouTube is having issues as well (via the Android app), does YouTube run partially on AWS?
- whalesalad 7y agoWell... I mean it’s owned by Google so I want to say that’s a negative ghost rider.
- deanCommie 7y agoWaze uses AWS: https://cloud.google.com/blog/products/gcp/guest-post-multi-cloud-continuous-delivery-using-spinnaker-at-waze https://cloud.google.com/blog/products/gcp/guest-post-multi-... You're probably not wrong, but still.
- moon2 7y agoSpinnaker on EC2... how painful.
- cthalupa 7y agoI'm fairly sure that's where the primary developers and original creators of Spinnaker use it.
- moon2 7y agoIt is, their infrastructure was (or maybe still is) mostly based on EC2. However, I think they kind of wanted to support many cloud providers, so maybe they stopped focusing on it througout the years.
- t3rabytes 7y ago"We want to give you further information regarding the occasional DNS resolution errors. The AWS DNS servers are currently under a DDoS attack. Our DDoS mitigations are absorbing the vast majority of this traffic, but these mitigations are also flagging some legitimate customer queries at this time. We are actively working on additional mitigations, as well as tracking down the source of the attack to shut it down. Amazon S3 customers experiencing impact from this event can update the configuration of their clients accessing S3 to specify the specific region that their bucket is in when making requests to mitigate impact. For example, instead of "mybucket.s3.amazonaws.com" a customer would instead specify "mybucket.s3.us-west-2.amazonaws.com" for their bucket in the US-WEST-2 region. If you are using the AWS SDK, you can specify the region as part of the configuration of the Amazon S3 client to make sure your requests use this region-specific endpoint name. The DNS resolution issues are also intermittently affecting other AWS Service endpoints like ELB, RDS, and EC2 that require public DNS resolution. We are actively working on this issue and will update you as soon as the issue is resolved on our end, however at this moment I won’t be able to provide an ETA. I am keeping this case in Pending Amazon Action, will update you as soon as I get further information on the resolution of this issue." https://www.reddit.com/r/aws/comments/dlnl28/route53_is_failing_to_resolve_s3/f4sj7tk/ https://www.reddit.com/r/aws/comments/dlnl28/route53_is_fail...
- kache_ 7y agoMaybe they should disclose when they are carrying out DDoS mitigation strategies that could have adverse effects on their customers. I can only imagine how many people wasted a more than few hours as they were misled by a green status page.
- th582ujdj 7y agoThe internet is about to halt. SQS & SNS are not resolving from many parts of the world.
- Aperocky 7y agoIt's amazing how much a ddos attack can do. I read a few days back that when attackers had trouble attacking cloudflare they then went for the internet infrastructure (Internet Exchange) itself. In this case attacking a DNS service can block connection to a much larger set of internet.
- gramakri 7y agoWe are facing this right now. Trying to push new pages to Cloudfront.
- th582ujdj 7y agoAWS is not really "Too Big To Fail". It's more like, "Too Big, Will Fail".
- ceejayoz 7y agoThere aren't many technological systems - big or small - that don't ever fail.
- haolez 7y agoI’ve noticed CloudWatch Dashboards malfunctioning today (not showing any data).
- myroon5 7y agohttps://aws.amazon.com/route53/sla/ https://aws.amazon.com/route53/sla/
- buboard 7y agoDont worry, the internet was designed to circumvent nuclear attacks like these
- jniedrauer 7y agoI was getting some weird notifications about "kms: server misbehaving". No production impact so far, fingers crossed.
- kache_ 7y agoIf you got paged and are currently dealing with this, one thing you can do is set better defaults for AWS_REGION. i.e update the configuration of your client that is accessing the AWS resource to specify the resource's region
- abvdasker 7y agoMy company hasn't been able to deploy any updates to our ElasticBeanstalk application for about 7 hours due to this. Luckily there's nothing urgent we need to deploy, but this makes me extremely nervous about using AWS going forward. If we experienced an outage that required a rollback or forward fix we would be totally hosed.
- Aperocky 7y agoIn defense of AWS, the ddos is probably one of the most massive and targets dns servers. I fully expect the dns to be strengthened after this incident. At the same time, using another provider doesn’t resolve the problem of ddos attack, which can happen at any point in a public network, not limited to DNS servers
- journalctl 7y agoAWS’s track record is still good, though. I mean, there will always be problems with cloud technology. But would you rather try to mitigate a DDoS attempt yourself or would you rather Amazon do it? I think giving up on AWS because of this event would be kind of throwing the baby out with the bath water.
- mrkurt 7y agoThe question is really "if you're not on AWS, do you have to mitigate a DDoS"? It's a big target for reasons unrelated to what most people run.
- cthalupa 7y agoIn my experience, you eventually piss SOMEONE off, or a competitor thinks it can drive business to them, etc. You probably won't get a huge one, but even a small DDoS can be difficult to mitigate for a company with limited resources.
- zargath 7y agoMaybe a stupid question, but what to do when eu-central-1.signin.aws.amazon.com is down?
- journalctl 7y agoMake some coffee and do something else until it comes back up.
- Rapzid 7y agoIs Route53 under attack or a customer on Route53? By all appearances the Route53 DDOS mitigation strategy is massive scale and distribution. This includes distributing customers and their NS records across infrastructure AND TLDs. I would have thought a blanket attack against Route53 impractical..
- Aperocky 7y agoNothing is impractical when you have enough tps.
- deleted 7y ago[deleted]
- nathanielkam 7y agoSwitching to googleDNS 8.8.8.8 or 8.8.4.4 bypasses the affected DNS servers and will resolve the simple URLs (no region in url). Still really surprised this isn't making bigger news yet.
- ph4 7y agoThanks, this fixed my issues with my local dev environment.
- herostratus101 7y agoTried that this morning and it solved the issue for me.
- BillNyeTheITguy 7y agoForgive my ignorance, but how does using Google's DNS get around Route53 not being able to resolve URLs? Does the default DNS just target Route53 DNS?
- yclept 7y agoChina
- holdenc 7y agoThis is affecting my websites that use cloudfront, for which Amazon apparently uses their own Route 53 for DNS (nslookup -query=ns cloudfront.net). Since the only way to call remote assets from cloudfront as a website asset is to use: xyz.cloudfront.net or a CNAME such as mysites-cdn.com that maps to xyz.cloudfront.net it seems there is no way to use cloudfront without the Route53 lookup. If route53 is down and that is required to use cloudfront how is this not affecting more people? I have had about dozen customers complain today.
- rudolph9 7y agoAWS has wasted so many hours of my life troubleshooting their “throw shit at the wall and see what sticks” services. I don’t know why developers put up, even push for, their garbage services.
- highprofittrade 7y agoBut it's not very hard to find out the issue is dns
- thrax 7y agoIs this the cache poisoning ddos posted earlier?
- Bob312371 7y agoWhy does the AWS status say the issue is resolved when it obviously isn't? S3 is still down in parts of US