3 ms·

us-east-2 already exists and wasn’t impacted. And the choice of where to deploy is yours!

fragmede 1y ago
Which is great, except for global services that you don't have control on where to deploy to, that ended up being in us-east-1, resulting in issues, no matter where your EC2 instances happened to be.
- otterley 1y ago
  Like what?
  - fragmede 1y ago
    AWS IAM, AWS Organizations, Amazon Route 53 (DNS); AWS services that rely on Route53 include ELB, API Gateway; Amazon S3 bucket creation, some other calls; sts.amazonaws.com is still us-east-1 for many cases; Amazon CloudFront, AWS WAF (for CloudFront), AWS Shield Advanced all have us-east-1 as their control plane. To be clear, the above list is a control plane depency on us-east-1. During the incident the service itself may have been fine but could not be (re)configured. The big one is really Route 53 though. DNS having issues caused a lot of downstream effects since it's used by ~everything to talk to everything else. Other services include Slack. No, it's not an AWS service, but for companies reliant on it or something like it, it doesn't matter how much you're not in EC2 if an outside service that you rely on went down. Full list: https://www.reddit.com/r/DigitalMarketing/comments/1oc2jtd/aws_outage_impacted_company_list/ https://www.reddit.com/r/DigitalMarketing/comments/1oc2jtd/a...
    - nijave 1y ago
      Which parts of STS? I was thinking that had regional endpoints but had some caveats around what existing clients default to
nijave 1y ago
Our bigger issue is SaaS like CircleCI and Quay also being in us-east-1. We have a working DR strategy (albeit about a 4-6 hour exercise) to switch regions but a lot of 3rd party dependencies are tied to us-east-1 It's a bit of a circular problem since it's usually useful to colocate in the same region for latency or security (privatelink)
- otterley 1y ago
  Insist that your SaaS providers have implemented and tested their regional failover capability!

Which is great, except for global services that you don't have control on where to deploy to, that ended up being in us-east-1, resulting in issues, no matter where your EC2 instances happened to be.

otterley 1y ago
Like what?
- fragmede 1y ago
  AWS IAM, AWS Organizations, Amazon Route 53 (DNS); AWS services that rely on Route53 include ELB, API Gateway; Amazon S3 bucket creation, some other calls; sts.amazonaws.com is still us-east-1 for many cases; Amazon CloudFront, AWS WAF (for CloudFront), AWS Shield Advanced all have us-east-1 as their control plane. To be clear, the above list is a control plane depency on us-east-1. During the incident the service itself may have been fine but could not be (re)configured. The big one is really Route 53 though. DNS having issues caused a lot of downstream effects since it's used by ~everything to talk to everything else. Other services include Slack. No, it's not an AWS service, but for companies reliant on it or something like it, it doesn't matter how much you're not in EC2 if an outside service that you rely on went down. Full list: https://www.reddit.com/r/DigitalMarketing/comments/1oc2jtd/aws_outage_impacted_company_list/ https://www.reddit.com/r/DigitalMarketing/comments/1oc2jtd/a...
  - nijave 1y ago
    Which parts of STS? I was thinking that had regional endpoints but had some caveats around what existing clients default to

nijave 1y ago

Our bigger issue is SaaS like CircleCI and Quay also being in us-east-1. We have a working DR strategy (albeit about a 4-6 hour exercise) to switch regions but a lot of 3rd party dependencies are tied to us-east-1 It's a bit of a circular problem since it's usually useful to colocate in the same region for latency or security (privatelink)

otterley 1y ago
Insist that your SaaS providers have implemented and tested their regional failover capability!