4 ms·

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of Ch

• sidewndr46 5mo ago
  Isn't this kind of circular dependency what lead to extended downtime a while back?
  • jethro_tell 5mo ago
    It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.
    • dlenski 5mo ago
      Yes, I concur. Sometimes the circular dependencies get almost cartoonishly silly. Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on an S3 bucket." I made that example up, but only barely.
      • MichaelZuo 5mo ago
        Why would such a critical server even be accessible with only one set of keys? I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.
        • bigfatkitten 5mo ago
          Other than for certain nuclear missile launches[1], that only happens in the movies. [1] https://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact-Sheets/Display/Article/197675/launching-missiles/ https://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact...
          • MichaelZuo 5mo ago
            I dont know how it is in the datacentre industry, but certainly in other industries that is how its done for anything truly mission critical and also easily tampered with. I guess it shows very few care enough to pay enough to make that a reasonable upgrade.
          • deleted 5mo ago
            [deleted]
        • michaelt 5mo ago
          They're not actually accessible with 'only one set of keys' in my experience. You actually have to present your photo ID at the site entry gatehouse, then again to the building entry guard (who will also check you have a work permit and a site-specific safety induction) then you swipe a badge at a turnstile to get from reception into the stairwell, then swipe your badge at a door to get into the relevant floor, then swipe your badge and key in a code to enter the room with the cages then you use the key.
      • wgjordan 5mo ago
        A real example, from Facebook's 2021 outage [1]: > Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems. But this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online. There was one (later denied) report that a 'guy with an angle grinder' was involved in gaining access to the server cage. [1] https://news.ycombinator.com/item?id=28762611 https://news.ycombinator.com/item?id=28762611
    • sidewndr46 5mo ago
      a circular dependency and a single point of failure are not the same thing. If I have a single point of failure and it is down, I fix that and things work again. If I have circular dependency, there is no obvious way to fix anything that is broken any longer.
  • grogenaut 5mo ago
    when you have a circular dependency, one strategy employed, is to have it be circular but interruptible for 18 or so hours. Call it an oh shit bar. I'm glad I never had to get that deep into the failure chain.
    • dlenski 5mo ago
      > Call it an oh shit bar. Amazon's equivalent of this (sort of) was the "andon cord." Not only was the physical metaphor that led to this name never properly explained (it's basically an "emergency stop" string in a Toyota factory), but the actual use of this mechanism was so heavily discouraged that I never saw it used in 4+ years at Amazon. except once very performatively by a VP who had already been paged awake at 2am or something like that. In my experience, a lot of the AWS engineers live in continuous fear of screwing up by using the huge array of extremely powerful, dangerous, poorly-explained, and ever-changing tools that they have access to.
  • superjan 5mo ago
    It reminds me of facebook. Staff was locked out of the office due to the outage they were supposed to fix.
    • deleted 5mo ago
      [deleted]
    • martin8412 5mo ago
      Luckily the plasma torch and bolt cutter didn’t require logging in with Facebook.
• stephenr 5mo ago
  > And IAM is not a wholly independent software stack: they rely on DynamoDB and a few other services, which in turn have a circular dependency on IAM. When you dogfood your own Rube Goldberg machine.
  • zaphirplane 5mo ago
    We should let the IAM service team know if this glaring gap the hn thread figured out /s I’m 99% ;) certain dependencies of foundational services are a well discussed topic
• zaphirplane 5mo ago
  Services outside of us-east-1 don’t call us-east-1 for IAM data plane thou right ?
  • cmiles8 5mo ago
    They’re talking about the backbone and what goes on behind the scenes. There have been issues with services in other regions when us-east-1 has issues. Folks built in other regions believing they were fully isolated only to discover later during an outage that they were not.
• Roark66 5mo ago
  Anyone who thinks one cloud provider will provide them full resilience is fooling themselves. You need multicloud for true high availability. But then you want to use the same stack across providers and all the proprietary technologies (even hidden from you with things like terraform) are suddenly loosing their luster.
  • hnlmorg 5mo ago
    I don’t think any actually believes that. What people usually think is “resilience up to a reasonable level of risk and cost”. Multi-cloud is simply isn’t cost beneficial for 99.9% of problems. And for a lot of businesses who talk about risk, saying “we followed AWS best practices but AWS went down” is an acceptable answer to the question of liability. If you are in a position where AWS going down is a reasonable risk, then you’re already in a specialised enough domain to have engineers who understand how to deliver HA across different vendors.
    • Roark66 5mo ago
      Depends on the product. I'd never guarantee 3 nines (a bit under 9h of downtime per year) to my customers of any product with moderate complexity without multicloud(except for a cloud provider issues that affect entire region). On the subject of cost. Multicloud doesn't have to be expensive, but it requires foresight during the initial design not to lock yourself up in proprietary tech. Things such as AWS cognito, lambda elastic beanstalk, SQS can make it completely uneconomical to do multicloud. But let's say you run roughly equivalent services (load balances, DNS, vms, containers for example under GKE, s3 compatible object storage, etc) it doesn't have to cost an arm and a leg.
  • doublerabbit 5mo ago
    I jest, Anyone who thinks multicloud will provide them full resilience is fooling themselves. You need colocated hardware for true high availability.
• myroon5 5mo ago
  > outside of China [Nitpick] There are a few more AWS partitions like GovCloud: https://jasonbutz.info/2023/07/aws-partitions/ https://jasonbutz.info/2023/07/aws-partitions/
  • electroly 5mo ago
    Since this article was written, AWS also added European Sovereign Cloud as a partition: aws-eusc.
  • dlenski 5mo ago
    Yes, I'm certainly aware of the other partitions. That's why I said all the public cloud regions outside China. Yeah, "govcloud" is technically available to the public, although there are other partitions reserved for government use that are not, and the naming is a big hairy mess. Many service teams don't have any US-citizens-in-the-USA working for them, and they cannot in any way adequately support these regions. My on-call experience improved significantly when I moved from the US to Canada, and I got taken off the (extremely thin!) list of engineers eligible to ssh into RDS instances in Govcloud. There were so few USA-citizen-in-USA engineers that I had been getting tickets for services and instances in Govcloud about which I had only the very thinnest knowledge… and then I was limited in my ability to consult with others who were actually experts. The customers in Govcloud paid a premium to be there, I got paged for a bunch of tickets which I was ill-prepared to handle, and it was generally a bad experience for everyone. Working with the airgapped secret/top-secret partitions was even worse. You would get paged incessantly and then someone who was cleared for access but knew almost nothing about the service in question would have to go to a SCIF in the DC area, and you would exchange screenshots and text instructions with a turnaround time of hours or days.
• jmsgwd 5mo ago
  > The idea that AWS's services are fully regionalized or isolated has always been a myth. This is highly misleading. It's true that there's a handful of global AWS services - but only their control planes operate from a single region (e.g. us-east-1). Their data planes are regionally isolated or globally distributed.[1] The only time you'd normally use a service control plane is to deploy changes, e.g. when you create, read, update or delete service resources or update configuration during a change window. Workloads should be designed for "static stability", as recommended by AWS.[2] A statically stable workload only depends upon the data planes of the services it uses at runtime. Statically stable workloads are designed to continue operating as normal even if there's a service event impairing one or more control planes (including for global services). > During us-east-1 outages it's sometimes possible to continue using existing auth tokens or sessions in other regions, while not possible to grant new ones. This is just plain wrong! The IAM Security Token Service (STS), which grants IAM tokens, is a data plane-only service and runs independently in each region [3]. The IAM data plane, which enforces access control, is also regional. If the IAM control plane is impaired, you might not be able to create new IAM roles (a control plane operation) - but you can continue generating and using temporary credentials for existing IAM roles (data plane operations) within the region your workload is running in. This allows statically stable workloads to continue using IAM without interruption. [1] https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/global-services.html https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso... "Global AWS services still follow the conventional AWS design pattern of separating the control plane and data plane in order to achieve static stability. The significant difference for most global services is that their control plane is hosted in a single AWS Region, while their data plane is globally distributed." [2] https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/static-stability.html https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso... "...eliminating dependencies on control planes (the APIs that implement changes to resources) in your recovery path helps produce more resilient workloads." [3] https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/appendix-a---partitional-service-guidance.html https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso... "STS is a data plane-only service that is separate from IAM, and does not depend on the IAM control plane."
  • dlenski 5mo ago
• master_crab 5mo ago
  IAM isn’t even really the most painful dependency. Route53 is. The control plane only runs out of use1. Better make sure the only DNS operations you run during an outage are data plane queries and health check failovers.
  • trollbridge 5mo ago
    Is there an architectural reason it’s not for replicas in the other AZs?
  • easton 5mo ago
    They actually kind of fixed this recently, you can ask them to move your route53 control plane to another region in the event of us-east-1 breaking: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/accelerated-recovery.html https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/ac... There’s a bunch of caveats but it’s worth enabling if you’re changing dns all the time (as most AWS networking doodads like to do).