5 ms·

AWS doesn't follow their own advice about hosting multi-regional so every time us-east-1 has significant issues pretty much every AZ and region is affected. Sp

by omh2 5y ago

AWS doesn't follow their own advice about hosting multi-regional so every time us-east-1 has significant issues pretty much every AZ and region is affected. Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1. So called Global endpoints are also dependent on us-east-1 and parts of AWS' internal event queues (eg. event bridge triggers) If your infrastructure is static you'll largely avoid the fallout, but if you rely on API calls or dynamically created resources you can get caught in the blast regardless of region

spmurrayzzz 5y ago
Your last comment is really important, I think. I have always petitioned for "passive over active" design in distributed cloud systems. The recent outages, and also ones from the past, demonstrate why. The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for similar reasons. Always felt like a brittle house of cards to me (N.B. I do still use aws lambda, but keep it constrained to non-critical workloads).
- pm90 5y ago
  > The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Agreed; however, this is somewhat difficult to do correctly. There are all sorts of systems that might have hidden dependencies on managed services. e.g. AWS IAM roles will almost always be checked at some point if your services need to interact with AWS managed services. I think cloud providers could meet developers half way here, by providing ways to reduce API usage; but I'm not sure if it aligns with their incentives.
  - electroly 5y ago
    AWS IAM is designed with a control plane / data plane dichotomy. Even if the control plane is completely dead and all API requests are failing, services in a steady-state (i.e. not responding to changes via API calls) can still rely on IAM roles using their cached information. For example, in the recent us-east-1 outage when you couldn't start new services because IAM checks would fail, existing EC2 instances that rely on IAM instance profiles to access services like S3 could still do so even though IAM was down.
    - spmurrayzzz 5y ago
      I was gonna respond with the same commentary here. That has been my experience beyond just IAM controls and why I advocate for passive systems for critical workloads. Sometimes this _can_ be costly. For example with something like autoscaling, thats an active system I've seen fail when seemingly unrelated systems are failing. The result is scaling out systems intentionally ahead of time to deal with oversubscription or burst traffic which can leave you with (costly) idle compute. I don't mind this tradeoff personally, but can understand that budget constraints are going to be different org to org.
  - 0xbadcafebee 5y ago
    It's not an AWS incentive thing really, it's a developer/consumer incentive thing. It's like the duality of modular code. If you want to manage one change in a lot of places, it's easiest to change it in the one module that everything else sources. But that means that one change to that module can take down everything. The alternative where you copy+paste the same change everywhere is the most resilient to failure, but also the most difficult and expensive. AWS provides a lot of modular, dynamic things because that's what their customers want to use. But using each of those things increases the probability of failure. It's up to the customer to decide how they want to design their system using the components available.... and the customers always chose the easy path rather than the resilient path. The great thing is that with AWS, at least you have the option to design a super freaking reliable system. But ultimately there's no way to make it easy, short of a sort of "Heroku for super reliable systems". (I know there are a few, but I don't know anything about them)
    - spmurrayzzz 5y ago
      I like the way you framed this, its a tradeoff mainly. You can build something fault-tolerant and highly-available, even during AWS outage events, but you have to give up a ton of the product offerings in their suite. I've managed to stick to EC2/ELB and S3 as passive systems for the vast majority of what we build at my org (~90% of our stack). And for the most part, AWS failures are hitless for us as a result.

by omh2 5y ago

spmurrayzzz 5y ago

Your last comment is really important, I think. I have always petitioned for "passive over active" design in distributed cloud systems. The recent outages, and also ones from the past, demonstrate why. The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for similar reasons. Always felt like a brittle house of cards to me (N.B. I do still use aws lambda, but keep it constrained to non-critical workloads).

pm90 5y ago
> The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Agreed; however, this is somewhat difficult to do correctly. There are all sorts of systems that might have hidden dependencies on managed services. e.g. AWS IAM roles will almost always be checked at some point if your services need to interact with AWS managed services. I think cloud providers could meet developers half way here, by providing ways to reduce API usage; but I'm not sure if it aligns with their incentives.
- electroly 5y ago
  AWS IAM is designed with a control plane / data plane dichotomy. Even if the control plane is completely dead and all API requests are failing, services in a steady-state (i.e. not responding to changes via API calls) can still rely on IAM roles using their cached information. For example, in the recent us-east-1 outage when you couldn't start new services because IAM checks would fail, existing EC2 instances that rely on IAM instance profiles to access services like S3 could still do so even though IAM was down.
  - spmurrayzzz 5y ago
    I was gonna respond with the same commentary here. That has been my experience beyond just IAM controls and why I advocate for passive systems for critical workloads. Sometimes this _can_ be costly. For example with something like autoscaling, thats an active system I've seen fail when seemingly unrelated systems are failing. The result is scaling out systems intentionally ahead of time to deal with oversubscription or burst traffic which can leave you with (costly) idle compute. I don't mind this tradeoff personally, but can understand that budget constraints are going to be different org to org.
- 0xbadcafebee 5y ago
  It's not an AWS incentive thing really, it's a developer/consumer incentive thing. It's like the duality of modular code. If you want to manage one change in a lot of places, it's easiest to change it in the one module that everything else sources. But that means that one change to that module can take down everything. The alternative where you copy+paste the same change everywhere is the most resilient to failure, but also the most difficult and expensive. AWS provides a lot of modular, dynamic things because that's what their customers want to use. But using each of those things increases the probability of failure. It's up to the customer to decide how they want to design their system using the components available.... and the customers always chose the easy path rather than the resilient path. The great thing is that with AWS, at least you have the option to design a super freaking reliable system. But ultimately there's no way to make it easy, short of a sort of "Heroku for super reliable systems". (I know there are a few, but I don't know anything about them)
  - spmurrayzzz 5y ago
    I like the way you framed this, its a tradeoff mainly. You can build something fault-tolerant and highly-available, even during AWS outage events, but you have to give up a ton of the product offerings in their suite. I've managed to stick to EC2/ELB and S3 as passive systems for the vast majority of what we build at my org (~90% of our stack). And for the most part, AWS failures are hitless for us as a result.