3 ms·

> Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing. We are talking

> Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing. We are talking about MultiAZ, Availability Zones. Not different regions. Setting up redundancy across regions is not easy. But for majority of the people using AWS a single region with MultiAZ is good enough.

bombcar 4y ago
It's hard to simulate an AZ failing, and when it does, half the internet goes offline, so people don't really complain that much.
- philliphaydon 4y ago
  About 5 or 6 years ago we had an alert in the middle of the night that our RDS instance dieded. It failed over in about 15 seconds (SQL Server so it’s a bit slow compared to PostgreSQL) but the MultiAZ worked as advertised. The downside is AWS never told us why it occurred.
  - ceejayoz 4y ago
    They don't tell you because you're not supposed to care, and there's no human involved in the process to do a post-mortem. Something on the instance host died, most likely.
  - WatchDog 4y ago
    I’ve seen a few AWS instance hardware failures, they happen with some regularity. You can handle single instance failure without being multi AZ. Testing an actual AZ failure, as in the whole AZ going offline or getting partitioned from the other AZs, is pretty much impossible.
    - fulafel 4y ago
      AZs are connected via normal user visible networks, you can just break those. They even provide examples, https://github.com/awslabs/aws-well-architected-labs/tree/master/static/Reliability/300_Testing_for_Resiliency_of_EC2_RDS_and_S3/Code/FailureSimulations https://github.com/awslabs/aws-well-architected-labs/tree/ma... Those are basic (don't cover flapping or glacial-speed slowdown degradation modes, some services only, etc) but a starting point at least that can be extended.
      - ceejayoz 4y ago
        Huh, didn't know about aws rds reboot-db-instance --force-failover

bombcar 4y ago

It's hard to simulate an AZ failing, and when it does, half the internet goes offline, so people don't really complain that much.

philliphaydon 4y ago
About 5 or 6 years ago we had an alert in the middle of the night that our RDS instance dieded. It failed over in about 15 seconds (SQL Server so it’s a bit slow compared to PostgreSQL) but the MultiAZ worked as advertised. The downside is AWS never told us why it occurred.
- ceejayoz 4y ago
  They don't tell you because you're not supposed to care, and there's no human involved in the process to do a post-mortem. Something on the instance host died, most likely.
- WatchDog 4y ago
  I’ve seen a few AWS instance hardware failures, they happen with some regularity. You can handle single instance failure without being multi AZ. Testing an actual AZ failure, as in the whole AZ going offline or getting partitioned from the other AZs, is pretty much impossible.
  - fulafel 4y ago
    AZs are connected via normal user visible networks, you can just break those. They even provide examples, https://github.com/awslabs/aws-well-architected-labs/tree/master/static/Reliability/300_Testing_for_Resiliency_of_EC2_RDS_and_S3/Code/FailureSimulations https://github.com/awslabs/aws-well-architected-labs/tree/ma... Those are basic (don't cover flapping or glacial-speed slowdown degradation modes, some services only, etc) but a starting point at least that can be extended.
    - ceejayoz 4y ago
      Huh, didn't know about aws rds reboot-db-instance --force-failover