4 ms·
Back in colo days, I saw a lot of post-mortems that read “the thing we thought was redundant wasn’t”, leading me to call them “redundant like an appendix [rathe
by sokoloff 4y ago
Back in colo days, I saw a lot of post-mortems that read “the thing we thought was redundant wasn’t”, leading me to call them “redundant like an appendix [rather than like a kidney]”.
We instituted quarterly “game day testing” where we forcibly turned off one of the redundant items in all of our systems. It took us about 6 such cycles before these tests didn’t turn up outages that were just waiting for us.
Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing.
- philliphaydon 4y ago> Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing. We are talking about MultiAZ, Availability Zones. Not different regions. Setting up redundancy across regions is not easy. But for majority of the people using AWS a single region with MultiAZ is good enough.
- bombcar 4y agoIt's hard to simulate an AZ failing, and when it does, half the internet goes offline, so people don't really complain that much.
- philliphaydon 4y agoAbout 5 or 6 years ago we had an alert in the middle of the night that our RDS instance dieded. It failed over in about 15 seconds (SQL Server so it’s a bit slow compared to PostgreSQL) but the MultiAZ worked as advertised. The downside is AWS never told us why it occurred.
- ceejayoz 4y agoThey don't tell you because you're not supposed to care, and there's no human involved in the process to do a post-mortem. Something on the instance host died, most likely.
- WatchDog 4y agoI’ve seen a few AWS instance hardware failures, they happen with some regularity. You can handle single instance failure without being multi AZ. Testing an actual AZ failure, as in the whole AZ going offline or getting partitioned from the other AZs, is pretty much impossible.
- fulafel 4y agoAZs are connected via normal user visible networks, you can just break those. They even provide examples, https://github.com/awslabs/aws-well-architected-labs/tree/master/static/Reliability/300_Testing_for_Resiliency_of_EC2_RDS_and_S3/Code/FailureSimulations https://github.com/awslabs/aws-well-architected-labs/tree/ma... Those are basic (don't cover flapping or glacial-speed slowdown degradation modes, some services only, etc) but a starting point at least that can be extended.
- ceejayoz 4y agoHuh, didn't know about aws rds reboot-db-instance --force-failover
- DonHopkins 4y agoRedundant like a Klingon penis. https://www.inverse.com/article/41443-star-trek-klingon-penis-canon-discovery-writers https://www.inverse.com/article/41443-star-trek-klingon-peni...