4 ms·

If you're not testing az failovers you're probably just wasting money, like untested backups... But it's true that most people don't know this and aren't testin

If you're not testing az failovers you're probably just wasting money, like untested backups... But it's true that most people don't know this and aren't testing it because they don't know that az failovers don't automatically just work(tm). And the second downside of course would be asymmetry between the envs and divergence in the IaC etc resulting in more complexity and engineering work. (Of course you're probably wasting money anyway since you don't actually business-wise need better uptime than the single AZ, and your complexity induced human fumbles will cause much more outages anyway, but this has been a main selling point of the decision to go to AWS, so the requirement needs to be defended) Yes, you can build automation to have the redundant stuff up only sometimes, if you eat the engineering effort and complexity in your IAC and build automation... in the general vein of justifying engineering spend to offset AWS operating costs where running containers is very expensive! TLDR: either way you end up paying for the very high markup in compute prices, it'll just be easier to excuse jumping through expensive hoops to "save money" on it.

philliphaydon 4y ago
If you’re building your own infrastructure in a data center then sure you absolutely want to test your redundancy. But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds.
- fulafel 4y ago
  IME not so in the real world, you'll have accidental state in your distributed system outside the DBs. You'll have some stuff that actually always runs in one AZ in normal circumstances and your integration partner in another org has whitelisted only its IP. Etc etc, everything that is untested will find ways to conspire to rot. Especially if you haven't learned by seeing these bugs so you can avoid them. Also you won't have a clear experience and understanding of what happens in the failover and you'll won't know to avoid failover breaking mistakes in your VPC configs, security groups, frontend-backend shared state etc. (And by "you" I mean "your dev team", it's not enough that one guy gets it). Also^2 if you read the news about all the outages it's very common for failover systems to fail generally, not just AWS - the general engineering wisdom is: always test your failovers. And there's no substitute for end-to-end testing it, instead of individually testing each layer/module. (Bad: "we can skip testing db failover", good: "let's test that the whole system works when there's a az failure")
  - russellendicott 4y ago
    Dealing with this now for a client. Can't test Redshift AZ relocation feature because there's no way to simulate AZ failure. Only safe bet is full multi region with DNS switcheroo.
- moonchrome 4y ago
  >But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds. Have you actually seen this work on your project in practice ? Like a region go down and another region picks up automatically and it kept working just by switching a checkbox ?
  - philliphaydon 4y ago
    Multi AZ is multiple availability zones. Not multi region. Distribution over multiple regions is obviously harder than within the same region and different zones.
- sokoloff 4y ago
  Back in colo days, I saw a lot of post-mortems that read “the thing we thought was redundant wasn’t”, leading me to call them “redundant like an appendix [rather than like a kidney]”. We instituted quarterly “game day testing” where we forcibly turned off one of the redundant items in all of our systems. It took us about 6 such cycles before these tests didn’t turn up outages that were just waiting for us. Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing.
  - philliphaydon 4y ago
    > Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing. We are talking about MultiAZ, Availability Zones. Not different regions. Setting up redundancy across regions is not easy. But for majority of the people using AWS a single region with MultiAZ is good enough.
    - bombcar 4y ago
      It's hard to simulate an AZ failing, and when it does, half the internet goes offline, so people don't really complain that much.
      - philliphaydon 4y ago
        About 5 or 6 years ago we had an alert in the middle of the night that our RDS instance dieded. It failed over in about 15 seconds (SQL Server so it’s a bit slow compared to PostgreSQL) but the MultiAZ worked as advertised. The downside is AWS never told us why it occurred.
        ceejayoz 4y ago
        They don't tell you because you're not supposed to care, and there's no human involved in the process to do a post-mortem. Something on the instance host died, most likely.
        4y ago
- deathanatos 4y ago
  > But with AWS it’s a checkbox. And I'd test the checkbox all the same. We learned just this week that one of our setups, which checks the cloud-provider provided box to have its VMs distributed across 3 AZs, is susceptible to the loss of a single AZ. Why? Because the resulting VMs … aren't actually distributed across 3 AZs as requested. (The provider has "reasons" for this, but they're dumb, IMO. It should have been as easy as checking the box.)
tpetry 4y ago
Are AZ failovers not done automatically? Where can I read more about that?
- fulafel 4y ago
  They are from user pov just separate networks where you can deploy separate copies of services, replicated db cluster nodes &whatnot. Each service handles (knock on wood) an az becoming unreachable/slow/crazy independently. Which can become fun with service interdependencies, and a mix of your self implemented services + AWS provided ones. There's high level description at https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio...

philliphaydon 4y ago

If you’re building your own infrastructure in a data center then sure you absolutely want to test your redundancy. But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds.

fulafel 4y ago
IME not so in the real world, you'll have accidental state in your distributed system outside the DBs. You'll have some stuff that actually always runs in one AZ in normal circumstances and your integration partner in another org has whitelisted only its IP. Etc etc, everything that is untested will find ways to conspire to rot. Especially if you haven't learned by seeing these bugs so you can avoid them. Also you won't have a clear experience and understanding of what happens in the failover and you'll won't know to avoid failover breaking mistakes in your VPC configs, security groups, frontend-backend shared state etc. (And by "you" I mean "your dev team", it's not enough that one guy gets it). Also^2 if you read the news about all the outages it's very common for failover systems to fail generally, not just AWS - the general engineering wisdom is: always test your failovers. And there's no substitute for end-to-end testing it, instead of individually testing each layer/module. (Bad: "we can skip testing db failover", good: "let's test that the whole system works when there's a az failure")
- russellendicott 4y ago
  Dealing with this now for a client. Can't test Redshift AZ relocation feature because there's no way to simulate AZ failure. Only safe bet is full multi region with DNS switcheroo.
moonchrome 4y ago
>But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds. Have you actually seen this work on your project in practice ? Like a region go down and another region picks up automatically and it kept working just by switching a checkbox ?
- philliphaydon 4y ago
  Multi AZ is multiple availability zones. Not multi region. Distribution over multiple regions is obviously harder than within the same region and different zones.
sokoloff 4y ago
Back in colo days, I saw a lot of post-mortems that read “the thing we thought was redundant wasn’t”, leading me to call them “redundant like an appendix [rather than like a kidney]”. We instituted quarterly “game day testing” where we forcibly turned off one of the redundant items in all of our systems. It took us about 6 such cycles before these tests didn’t turn up outages that were just waiting for us. Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing.
- philliphaydon 4y ago
  > Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing. We are talking about MultiAZ, Availability Zones. Not different regions. Setting up redundancy across regions is not easy. But for majority of the people using AWS a single region with MultiAZ is good enough.
  - bombcar 4y ago
    It's hard to simulate an AZ failing, and when it does, half the internet goes offline, so people don't really complain that much.
    - philliphaydon 4y ago
      About 5 or 6 years ago we had an alert in the middle of the night that our RDS instance dieded. It failed over in about 15 seconds (SQL Server so it’s a bit slow compared to PostgreSQL) but the MultiAZ worked as advertised. The downside is AWS never told us why it occurred.
      - ceejayoz 4y ago
        They don't tell you because you're not supposed to care, and there's no human involved in the process to do a post-mortem. Something on the instance host died, most likely.
      - 4y ago
deathanatos 4y ago
> But with AWS it’s a checkbox. And I'd test the checkbox all the same. We learned just this week that one of our setups, which checks the cloud-provider provided box to have its VMs distributed across 3 AZs, is susceptible to the loss of a single AZ. Why? Because the resulting VMs … aren't actually distributed across 3 AZs as requested. (The provider has "reasons" for this, but they're dumb, IMO. It should have been as easy as checking the box.)

tpetry 4y ago

Are AZ failovers not done automatically? Where can I read more about that?

fulafel 4y ago
They are from user pov just separate networks where you can deploy separate copies of services, replicated db cluster nodes &whatnot. Each service handles (knock on wood) an az becoming unreachable/slow/crazy independently. Which can become fun with service interdependencies, and a mix of your self implemented services + AWS provided ones. There's high level description at https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio...