4 ms·

How do you do failover if a server fails or if connectivity to one of those datacenters is lost? With AWS I could just set up a multi-availability-zone RDS depl

How do you do failover if a server fails or if connectivity to one of those datacenters is lost? With AWS I could just set up a multi-availability-zone RDS deployment for the database and an auto-scaling group for the web tier and be confident that AWS will recover the system from most failures. To me, that is the major selling point of any of the hyperscale cloud providers.

KingOfCoders 6y ago
"connectivity to one of those datacenters is lost" Anecdotal, but AWS had more global problems than the triple connected data centers I've used in twenty years. I suffered through many rough times, data center connection or power problems were not (very seldom) one of those. Most of the problems came from apps that we didn't build well to scale or that had bugs (most frequent cause of problems and sitedowns).
HelloNurse 6y ago
> With AWS I could just set up a multi-availability-zone RDS deployment for the database and an auto-scaling group for the web tier and be confident that AWS will recover the system from most failures "Confident"? "Most" failures? Are you merely hopeful that the probability of a bad failure is low, or are you able to test the AWS resiliency techniques you mention and to ensure that they stay working? At what cost?
- milesvp 6y ago
  My experience with RDS instances that had multi region failover, was that the failovers worked every time we needed them for deployments (I don’t think we ever needed them for RDS failures). The cost though was enormous. Our write db represented the lions share of our AWS cost, and doubling it for disaster mitigation increased our costs by something like 50%. It was mostly worth it from a business perspective when we were less sure about AWS uptimes, but I’m not sure I could keep justifying the cost given how little problems we had with RDS over time.
erikbye 6y ago
> How do you do failover if a server fails Not sure if you do not know anything about typical ESXi and vSphere setups, but if a server fails, all virtual machines are automatically migrated to a healthy server. And of course, your HPE G10s are compute only, all storage is on the fiber channel connected SAN.
- Spivak 6y ago
  Yep! We have a KVM and Ceph based setup but it's basically the same as you describe.
- tatersolid 6y ago
  You don’t get 17 server licenses for vSphere or a SAN for $55k as outlined above. More like $500k plus another $100k/year in maintenance. You’re still paying a cloud provider, but in this case it’s VMware
Spivak 6y ago
Physical hardware failures are handled by having everything in VMs and storage handled by Ceph. We can lose plenty of physical boxes simultaneously before we run into capacity issues. Multi-DC failover is handled by announcing our public IP block at both locations with different weights. It’s technically active/active because traffic can come in at the secondary DC but we have a internal site-to-site VPN that is used to direct traffic to the primary. If the primary DC goes down the secondary starts handling the traffic instead of passing it along. All the database masters flip to the secondary and things keep humming along. If we lose the site-to-site then the secondary stops advertising altogether and all traffic is forced to the primary. So we can lose the site-to-site (which is dedicated) or one of the DCs at any time.

KingOfCoders 6y ago

"connectivity to one of those datacenters is lost" Anecdotal, but AWS had more global problems than the triple connected data centers I've used in twenty years. I suffered through many rough times, data center connection or power problems were not (very seldom) one of those. Most of the problems came from apps that we didn't build well to scale or that had bugs (most frequent cause of problems and sitedowns).

HelloNurse 6y ago

> With AWS I could just set up a multi-availability-zone RDS deployment for the database and an auto-scaling group for the web tier and be confident that AWS will recover the system from most failures "Confident"? "Most" failures? Are you merely hopeful that the probability of a bad failure is low, or are you able to test the AWS resiliency techniques you mention and to ensure that they stay working? At what cost?

milesvp 6y ago
My experience with RDS instances that had multi region failover, was that the failovers worked every time we needed them for deployments (I don’t think we ever needed them for RDS failures). The cost though was enormous. Our write db represented the lions share of our AWS cost, and doubling it for disaster mitigation increased our costs by something like 50%. It was mostly worth it from a business perspective when we were less sure about AWS uptimes, but I’m not sure I could keep justifying the cost given how little problems we had with RDS over time.

erikbye 6y ago

> How do you do failover if a server fails Not sure if you do not know anything about typical ESXi and vSphere setups, but if a server fails, all virtual machines are automatically migrated to a healthy server. And of course, your HPE G10s are compute only, all storage is on the fiber channel connected SAN.

Spivak 6y ago
Yep! We have a KVM and Ceph based setup but it's basically the same as you describe.
tatersolid 6y ago
You don’t get 17 server licenses for vSphere or a SAN for $55k as outlined above. More like $500k plus another $100k/year in maintenance. You’re still paying a cloud provider, but in this case it’s VMware

Spivak 6y ago

Physical hardware failures are handled by having everything in VMs and storage handled by Ceph. We can lose plenty of physical boxes simultaneously before we run into capacity issues. Multi-DC failover is handled by announcing our public IP block at both locations with different weights. It’s technically active/active because traffic can come in at the secondary DC but we have a internal site-to-site VPN that is used to direct traffic to the primary. If the primary DC goes down the secondary starts handling the traffic instead of passing it along. All the database masters flip to the secondary and things keep humming along. If we lose the site-to-site then the secondary stops advertising altogether and all traffic is forced to the primary. So we can lose the site-to-site (which is dedicated) or one of the DCs at any time.