2 ms·

How much good do you think those disaster recovery datacenters actually do? Adapting all our software for multi-region was an epic lift. Really, we had to stop

How much good do you think those disaster recovery datacenters actually do? Adapting all our software for multi-region was an epic lift. Really, we had to stop thinking of it as "the backup datacenter" and instead run a portion of traffic there continuously. Most teams dragged their feet until we started weekly failover drills, which would usually break something in a big way. Application owners got embarrassed into compliance over the course of a few years. They're less exciting now, but still the proximate cause of some outages. Given all that, I just can't imagine that a company which has never actually cut over (or attempts it only in quarterly/annual all-hands-on-deck events) is going to have a fun time when it becomes necessary to flip the switch for real. We fought tooth and nail to build this capability and it still takes balls of steel to press the button.

freehunter 6y ago
I have never worked with a client who did not test their DR solution. If they were not under regulation to test, they had customers who demanded they test. Although yes it’s mostly quarterly. You’re right it’s not fun. It’s not easy. It’s all hands on deck at most companies. But disaster recovery isn’t something you can build after the fact so even a neglected DR solution is better than none. If a tornado or earthquake rips through your main DC, you’re not getting those machines or that data back. How much good that does depends on how much you like staying in business.
- closeparen 6y ago
  So is the goal just to have something to work with in order to get back online? I guess that is more achievable than trying to maintain continuous service, which is what ours are about (failover isn't just for black swan events, but for ordinary outages too).
  - freehunter 6y ago
    We tend to differentiate between disaster recovery (where downtime is acceptable) versus high availability (where downtime is not acceptable). High availability is much harder and more expensive. DR typically means your business is down and all hands bring it back at another location at bare minimum capacity just to keep money flowing.

freehunter 6y ago

I have never worked with a client who did not test their DR solution. If they were not under regulation to test, they had customers who demanded they test. Although yes it’s mostly quarterly. You’re right it’s not fun. It’s not easy. It’s all hands on deck at most companies. But disaster recovery isn’t something you can build after the fact so even a neglected DR solution is better than none. If a tornado or earthquake rips through your main DC, you’re not getting those machines or that data back. How much good that does depends on how much you like staying in business.

closeparen 6y ago
So is the goal just to have something to work with in order to get back online? I guess that is more achievable than trying to maintain continuous service, which is what ours are about (failover isn't just for black swan events, but for ordinary outages too).
- freehunter 6y ago
  We tend to differentiate between disaster recovery (where downtime is acceptable) versus high availability (where downtime is not acceptable). High availability is much harder and more expensive. DR typically means your business is down and all hands bring it back at another location at bare minimum capacity just to keep money flowing.