4 ms·

That's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their f

That's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their failover procedure.

ht85 10y ago
It depends how much it would cost you to maintain vs the cost of a possible downtime. There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial. Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.
toomuchtodo 10y ago
Both Equinix and AWS have had cases where the genset tests pass, but entire DC blackouts occurred due to a fault only a few months later.
kpil 10y ago
Yup. I've worked at several places where there has been battery backup, diesel generators, and a a disaster recovery site with generally nice and thorough strategies for replication including really expensive "sysplex-mode" for mainframes, topped with regular contigency tests where one half of the site is shut down. And somehow, at every place, eventually something bad happened in a way that took down the applications. It could be a power outage in the exactly the wrong circuit, fluctuating power messed up exactly the wrong storage system, someone dug up exactly the wrong fiber, the fire extinguisher set off by mistake and sprayed just the wrong cabinet with water fog through a door(!) that was supposed to automatically close, the diesel generator caught fire and the backup battery had to be taken offline, etc etc. I'm not saying that you shouldn't try, but it's damn hard.
deleted 10y ago
[deleted]

That's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their failover procedure.

ht85 10y ago

It depends how much it would cost you to maintain vs the cost of a possible downtime. There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial. Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.

toomuchtodo 10y ago

Both Equinix and AWS have had cases where the genset tests pass, but entire DC blackouts occurred due to a fault only a few months later.

kpil 10y ago

Yup. I've worked at several places where there has been battery backup, diesel generators, and a a disaster recovery site with generally nice and thorough strategies for replication including really expensive "sysplex-mode" for mainframes, topped with regular contigency tests where one half of the site is shut down. And somehow, at every place, eventually something bad happened in a way that took down the applications. It could be a power outage in the exactly the wrong circuit, fluctuating power messed up exactly the wrong storage system, someone dug up exactly the wrong fiber, the fire extinguisher set off by mistake and sprayed just the wrong cabinet with water fog through a door(!) that was supposed to automatically close, the diesel generator caught fire and the backup battery had to be taken offline, etc etc. I'm not saying that you shouldn't try, but it's damn hard.

deleted 10y ago

[deleted]