3 ms·
We once had a long power outage (3.5 hrs ish), the data center remained on the entire time, but after about 60 minutes the data center lost internet access(!) b
by Someone1234 10y ago
We once had a long power outage (3.5 hrs ish), the data center remained on the entire time, but after about 60 minutes the data center lost internet access(!) because their ISP only had a battery backup with a limited capacity. So it does happen.
Plus sometimes backup power isn't reliable (e.g. generator problem) or they have different tiers of coverage (and Delta didn't pay for it). We'll just have to wait and see.
- philipov 10y agoThat's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their failover procedure.
- ht85 10y agoIt depends how much it would cost you to maintain vs the cost of a possible downtime. There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial. Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.
- toomuchtodo 10y agoBoth Equinix and AWS have had cases where the genset tests pass, but entire DC blackouts occurred due to a fault only a few months later.
- kpil 10y agoYup. I've worked at several places where there has been battery backup, diesel generators, and a a disaster recovery site with generally nice and thorough strategies for replication including really expensive "sysplex-mode" for mainframes, topped with regular contigency tests where one half of the site is shut down. And somehow, at every place, eventually something bad happened in a way that took down the applications. It could be a power outage in the exactly the wrong circuit, fluctuating power messed up exactly the wrong storage system, someone dug up exactly the wrong fiber, the fire extinguisher set off by mistake and sprayed just the wrong cabinet with water fog through a door(!) that was supposed to automatically close, the diesel generator caught fire and the backup battery had to be taken offline, etc etc. I'm not saying that you shouldn't try, but it's damn hard.
- deleted 10y ago[deleted]
- DoubleGlazing 10y agoI worked for a company that thought it had a really top level power backup system. The server room had 15 minutes of UPS, all PCs had at least 5 mins. Out the back was a big generator with 24 hours worth of diesel. It was all controlled by a fancy control system that fed light displays in every room indicating if we were on mains power, UPS power or generator power. One day during a power cut the generator kicked in as it was supposed to, but a minute later it spluttered to a halt. After much fiddling around by maintenance staff they gave up and we were all told to go home. The postmortem revealed that the diesel had become contaminated with all sorts of gunk - water, grit/sand and lumps of unidentifiable "stuff". The best laid plans and all that. Edit: And cigarette butts as well.