20 ms·

Oh, they had the whole line. UPS in the server room, with batteries and diesel engine outside. Tested quarterly, I was told. In server room batteries did the

Oh, they had the whole line. UPS in the server room, with batteries and diesel engine outside. Tested quarterly, I was told. In server room batteries did their job, but the stuff outside didn't. Took 4 hours to restore power and another 32 hours to restart the servers and verify their data. [edit: about the site visit calling me paranoid] I was a bit ticked off, but I guess someone who didn't even do the basics, as we later found out, would not get the need.

jacquesm 9y ago
Ok. Even a small DC that I'm intimately familiar with has a back-up diesel that they test religiously on the 1st of every month, they have the whole building running on the genset for an hour or two and then they shut it all down again. Iirc it failed to start twice in the years that this setup has been there, which means the chances of a problem will never be quite '0'. When I ran a gas station in North America one of the first things I did was install a diesel generator. My reasoning was that with on average 50K liters of diesel in the underground tanks and the genset powering the pumps it should be good for most problems that we'd face. And in fact it was, which we managed to prove beyond any doubt when all of the North-West of the US and a good chunk of Canada went without power for a couple of days within a year of installing that genset. Sometimes being prepared is a waste of money and time in retrospect, but it helps you sleep well, and sometimes it is the difference between a community continuing to function or people dying.
- Animats 9y ago
  The classic Bell System standard was that generators were started once a month and run for several hours. Once a year, each central office ran on emergency power for 24 hours, with external power shut off. In the entire history of the Bell System prior to electronic switching, no central office was ever offline for more than 30 minutes for any reason other than a natural disaster.
  - MandieD 9y ago
    Of course I've known that a DR plan isn't valid until you've tested it against your production environment, but I'd never heard that Bell System story before. It's inspirational - run 24 hours in emergency mode each year, with the normal way turned off/disconnected, don't just do failover/failback. Thanks for bringing it up!
    - Animats 9y ago
      They'd discover such things as a basement sump pump that wasn't on emergency power.

jacquesm 9y ago

Ok. Even a small DC that I'm intimately familiar with has a back-up diesel that they test religiously on the 1st of every month, they have the whole building running on the genset for an hour or two and then they shut it all down again. Iirc it failed to start twice in the years that this setup has been there, which means the chances of a problem will never be quite '0'. When I ran a gas station in North America one of the first things I did was install a diesel generator. My reasoning was that with on average 50K liters of diesel in the underground tanks and the genset powering the pumps it should be good for most problems that we'd face. And in fact it was, which we managed to prove beyond any doubt when all of the North-West of the US and a good chunk of Canada went without power for a couple of days within a year of installing that genset. Sometimes being prepared is a waste of money and time in retrospect, but it helps you sleep well, and sometimes it is the difference between a community continuing to function or people dying.

Animats 9y ago
The classic Bell System standard was that generators were started once a month and run for several hours. Once a year, each central office ran on emergency power for 24 hours, with external power shut off. In the entire history of the Bell System prior to electronic switching, no central office was ever offline for more than 30 minutes for any reason other than a natural disaster.
- MandieD 9y ago
  Of course I've known that a DR plan isn't valid until you've tested it against your production environment, but I'd never heard that Bell System story before. It's inspirational - run 24 hours in emergency mode each year, with the normal way turned off/disconnected, don't just do failover/failback. Thanks for bringing it up!
  - Animats 9y ago
    They'd discover such things as a basement sump pump that wasn't on emergency power.