3 ms·
I work at a telco with a lot of aging systems and servers. A couple of years ago we had a total power failure of our major data center and central office, incl
by grecy 12y ago
I work at a telco with a lot of aging systems and servers.
A couple of years ago we had a total power failure of our major data center and central office, including the backup gen so literally everything failed and an entire province of Canada (and most of two more) were without any telco, data, wireless.
It was eye opening and extremely scary to see what happens when everything is turned off at once and you need to bring stuff back up. Some of the equipment had never been turned off, and certainly not all at once.
During the next weeks we encountered many, many catch-22 scenarios like "Before we turn on System A, it needs system B to be online. B needs C, C needs A." Oh.
For reference of how severe this is, paramedics were posting up at major intersections in towns and communities in case people needed help - and the radio stations in half of Canada were telling people what to do. When even 911 doesn't work you have problems.
- OldSchoolJohnny 12y agoAs a Canadian I would love to see a citation to this because it would have been pretty big news here and I recall nothing about it.
- grecy 12y agohttp://www.ctvnews.ca/canada/power-outage-in-yukon-knocks-out-communication-systems-1.964656 http://www.ctvnews.ca/canada/power-outage-in-yukon-knocks-ou... It took out everything in the Yukon, and tons of stuff in NWT, Nunavut and Northern BC and Alberta.
- serve_yay 12y agoThis is how "if it ain't broke, don't fix it" calcifies. People point to old systems like this (when they are working, that is) all the time as proof that developers are too obsessed with shiny new things. And I'm not saying that isn't true, but there is such a thing as being too conservative.
- grecy 12y ago> there is such a thing as being too conservative In my experience, a telco is the definition of "too conservative".
- digi_owl 12y agoI guess the catch-22 came into being because A was at some point bootstrapped from some other system that was since replaced by B or C.