3 ms·
and for over 3 hours? What could they do accidentally that would cause an outage for this long? The tweet says they're failing over to their backup facility. I
by MichaelApproved 6y ago
and for over 3 hours? What could they do accidentally that would cause an outage for this long?
The tweet says they're failing over to their backup facility. I would've expected that fail over to happen much faster.
Seems like they have two issues going on. First, the remote hands could take down the datacenter. Second, their fail over is taking this long to come online.
I also wonder how much Covid impacted the process, if at all.
I'm looking forward to the details after they get things back online.
Good luck CF engineers!
- foobarbazetc 6y agoProbably nuked the main DB.
- eb0la 6y agoThree hours for a physical problem like this is very little time. Once you discover your equipment is not there, you still have to put it on its place, power up, reconnect al cabling, and check networking. From this kind of mess to a fully functional infrastructure you need at least 12h-16h to function minimally. Probably takes 2-3 days to have that node working as before. When I was on call, I always jocked with my colleages the worst incident you could have in a datatenter was someone swapping two cables.