3 ms·

> Who is surprised here it took them this long to recover? They have no independent management infrastructure. Their management network rides logically on top

> Who is surprised here it took them this long to recover? They have no independent management infrastructure. Their management network rides logically on top of their main physical network. So when the latter went down you probably had people going to the physical locations of various DCs. But then you have to coördinate between them, and if your employees use your own cell phone service for communication, and that's down… So the first couple of hours was figuring out what went wrong, then it was having people scramble to locations to plug in serial consoles, then it was buying a bunch of SIM card from other companies, and finally folks managed to wrap their heads around the big picture and come up with a way to bootstrap the network.

mjevans 4y ago
An independent management infra would also help to assess as well as recover. There might be a need to require either such an independent network, or one that fully does not reuse your own internal network as a backup link for the management interface.
yourad_io 4y ago
> buying a bunch of SIM card from other companies Sounds like some employees already had those for such an event, which is impressive. From the article: > some employees started swapping out their SIM cards for Bell or Telus SIM cards that they had received back in 2015 as part of an emergency contingency plan established between the wireless carriers.