4 ms·
It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of
by londons_explore 7y ago
It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.
- karlding 7y agoI don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case? The article states: > The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.
- log_n 7y agoYou don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance. Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).
- smileypete 7y agoUse a 28k modem? :)
- _n_b_ 7y agoAn anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work. My point being, it seems that modems are becoming less-and-less viable for out-of-band management.
- Bluecobra 7y agoFun story, AT&T forced our hands to get off our PRI (voice T1) and move to their fiber service. They also insisted that they have a dedicated phone line installed so they can dial into their modem in case of circuit failure. We can’t buy a cooper phone line from them, so it gets routed over the same fiber circuit and goes to a digital to analog device back to the router. I don’t think one hand talks to the other over there...
- bnjms 7y agoIf this is a problem consider Opengear. https://opengear.com/products/acm7000-resilience-gateway https://opengear.com/products/acm7000-resilience-gateway