8 ms·
> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.
by dnautics 7y ago
> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network.
Man that's got to suck.
- londons_explore 7y agoIt isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.
- karlding 7y agoI don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case? The article states: > The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.
- log_n 7y agoYou don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance. Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).
- smileypete 7y agoUse a 28k modem? :)
- _n_b_ 7y agoAn anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work. My point being, it seems that modems are becoming less-and-less viable for out-of-band management.
- Bluecobra 7y agoFun story, AT&T forced our hands to get off our PRI (voice T1) and move to their fiber service. They also insisted that they have a dedicated phone line installed so they can dial into their modem in case of circuit failure. We can’t buy a cooper phone line from them, so it gets routed over the same fiber circuit and goes to a digital to analog device back to the router. I don’t think one hand talks to the other over there...
- bnjms 7y agoIf this is a problem consider Opengear. https://opengear.com/products/acm7000-resilience-gateway https://opengear.com/products/acm7000-resilience-gateway
- raverbashing 7y agoThis happens way more often than you think A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive
- Cthulhu_ 7y agoIs there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?
- Causality1 7y agoI'm a bit surprised there's no sort of SSH undo subroutine that reverses the previous command if connectivity is lost. Of course it couldn't cover every possible stupid thing but it could fix simple stupid mistakes like fouling up a port assignment or disabling the wrong network adapter.
- Spearchucker 7y agoRecovery-oriented computing. Seperate Bohr bugs (easy to reproduce) from Heisenbugs (difficult to reproduce/change their behaviour once investigated). Bohr bugs generate an alert and happily meander through normal support channels. Heisenbugs go through phases - 1. Probation. On continued failure, 2. Restart. If the app or service fails after a restart, 3. Reboot. If the app or service fails after a reboot, 4. Re-image. If the app or service fails after re-imaging, 5. Remove/elimate the node.
- bpchaps 7y ago
- KirinDave 7y agoYou have no idea.
- oldcreek12 7y agoThis is inconceivable ... they don't have an OOB management network?
- kbirkeland 7y agoA completely OOB management network is an amazingly high cost when you have presence all over the world. I don't think anybody has gone to the length to double up on dark fiber and OTN gear just for management traffic.
- riskrisk 7y agoThat's less of an issue. Issue is in how you classify traffic on a network. e.g. Gmail, it's helpful to incident response, should it be used for OOB management.
- ikiris 7y agoStill have to get to it.
- abbeyj 7y agoThis reminded me of https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.
- theflyinghorse 7y agoWhat an amazing read!
- tbodt 7y agoYou joke, but I've read a Google postmortem with that exact quote at the top. This incident would be more fitting though.
- ikiris 7y agoIt's in this one now.