5 ms·
Well, there’s the problem right there. There was an individual responsible, meaning one set of hands unreviewed on a keyboard, for a change that could cause a g
by brians 9y ago
Well, there’s the problem right there. There was an individual responsible, meaning one set of hands unreviewed on a keyboard, for a change that could cause a global outage.
If I’m that person responsible, I’m going to hire two staff, ask them each to write the command scripts, justify any differences, produce a consensus script for my review, and then implement it. That seems like the minimum level of responsible engineering. Lint tools, notices of novel commands, and other ornaments have a place too, but the core idea is this:
The person with hands on keyboard is not the individual responsible for this error.
- jeremyjh 9y agoWhere did they say the change was not reviewed?
- jdwithit 9y agoIt's implied by their claim that there is a single responsible individual. If someone else reviewed it and said "yep, looks good, deploy the change" how is there one "individual responsible"? The idea of an isolated root cause or single human error in the failure of complex systems is bogus anyway. I'm a huge fan of the work in this area championed by John Allspaw [0]. [0]: https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...
- dboreham 9y agoIndeed. Someone runs the department that individual works in and allowed this kind of uncontrolled process. Someone is that person’s boss and should have been asking how well controlled are our processes, and so on. Turned Turtles all the way up..
- jlgaddis 9y agoThis could have been as simple as the individual accidentally "missing" a single line of a large, multi-line configuration when copy/pasting it into the router's console -- after all of the config review, etc., already occurred. I'm not sure what vendor's gear was in use in this particular case, but the configs for a BGP peering session are typically (as mentioned above) large, multi-line configurations. For example, here's the (slightly redacted) configuration for one single BGP session on one of my routers: neighbor 10.10.10.10 remote-as 65432 neighbor 10.10.10.10 transport connection-mode passive neighbor 10.10.10.10 description TO CUST FOO BAR INC ... neighbor 10.10.10.10 ebgp-multihop 3 neighbor 10.10.10.10 update-source Loopback0 neighbor 10.10.10.10 send-community neighbor 10.10.10.10 soft-reconfiguration inbound neighbor 10.10.10.10 prefix-list ACCEPTED-PREFIXES-AS65432 in neighbor 10.10.10.10 prefix-list ADVERTISED-PREFIXES-AS65432 out neighbor 10.10.10.10 password 7 0123456789ABCDEF0123456789ABCDEF neighbor 10.10.10.10 maximum-prefix 200 neighbor 10.10.10.10 default-originate That doesn't even include the associate prefix lists (or filter lists or route-maps or ...). All it would take is fat-fingering/typo'ing one of these lines or missing one to cause some very unintended effects. Since we don't know exactly what happened, it's easy to say "they should've done this" or "they didn't do that". In reality, however, we simply don't know what they did or didn't do. You've shown no evidence that they didn't do any of the things you mention and, in some cases, you can do all of that and still have things go wrong.