6 ms·
From elsewhere: > Root Cause: A configuration issue impacted IP services in various markets across the United States. > Fix Action: The IP NOC reverted a poli
by jlgaddis 9y ago
From elsewhere:
> Root Cause: A configuration issue impacted IP services in various markets across the United States.
> Fix Action: The IP NOC reverted a policy change to restore services to a stable state.
> Summary: The IP NOC was informed of a significant client impact which seemed to originate on the east coast. The IP NOC began investigating, and soon discovered that the service impact was occurring in various markets across the United States. The issue was isolated to a policy change that was implemented to a single router in error while trying to configure an individual customer BGP. This policy change affected a major public peering session. The IP NOC reverted the policy change to restore services to a stable state.
> Corrective Actions: An extensive post analysis review will be
conducted to evaluate preventative measures and corrective actions that can be implemented to prevent network impact of this magnitude. The individual responsible for this policy change has been identified.
[snip]
Sounds like "the individual responsible" forgot to set some communities on the peering session. Oops.
- irl_zebra 9y agoDo you have a link for this? Interested in the corrective action. Also, any suggestions on reading to learn about bgp generally to the specificity one might learn IP/TCP from a networking book?
- jlgaddis 9y agoI don't, sorry, it's in my inbox. The part I snipped is completely unrelated to the incident. As far as BGP goes, Halabi's _Internet Routing Architectures_ [0] is pretty much considered the "bible". It's really old nowadays but it covers BGP4 (the current version in use) and not much has really changed. I'm sure some of the newer BGP books are excellent as well but I can't personally recommend them as IRA and the (old) CCNP BGP book are all I've ever read/used (while preparing for the CCNP certification and in my day job). Of course, pretty much everything is covered in RFC 4271 [1] (and updates) although the RFCs can be a bit "dry". [0]: https://www.amazon.com/dp/157870233X https://www.amazon.com/dp/157870233X [1]: https://tools.ietf.org/html/rfc4271 https://tools.ietf.org/html/rfc4271
- knorrie 9y agoI wrote a routing protocols tutorial a while ago, that tries not to be overly complex and not "dry". https://github.com/knorrie/network-examples/blob/master/README.md https://github.com/knorrie/network-examples/blob/master/READ... It uses the bird routing daemon on linux to build some networks on the go and see OSPF and BGP happening. Maybe it can help you a bit. :-)
- jdwithit 9y agoI found "BGP4: Inter-domain Routing in the Internet" by John Stewart III [0] to be very approachable. It's likewise ancient by tech world standards (published in 1998!) but that's because, amazingly, BGP has not fundamentally changed in all that time. It's cheap and less than 200 pages; definitely recommend it as a primer. [0]: https://www.amazon.com/dp/0201379511/ https://www.amazon.com/dp/0201379511/
- brians 9y agoWell, there’s the problem right there. There was an individual responsible, meaning one set of hands unreviewed on a keyboard, for a change that could cause a global outage. If I’m that person responsible, I’m going to hire two staff, ask them each to write the command scripts, justify any differences, produce a consensus script for my review, and then implement it. That seems like the minimum level of responsible engineering. Lint tools, notices of novel commands, and other ornaments have a place too, but the core idea is this: The person with hands on keyboard is not the individual responsible for this error.
- jeremyjh 9y agoWhere did they say the change was not reviewed?
- jdwithit 9y agoIt's implied by their claim that there is a single responsible individual. If someone else reviewed it and said "yep, looks good, deploy the change" how is there one "individual responsible"? The idea of an isolated root cause or single human error in the failure of complex systems is bogus anyway. I'm a huge fan of the work in this area championed by John Allspaw [0]. [0]: https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...
- dboreham 9y agoIndeed. Someone runs the department that individual works in and allowed this kind of uncontrolled process. Someone is that person’s boss and should have been asking how well controlled are our processes, and so on. Turned Turtles all the way up..
- jlgaddis 9y agoThis could have been as simple as the individual accidentally "missing" a single line of a large, multi-line configuration when copy/pasting it into the router's console -- after all of the config review, etc., already occurred. I'm not sure what vendor's gear was in use in this particular case, but the configs for a BGP peering session are typically (as mentioned above) large, multi-line configurations. For example, here's the (slightly redacted) configuration for one single BGP session on one of my routers: neighbor 10.10.10.10 remote-as 65432 neighbor 10.10.10.10 transport connection-mode passive neighbor 10.10.10.10 description TO CUST FOO BAR INC ... neighbor 10.10.10.10 ebgp-multihop 3 neighbor 10.10.10.10 update-source Loopback0 neighbor 10.10.10.10 send-community neighbor 10.10.10.10 soft-reconfiguration inbound neighbor 10.10.10.10 prefix-list ACCEPTED-PREFIXES-AS65432 in neighbor 10.10.10.10 prefix-list ADVERTISED-PREFIXES-AS65432 out neighbor 10.10.10.10 password 7 0123456789ABCDEF0123456789ABCDEF neighbor 10.10.10.10 maximum-prefix 200 neighbor 10.10.10.10 default-originate That doesn't even include the associate prefix lists (or filter lists or route-maps or ...). All it would take is fat-fingering/typo'ing one of these lines or missing one to cause some very unintended effects. Since we don't know exactly what happened, it's easy to say "they should've done this" or "they didn't do that". In reality, however, we simply don't know what they did or didn't do. You've shown no evidence that they didn't do any of the things you mention and, in some cases, you can do all of that and still have things go wrong.
- scurvy 9y agoThese large companies need much more SDN in their operations like NTT GIN. I think GIN only has 50 people in their entire organization to run that whole business. They do it by investing heavily in software (and not building out their own fiber all over the place).
- userbinator 9y agoHere's my paraphrasing in more straightforward fashion, since I know at least some of us would like such notices to be more to-the-point and less business-speak-ish: Root Cause: Incorrect router configuration. Fix Action: Revert the configuration. Summary: Someone made the wrong settings on a router and made packets go the wrong way in parts of the US. We changed the settings back to what they were before. Corrective Actions: We'll try to find ways to avoid doing this again. We know who did it.
- rando444 9y agoSounds like "the individual responsible" forgot to set some communities on the peering session. Not exactly: https://bgpstream.com/event/112734 https://bgpstream.com/event/112734
- jlgaddis 9y agoYeah, I saw all of that (and more). Doesn't rule out what I said. I presume Comcast was advertising those (longer) prefixes to Level 3 to manage traffic flow and they shouldn't have been propoagated to other customers. To do that, you'd typically apply BGP communities (no-export or other, Level3-specific ones) to those prefixes. A lack of those communities on the prefixes would result in them getting propagated to other Level3 peers. When that happened, it would look exactly like and result in this route leak.
- rando444 9y agoAh, thank you very much for the explanation :)