3 ms·
The implication in the article is that this happened during a scheduled change window (the 6th change in a set of 8), but it's also just the strong implication.
by motohagiography 4y ago
The implication in the article is that this happened during a scheduled change window (the 6th change in a set of 8), but it's also just the strong implication. I'm still of the apopleptically hawkish view it was an attack, and equivocal language just reinforces that belief, but charitably I'm less and less attached to that view.
Let's say this was a mistake. When they tested the config in a pre-prod environment, they would have noticed the redistribution of those routes. If they were pasting the config into a router instead of sending it via scp, maybe there's a mouse buffer/paste error, but you don't do that on a core device anymore. Judging by the response time, they weren't doing this onsite at the console either and their OOB access didn't work because it probably used the bunged cell network. Not negligent, but certainly risky.
I have personally made routing mistakes that caused peer resets, like mistyping a prefix on a static route that overrode the route to a peer interface over an exchange, and I have also overloaded circuits because of misguided path prepends. I haven't had enable privs on a core router in probably 20 years, but this wholesale redistribution is caused when you take one peers routes and announce them to another with you as the origin, known as "announcing the internet." As I remember, this isn't a single filter.
Not to muddy it, and it doesn't help my attack theory much, and please call this out as bullshit with a correction if you recognize it to be as such, but someone who used to work there told me something to the effect of their architecture had combined some of the older ASNs from previous Rogers acquisitions and that was part of the problem, whereby they essentially have the old AS's speaking iBGP to each other internally within Rogers, with only the single Rogers AS facing the internet as a peer, sort of the way the some of the old mega ASNs. It implies there were different policy domains with different levels of maturity inside the main AS, and those internal peers don't filter routes they recieved from the core. When you have a network that has grown organically over the last couple of decades, this happening at least once is inevitable. Legacy policies just redistributed what they were sent, and between flapping, dampening penalties, transiting and backhauling mobile voice traffic over IP networks, and backup and OOB reachability, this one of those perfect storms where for years the network both worked magically, and then one day failed just as magically.
Undoubtedly, there are probably some senior engineers sitting around saying, "I told you this would happen, and we've been trying to tell you for years," but when you know this stuff, it's your responsibility to also be persuasive.
I was very attached to the attack theory for a bunch of reasons, but the conceptual chain of events above makes more sense to me, and it is a lot more like a rubber gasket on a fuel tank expanding too rapidly an unexpectedly cool day. Obvious in hindsight and simple to explain, but invisible up until the moment of ignition.