10 ms·
A coding error caused Rogers outage that left millions without service
- happyopossum 4y agoIt's the Network Law - it's BGP. It's always BGP. Even when you want to blame the firewall, it's probably the firewall's BGP config. [source - 20 years of NetSec and NetEng experience]
- jonny_eh 4y agobut sometimes it's DNS [1,2,3]. [1] https://techcrunch.com/2021/07/22/a-dns-outage-just-took-down-a-good-chunk-of-the-internet/ https://techcrunch.com/2021/07/22/a-dns-outage-just-took-dow... [2] https://techcrunch.com/2020/07/17/cloudflare-dns-goes-down-taking-a-large-piece-of-the-internet-with-it/ https://techcrunch.com/2020/07/17/cloudflare-dns-goes-down-t... [3] https://techcrunch.com/2021/02/12/notion-outage-dns-domain-issues/ https://techcrunch.com/2021/02/12/notion-outage-dns-domain-i...
- happyopossum 4y agoBah - your 3rd example is just a single site, and the second was actually caused by BGP. I'm not convinced BGP wasn't the problem with the first either ;-)
- FranksTV 4y agoI have it on good information that BGP was the second gunman on the grassy knoll at the Kennedy Assassination.
- addingnumbers 4y agoBGP is something that makes everything more resilient when it's achieving its goals, not like an unknown assassin. Large-scale BGP harm almost always comes from a trusted, good-willed peer making an honest mistake.
- Victerius 4y agoThe key bit: > But, at 4:43 a.m. on July 8, a piece of code was introduced that deleted a routing filter. In telecom networks, packets of data are guided and directed by devices called routers, and filters prevent those routers from becoming overwhelmed, by limiting the number of possible routes that are presented to them. > Deleting the filter caused all possible routes to the internet to pass through the routers, resulting in several of the devices exceeding their memory and processing capacities. This caused the core network to shut down. Lesson no 1: Do not design your system to have a single point of failure.
- erentz 4y ago> But, in the early hours, the company’s technicians had not yet pinpointed the cause of the catastrophe. Rogers apparently considered the possibility that its networks had been attacked by cybercriminals. I mean, if you just pushed a config change and the whole network goes kaput, take a look at the config change before you start suspecting hackers.
- pitched 4y agoI heard that the teams were having trouble communicating with each other and so the ones who pushed the config might not have been the ones looking for hackers. This is why some hospitals still use the old pager systems to contact people in the city. One hospital-owned antenna on a battery can coordinate a lot of people. I don’t know what the equivalent to that would be in this case though.
- gwen-shapira 4y ago"Although Bell and Telus offered to help, Rogers quickly determined that it would not be able to transfer its customers to its rivals’ networks because certain elements of the Rogers network, such as its centralized user database, were inaccessible as a result of the outage." It sounds like their control/management plane (with the user's database) was dependent on their data plane. So a data plane outage was more challenging to mitigate than it should have been in a decoupled architecture. Good lesson for any architecture.
- mabbo 4y agoBoth my home internet and mobile are through Rogers (for the time being). I had no access to the internet for 15 hours, 6am to 9pm. Couldn't do my job as a remote developer. And all through the day I kept thinking to myself "I bet someone pushed an update to prod, causing this. And I'm glad that this time it wasn't me."
- sammy2244 4y ago
- amatecha 4y agoYou can see Rogers' own report (with some redactions) as provided to the CRTC. See the doc linked under the first (2022-07-22) heading here: https://crtc.gc.ca/otf/eng/2022/8000/c12-202203868.htm https://crtc.gc.ca/otf/eng/2022/8000/c12-202203868.htm
- herpderperator 4y ago> CKOT-FM and CJDL-FM (Tillsonburg) were not able to air live programming from 4:45AM EDT to 10:58PM on the same day. During that time, evergreen programming was aired from an MP3 player at the base of the transmitter until an Internet connection was re-established between the studio and the transmitter site I'm impressed they went to these lengths to get services up!
- chx 4y agoBotching a prod update -- that happens to the best of us. And sorry to say I doubt Rogers would attract the best of us but I digress. But only kind of digress. When the human waste hits the circular cooler device, that's when we find out how well an organization is built and managed. Who is surprised here it took them this long to recover? The necessary no-blame culture, temporary removal of decision barriers, all-hands-on-deck to get back to normal, all the things the lucky of us take for granted are guaranteed to be missing here.
- Victerius 4y ago> And sorry to say I doubt Rogers would attract the best of us but I digress "The typical Rogers Communications Systems Administrator salary is $67,329" (https://www.glassdoor.ca/Salary/Rogers-Communications-Systems-Administrator-Salaries-E3783_D_KO22,43.htm https://www.glassdoor.ca/Salary/Rogers-Communications-System...) "The average system administrator salary in Canada is $71,981 per year or $36.91 per hour. Entry-level positions start at $61,425 per year, while most experienced workers make up to $94,400 per year." (https://ca.talent.com/salary?job=system+administrator https://ca.talent.com/salary?job=system+administrator)
- doubled112 4y agoMaybe related are the exceptions to labour laws while you make your 67K a year https://www.ontario.ca/document/industries-and-jobs-exemptions-or-special-rules/government-employees-and-professionals#section-3 https://www.ontario.ca/document/industries-and-jobs-exemptio... But I'm sure Rogers would never abuse any of those.
- eru 4y agoThat's not a lot of money for software folks. Especially folks with access to the American labour market.
- amscanne 4y agoNote that $1 CAD is also $0.80 USD, which is often omitted but meaningful when comparing pay just over the border.
- boplicity 4y agoCan we switch this to the original publisher? I assume it was republished without consent. https://www.theglobeandmail.com/business/article-how-a-coding-error-caused-rogers-outage-that-left-millions-without/ https://www.theglobeandmail.com/business/article-how-a-codin...
- j4_hnews 4y agoPaywall
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- anewpersonality 4y agoHorrible! Why doesn't Canada have Comcast or any US ISP?
- wvenable 4y agoCanada limits foreign ownership of telecommunications.
- kitten_mittens_ 4y agoIs America particularly foreign? Looking at the EU as a model, it makes sense for the two anglophone countries to have a closer union.
- bpye 4y agoWhilst I agree in abstract, I imagine closer ties with the US would be hard to stomach for many in Canada for a while given the last presidency…
- Teever 4y agoCanada is effectively a vassal state of The US. any closer union between the two would result in Canada becoming a full fledged American state. As a Canadian I feel that the current arrangement gives Canada most of the perks of America with very few of the downsides.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- stevenjgarner 4y agoI would strongly argue that Comcast would be a huge step backwards even for the Canadians. "Comcast Is America's Most Hated Company" [0]. Roger's 10.2 million wireless subscribers lost connectivity for maybe a couple of days. Comcast has been failing at servicing more than 31 million broadband internet subscribers for decades. Comcast has also done everything they can to destroy net neutrality. The sooner Comcast is a thing of the past, the better. [0] https://www.pcmag.com/news/comcast-is-americas-most-hated-company https://www.pcmag.com/news/comcast-is-americas-most-hated-co...
- thrusong 4y agoI think the most shocking thing about this outage was that the country's national payment system, Interac, was down most of the day. This is an interbank network which is essentially a cooperative operated by the country's major financial institutions. Who would have thought Interac would've only had one ISP? Thankfully, they've stated they're adding at least one additional service provider. I drove from Winnipeg to Calgary that day (a 12 hour drive) and had to stop in several small towns with TDs to pull cash to fill up. It was quite bizarre checking Twitter all day to see if it was back yet and seeing it was still down.
- notatoad 4y agothe story i heard, which i'm going to irresponsibly repeat without knowing the truth of, is that Interac did have a backup ISP but it was one of the smaller ISPs who buys access to rogers fibre, so their backup went down because rogers was down.
- throwawaysleep 4y agoCanadians are very risk averse people except when it comes to preparedness. They avoid change if at all possible.
- FpUser 4y agoNot only that. They often are pathologically incapable of giving straight answers (I am talking about business dealings). It was / still is a major pain in the ass for me as I am very straight guy and have no desire to decipher what their evasive answers really mean. There are some rare exceptions though. Dealing with XXX Motors Canada almost shocked me in comparison because when their guy said they would do something it was like set in stone. They went out of their way to meet my needs. Dealing with Americans was a pleasure for me as well. They say A and it means A and it will be done. Disclaimer - I am Canadian but was born in USSR, came here some 30 years ago.
- VancouverMan 4y agoWho/what do you consider to be a "Canadian" in 2022? In Toronto and the surrounding cities, for example, about half of the population are foreign-born. A significant proportion of the population of many other Canadian urban areas are foreign-born. About 20% of the overall Canadian population are foreign-born. Many of these people are from cultures that are quite different to anything resembling "traditional" Canadian culture. Acquiring Canadian citizenship later in life doesn't necessarily change a person's values, attitudes, and so on. A lot of Canadian-born individuals have one or both parents who are foreign-born, which also can have an impact on one's values, attitudes, and behaviour. Even among those with multi-generational ties to Canada, there were already significant cultural/values/attitude/behavioural differences among the various groupings. Ultimately, in any given interaction with somebody in Canada today, there's a good chance you're dealing with somebody whose ties to "traditional" Canadian culture are limited, or even non-existent. Maybe some kind of a relatively cohesive "Canadian" culture or identity existed at one point, several decades ago, but I don't think that's the case any longer, especially in the urban areas.
- somerandomqaguy 4y agoThere's a reddit thread going on that was discussing an interesting edge case in regards to 911 and the outage. In Canada 911 service is accessible without a SIM card; if your phone doesn't have a SIM, any cell tower should in theory still accept and route the call as normal. However in Roger's case, because the cell towers and their authentication mechanisms were still operational, any 911 call with a Roger's SIM card in it would route through Roger's network and only Roger's network; the one that couldn't service any calls. In essence, your 911 service was completely cut off. The workaround is to pull the SIM card prior to making the 911 call, but it leaves an interesting question about what you're supposed to do in an eSIM world where pulling the SIM is not possible but you're again in this kind of situation. To be clear I don't know if this is a real problem or not, but it is an interesting thought either way.
- throwaway235879 4y agoCopied from the docx - pruning the bits leading into and out of hidden content. The Outage The configuration change deleted a routing filter and allowed for all possible routes to the Internet to pass through the routers. As a result, the routers immediately began propagating abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their capacity levels and were then unable to route traffic, causing the common core network to stop processing traffic. As a result, the Rogers network lost connectivity to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers. The Recovery To resolve the outage, the Rogers Network Team assembled in and around our Network Operations Centre (“NOC”) and re-established access to the IP network. They then started the detailed process of determining the source of the outage, leading to identifying the three Distribution Routers as the cause. Once determined, the team then began the process of restarting all the Internet Gateway, Core and Distribution Routers in a controlled manner to establish connectivity to our wireless (including 9-1-1), enterprise and cable networks which deliver voice, video and data connectivity to our customers. Service was slowly restored, starting in the afternoon and continuing over the evening. Although Rogers continued to experience some instability issues over the weekend that did impact some customers, the network had effectively recovered by Friday night. what was the root cause of the outage (including what processes, procedures or safeguards failed to prevent the outage, such as planned redundancy or patch upgrade validation procedures); Like many large Telecommunications Services Providers (“TSPs”), Rogers uses a common core network, essentially one IP network infrastructure, that supports all wireless, wireline and enterprise services. The common core is the brain of the network that receives, processes, transmits and connects all Internet, voice, data and TV traffic for our customers. Again, similar to other TSPs around the world, Rogers uses a mixed vendor core network consisting of IP routing equipment from multiple tier one manufacturers. This is a common industry practice as different manufacturers have different strengths in routing equipment for Internet gateway, core and distribution routing. Specifically, the two IP routing vendors Rogers uses have their own design and approaches to managing routing traffic and to protect their equipment from being overwhelmed. In the Rogers network, one IP routing manufacturer uses a design that limits the number of routes that are presented by the Distribution Routers to the core routers. The other IP routing vendor relies on controls at its core routers. The impact of these differences in equipment design and protocols are at the heart of the outage that Rogers experienced. The Rogers outage on July 8, 2022, was unprecedented. As discussed in the previous response, it resulted during a routing configuration change to three Distribution Routers in our common core network. Unfortunately, the configuration change deleted a routing filter and allowed for all possible routes to the Internet to be distributed; the routers then propagated abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their memory and processing capacity and were then unable to route and process traffic, causing the common core network to shut down. As a result, the Rogers network lost connectivity internally and to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers. how did the outage impact Rogers’ own staff and their ability to determine the cause of the outage and restore services; At the early stage of the outage, many Rogers’ network employees were impacted and could not connect to our IT and network systems. This impeded initial triage and restoration efforts as teams needed to travel to centralized locations where management network access was established. To complicate matters further, the loss of access to our VPN system to our core network nodes affected our timely ability to begin identifying the trouble and, hence, delayed the restoral efforts. Despite these hurdles, our preestablished business continuity plans enabled staff to converge at specific rally points. Those equipped with emergency SIMs on alternate carriers that enabled our teams to switch carriers and assist in the initial coordination efforts. Further, we rapidly relocated our employees to two of our main offices in the GTA (# #). The critical network employees were able to gain physical access to our network equipment. Other essential employees were able to use alternate SIM cards, as per our “Alternate Carrier SIM Card Program” (described in Rogers(CRTC)11July2022-1.xiii below). Other employees were able to work from # #. Together, these groups were able to establish the necessary team to identify the cause of the outage and recover the network. what contingencies, if any, did Rogers have in place to ensure that its staff could communicate with each other particularly in the early hours of the outage; On July 17th, 2015, the Canadian Telecom Resiliency Working Group (“CTRWG”), formerly called Canadian Telecom Emergency Preparedness Association, established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity. This is to allow TSPs to communicate within their organizations in the event of loss of their respective networks. Bell, Rogers and TELUS took the lead to provide SIM cards to all CTRWG members. # #. When it was realized that Rogers entire core network was offline, employees started swapping out our Rogers SIM cards with our alternate carrier SIM Cards. This previously established contingency plan allowed us to begin communicating within our organization in the early hours of the outage and to start restoring services.
- motohagiography 4y agoThe implication in the article is that this happened during a scheduled change window (the 6th change in a set of 8), but it's also just the strong implication. I'm still of the apopleptically hawkish view it was an attack, and equivocal language just reinforces that belief, but charitably I'm less and less attached to that view. Let's say this was a mistake. When they tested the config in a pre-prod environment, they would have noticed the redistribution of those routes. If they were pasting the config into a router instead of sending it via scp, maybe there's a mouse buffer/paste error, but you don't do that on a core device anymore. Judging by the response time, they weren't doing this onsite at the console either and their OOB access didn't work because it probably used the bunged cell network. Not negligent, but certainly risky. I have personally made routing mistakes that caused peer resets, like mistyping a prefix on a static route that overrode the route to a peer interface over an exchange, and I have also overloaded circuits because of misguided path prepends. I haven't had enable privs on a core router in probably 20 years, but this wholesale redistribution is caused when you take one peers routes and announce them to another with you as the origin, known as "announcing the internet." As I remember, this isn't a single filter. Not to muddy it, and it doesn't help my attack theory much, and please call this out as bullshit with a correction if you recognize it to be as such, but someone who used to work there told me something to the effect of their architecture had combined some of the older ASNs from previous Rogers acquisitions and that was part of the problem, whereby they essentially have the old AS's speaking iBGP to each other internally within Rogers, with only the single Rogers AS facing the internet as a peer, sort of the way the some of the old mega ASNs. It implies there were different policy domains with different levels of maturity inside the main AS, and those internal peers don't filter routes they recieved from the core. When you have a network that has grown organically over the last couple of decades, this happening at least once is inevitable. Legacy policies just redistributed what they were sent, and between flapping, dampening penalties, transiting and backhauling mobile voice traffic over IP networks, and backup and OOB reachability, this one of those perfect storms where for years the network both worked magically, and then one day failed just as magically. Undoubtedly, there are probably some senior engineers sitting around saying, "I told you this would happen, and we've been trying to tell you for years," but when you know this stuff, it's your responsibility to also be persuasive. I was very attached to the attack theory for a bunch of reasons, but the conceptual chain of events above makes more sense to me, and it is a lot more like a rubber gasket on a fuel tank expanding too rapidly an unexpectedly cool day. Obvious in hindsight and simple to explain, but invisible up until the moment of ignition.
- 102030485868 4y agoour federal industry minister and rogers execs answer some questions: https://www.cpac.ca/episode?id=84243ade-0b57-45f6-8144-c4d62e041daf https://www.cpac.ca/episode?id=84243ade-0b57-45f6-8144-c4d62... i feel like there's a lot of passing the buck in the video
- prvc 4y agoThe opposite--- they are effectively covering for each other. They (the orgs, more generally) are complicit in allowing this dysfunctional rent-seeking monopoly to continue and expand. This was a political failure long before before the "coding error" took place.
- j_not_j 4y ago"Canadian telecoms resiliency working group" It is ineffective. Their remit includes "architecture" but the Rogers network architecture is a clear failure. There was no political will to evaluate the results of such a group.
- 0xbadc0de5 4y agoRogers is also known to historically have had a somewhat 'flatter' and less mature network architecture as compared to their peers, Telus and Bell. Likely owing to their roots as a cable television provider as opposed to landline telephony - ie: dumb pipe vs. less-dumb pipe. Lots of things went wrong here, to name a few: - lack of rigour in their process that allowed a critical modification without understanding its effects in context - lack of monitoring and metrics to alert them immediately of the problem - lack of emergency rollback capability to revert back to the known-good config - lack of independent business-continuity comms channels. if you're a Rogers CTO or senior responsible adult and you don't have a backup Telus or Bell SIM card (and vice versa), you learned why you need one.
- Scoundreller 4y agoAlso lack of downtime procedures. They seemed to have assumed that nothing could go wrong when you should assume that everything could go wrong. From their own documentation, in mid-2015 (kinda late, but better than never), the providers issued eachother eachother's SIMs to use in these situations. Who qualified exactly is a mystery.
- interestica 4y agoIt's funny that all of Canada's internet providers have marketed all their bundling of services as an advantage (bundle and save money!) I'd love for someone to spoof the ads to show that all you're doing is helping create a single point of failure instead of redundancy.
- perryizgr8 4y ago> Deleting the filter caused all possible routes to the internet to pass through the routers, resulting in several of the devices exceeding their memory and processing capacities. This caused the core network to shut down. I don't get this. If a switch can handle only x number of routes, it won't get "overwhelmed" if you try to add more, it would just refuse to accept any more routes. Why would the network go down completely? These things are designed with extremely robust software, you simply start getting error logs saying that the device has run out of memory, but the operation continues. Most of the heavy lifting is done in hardware chips, so the software just can't program new routes, but the routing itself should continue unaffected.
- citrin_ru 4y ago> it won't get "overwhelmed" if you try to add more, it would just refuse to accept any more routes It not how it worked in old Cisco routers - when more and more routes are added at some point it runs out of fast memory (TCAM) and becomes either too slow or not operational at all. Unless you manually configure route number limit it would install more routes into FIB than it can handle. So one have to monitor TCAM utilization and based on this configure various limits to prevent its exhaustion. May be TCAM exhaustion is still a problem even in modern Cisco equipment - I've moved to another field and don't know much about it.
- perryizgr8 4y agoI've personally tested this scenario on relatively newer Cisco devices and it literally just prints a message on the console to the tune of "TCAM exhausted, refusing to add route A.B.C.D/X" and continues working without problems. I think there is a config on some models to install the route in the slow path, but by default it just doesn't use the route.
- rexreed 4y agoThe fact that the CTO was replaced so quickly is not just a sign of scapegoating (which I am sure it is), but also the truth is that so many of these C-level tech executives are utterly incompetent at their jobs, and it takes a crisis for many to realize it. The C-levels at so many large companies are akin to high-level political appointments at government agencies. It's a good thing the lower levels know what's going on. In any crisis, the know-nothing C-levels are first to be ejected, where they can go look for their next appointment.
- cameldrv 4y agoAt the level of CTO of a major telecom, I don't his firing as scapegoating, it's accountability. The buck has to stop with someone, and the CTO is responsible for the technical infrastructure of the company.
- ethbr0 4y agoI think parent's point was that absent a crisis the same (incompent) CTO would still be in his job. So if that's what it takes to boot a CTO, who will then land in another executive position, then it's a poor metric for ensuring technology orgs (in old companies) are well-run.
- cameldrv 4y agoIf the current CTO is fired, the next CTO will pay more attention to reliability. This was clearly an organizational failure. Blame free retrospectives have a purpose, but if there is no accountability at the top level, there can simply be repeated failures regardless of the "lessons learned."
- _8j50 4y agoBasically they configured all their routers to install all the routes in the GRT? Seems like an innocent and unavoidable mistake. My opinion: You know how Juniper routers require you to commit a change? That should be enforced by a dedicated firmware and there should be multiple rollback confirmations (optional). So you commit a change and confirm, an hour later you have to confirm again else a hard reset and rollback takes place with a protocol in place where the rollback can be delayed when other devices in connected segments are also doing a rollback.
- buscoquadnary 4y agoFeels like an automatic rollback one hour after a change would cause more problems than it prevents
- _8j50 4y agoOnly if you have incompetent people. Everyone overlooks things but forgetting to commit a change is on you. Ideally the rollback would revert the network state.
- throwaway787544 4y agoAvoidable mistakes are still mistakes, and mistakes happen, even when they shouldn't. There is no one thing you could do to avoid catastrophic failure. The best thing you can do is break your system up entirely and simply never do any change which can affect the entire system. Which is still very hard to do. Imagine you have a route that crosses all of Canada. For each end of Canada you want at least 3 separate networks that can each carry 1/2 of the traffic. On failure of one network, somebody has to take that failed network's traffic and send half to each of the two remaining networks. But if somebody fucks up and sends that traffic to one of the networks, that network will be flooded and inoperable, and the first network also will be inoperable, leaving only one network with 1/3 the country's traffic online. Whatever customers designed their own stuff to depend on the first or second network is also screwed. This is just one of a million different scenarios that needs complex controls to prevent a (mostly) catastrophic failure. There is nobody using formal proofs to show the design makes sense, so it's just humans winging it and hoping their contingencies work out.