4 ms·

> (The CTO was roaming out-of-country, had zero bars and thought nothing of it... how they had no idea an enterprise-risking update was scheduled, we'll never k

> (The CTO was roaming out-of-country, had zero bars and thought nothing of it... how they had no idea an enterprise-risking update was scheduled, we'll never know) How does one know ahead of time if any particular change is "enterprise-risking"? It appeared to be a fairly routine set of changes that were going just fine: > The report summary says that in the weeks leading up to the outage, Rogers was undergoing a seven-phase process to upgrade its network. The outage occurred during the sixth's phase of the upgrade. * https://www.cbc.ca/news/politics/rogers-outage-human-error-system-deficiencies-1.7255641 https://www.cbc.ca/news/politics/rogers-outage-human-error-s... It turns out that they self-DoSed certain components: > Staff at Rogers caused the shutdown, the report says, by removing a control filter that directed information to its appropriate destination. > Without the filter in place, a flood of information was sent into Rogers' core network, overloading and crashing the system within minutes of the control filter being removed. * Ibid > In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network. * https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage#Postmortem https://en.wikipedia.org/wiki/2022_Rogers_Communications_out... > Rogers staff removed the Access Control ListFootnote 5 policy filter from the configuration of the distribution routers. This consequently resulted in a flood of IP routing information into the core network routers, which triggered the outage. The core network routers allow Rogers wireline and wireless customers to access services such as voice and data. The flood of IP routing data from the distribution routers into the core routers exceeded their capacity to process the informationFootnote 6. The core routers crashed within minutes from the time the policy filter was removed from the distribution routers configuration. * https://crtc.gc.ca/eng/publications/reports/xona2024.htm https://crtc.gc.ca/eng/publications/reports/xona2024.htm These types of things happen: > In October, Facebook suffered a historic outage when their automation software mistakenly withdrew the anycasted BGP routes handling its authoritative DNS rendering its services unusable. Last month, Cloudflare suffered a 30-minute outage when they pushed a configuration mistake in their automation software which also caused BGP routes to be withdrawn. * https://www.kentik.com/blog/a-deeper-dive-into-the-rogers-outage/ https://www.kentik.com/blog/a-deeper-dive-into-the-rogers-ou...

immibis 2y ago
BTW: anyone who wants to really experience how complex internet routing is, go and join DN42 (https://dn42.dev https://dn42.dev). This is a fake internet built on a network of VPN tunnels and using the same routing systems. As long as you're just acting as a leaf node, it's pretty straightforward. If you want to attach to the network at multiple points and not just VPN them all to the same place, now you have to design a network just like an ISP would, with IGP and so on.
vitus 2y ago
Router config changes are simultaneously very commonplace and incredibly risky. I've seen outages caused by a single bad router advertisement that caused global crashes due to route poisoning interacting with a vendor bug. RPKI enforcement caused massive congestion on transit links. Route leaks have DoSed entire countries (https://www.internetsociety.org/blog/2017/08/google-leaked-prefixes-knocked-japan-off-internet/ https://www.internetsociety.org/blog/2017/08/google-leaked-p...). Even something as simple as a peer removing rules for clearing ToS bits resulted in a month of 20+ engineers trying to figure out why an engineering director was sporadically being throttled to ~200kbps when trying to access Google properties. Running a large-scale production network is hard. edit: in case it is not obvious: I agree entirely with you -- the routine config changes that do risk the enterprise are often very hard to identify ahead of time.
- Scoundreller 2y ago
  the report stated: https://crtc.gc.ca/eng/publications/reports/xona2024.htm https://crtc.gc.ca/eng/publications/reports/xona2024.htm > "this configuration change was the sixth phase of a seven-phase network upgrade process that had begun weeks earlier. Before this sixth phase configuration update, the previous configuration updates were completed successfully without any issue. Rogers had initially assessed the risk of this seven-phased process as “High.” > However, as changes in prior phases were completed successfully, the risk assessment algorithm downgraded the risk level for the sixth phase of the configuration change to “Low” risk" > Downgrading the risk assessment to “Low” for changing the Access Control List filter in a routing policy contravenes industry norms, which require high scrutiny for such configuration changes, including laboratory testing before deploying in the production network. Overall, the lack of detail of the (regulator forced) post-mortem makes it impossible for the public to decide. It's a Canadian telecom: They'll release detail when it makes them look good, and hide it if it makes them look bad.

immibis 2y ago

BTW: anyone who wants to really experience how complex internet routing is, go and join DN42 (https://dn42.dev https://dn42.dev). This is a fake internet built on a network of VPN tunnels and using the same routing systems. As long as you're just acting as a leaf node, it's pretty straightforward. If you want to attach to the network at multiple points and not just VPN them all to the same place, now you have to design a network just like an ISP would, with IGP and so on.

vitus 2y ago

Router config changes are simultaneously very commonplace and incredibly risky. I've seen outages caused by a single bad router advertisement that caused global crashes due to route poisoning interacting with a vendor bug. RPKI enforcement caused massive congestion on transit links. Route leaks have DoSed entire countries (https://www.internetsociety.org/blog/2017/08/google-leaked-prefixes-knocked-japan-off-internet/ https://www.internetsociety.org/blog/2017/08/google-leaked-p...). Even something as simple as a peer removing rules for clearing ToS bits resulted in a month of 20+ engineers trying to figure out why an engineering director was sporadically being throttled to ~200kbps when trying to access Google properties. Running a large-scale production network is hard. edit: in case it is not obvious: I agree entirely with you -- the routine config changes that do risk the enterprise are often very hard to identify ahead of time.

Scoundreller 2y ago
the report stated: https://crtc.gc.ca/eng/publications/reports/xona2024.htm https://crtc.gc.ca/eng/publications/reports/xona2024.htm > "this configuration change was the sixth phase of a seven-phase network upgrade process that had begun weeks earlier. Before this sixth phase configuration update, the previous configuration updates were completed successfully without any issue. Rogers had initially assessed the risk of this seven-phased process as “High.” > However, as changes in prior phases were completed successfully, the risk assessment algorithm downgraded the risk level for the sixth phase of the configuration change to “Low” risk" > Downgrading the risk assessment to “Low” for changing the Access Control List filter in a routing policy contravenes industry norms, which require high scrutiny for such configuration changes, including laboratory testing before deploying in the production network. Overall, the lack of detail of the (regulator forced) post-mortem makes it impossible for the public to decide. It's a Canadian telecom: They'll release detail when it makes them look good, and hide it if it makes them look bad.