Copied from the docx - pruning the bits leading into and out of hidden content. The Outage The configuration change deleted a routing filter and allowed for al

Copied from the docx - pruning the bits leading into and out of hidden content. The Outage The configuration change deleted a routing filter and allowed for all possible routes to the Internet to pass through the routers. As a result, the routers immediately began propagating abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their capacity levels and were then unable to route traffic, causing the common core network to stop processing traffic. As a result, the Rogers network lost connectivity to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers. The Recovery To resolve the outage, the Rogers Network Team assembled in and around our Network Operations Centre (“NOC”) and re-established access to the IP network. They then started the detailed process of determining the source of the outage, leading to identifying the three Distribution Routers as the cause. Once determined, the team then began the process of restarting all the Internet Gateway, Core and Distribution Routers in a controlled manner to establish connectivity to our wireless (including 9-1-1), enterprise and cable networks which deliver voice, video and data connectivity to our customers. Service was slowly restored, starting in the afternoon and continuing over the evening. Although Rogers continued to experience some instability issues over the weekend that did impact some customers, the network had effectively recovered by Friday night. what was the root cause of the outage (including what processes, procedures or safeguards failed to prevent the outage, such as planned redundancy or patch upgrade validation procedures); Like many large Telecommunications Services Providers (“TSPs”), Rogers uses a common core network, essentially one IP network infrastructure, that supports all wireless, wireline and enterprise services. The common core is the brain of the network that receives, processes, transmits and connects all Internet, voice, data and TV traffic for our customers. Again, similar to other TSPs around the world, Rogers uses a mixed vendor core network consisting of IP routing equipment from multiple tier one manufacturers. This is a common industry practice as different manufacturers have different strengths in routing equipment for Internet gateway, core and distribution routing. Specifically, the two IP routing vendors Rogers uses have their own design and approaches to managing routing traffic and to protect their equipment from being overwhelmed. In the Rogers network, one IP routing manufacturer uses a design that limits the number of routes that are presented by the Distribution Routers to the core routers. The other IP routing vendor relies on controls at its core routers. The impact of these differences in equipment design and protocols are at the heart of the outage that Rogers experienced. The Rogers outage on July 8, 2022, was unprecedented. As discussed in the previous response, it resulted during a routing configuration change to three Distribution Routers in our common core network. Unfortunately, the configuration change deleted a routing filter and allowed for all possible routes to the Internet to be distributed; the routers then propagated abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their memory and processing capacity and were then unable to route and process traffic, causing the common core network to shut down. As a result, the Rogers network lost connectivity internally and to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers. how did the outage impact Rogers’ own staff and their ability to determine the cause of the outage and restore services; At the early stage of the outage, many Rogers’ network employees were impacted and could not connect to our IT and network systems. This impeded initial triage and restoration efforts as teams needed to travel to centralized locations where management network access was established. To complicate matters further, the loss of access to our VPN system to our core network nodes affected our timely ability to begin identifying the trouble and, hence, delayed the restoral efforts. Despite these hurdles, our preestablished business continuity plans enabled staff to converge at specific rally points. Those equipped with emergency SIMs on alternate carriers that enabled our teams to switch carriers and assist in the initial coordination efforts. Further, we rapidly relocated our employees to two of our main offices in the GTA (# #). The critical network employees were able to gain physical access to our network equipment. Other essential employees were able to use alternate SIM cards, as per our “Alternate Carrier SIM Card Program” (described in Rogers(CRTC)11July2022-1.xiii below). Other employees were able to work from # #. Together, these groups were able to establish the necessary team to identify the cause of the outage and recover the network. what contingencies, if any, did Rogers have in place to ensure that its staff could communicate with each other particularly in the early hours of the outage; On July 17th, 2015, the Canadian Telecom Resiliency Working Group (“CTRWG”), formerly called Canadian Telecom Emergency Preparedness Association, established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity. This is to allow TSPs to communicate within their organizations in the event of loss of their respective networks. Bell, Rogers and TELUS took the lead to provide SIM cards to all CTRWG members. # #. When it was realized that Rogers entire core network was offline, employees started swapping out our Rogers SIM cards with our alternate carrier SIM Cards. This previously established contingency plan allowed us to begin communicating within our organization in the early hours of the outage and to start restoring services.