6 ms·

One thing that was fascinating about the Rogers outage was on the wireless side: because "just" the core was down, the towers were still up. So mobile phones w

One thing that was fascinating about the Rogers outage was on the wireless side: because "just" the core was down, the towers were still up. So mobile phones would try to make a connection to the tower just enough to connect but not be able to do anything, like call 9-1-1 without trying to fail-over to other mobile networks. Devices showed zero bars, but field test mode would show some handshake succeeding. (The CTO was roaming out-of-country, had zero bars and thought nothing of it... how they had no idea an enterprise-risking update was scheduled, we'll never know) Supposedly you could remove your SIM card (who carries that tool doohickey with them at all times?), or disable that eSIM, but you'd have to know that you can do that. Unsure if you'd still be at the mercy of Rogers being the most powerful signal and still failing to get your 9-1-1 call through. Rogers claimed to have no ability to power down the towers without a truck-roll (which is how another aspect where widespread OOB could have come in handy). Various stories of radio stations (which Rogers also owns a lot of) not being able to connect the studio to the transmitter, so some tech went with an mp3 player to play pre-recorded "evergreen" content. Others just went off-air. https://www.theregister.com/2022/07/25/canadian_isp_rogers_outage/ https://www.theregister.com/2022/07/25/canadian_isp_rogers_o...

wannacboatmovie 2y ago
> Supposedly you could remove your SIM card (who carries that tool doohickey with them at all times?) In sane handsets (ones where the battery is still removable), that tool was and still is a fingernail, which most have on their person. I believe the innovation of the need for a special SIM eject tool was bestowed upon us by the same fruit company that gave us floppy and optical drives without manual eject buttons over 30 years ago.
- rzzzt 2y ago
  You could operate the ejection mechanism by hand both on optical and floppy disk drives with an uncurled paperclip (or a SIM card ejection tool were they to exist at that point in time). But I wouldn't ascribe the introduction of the motorized tray to the fruit company, it was the wordmark company: https://youtu.be/bujOWWTfzWQ https://youtu.be/bujOWWTfzWQ
- amluto 2y ago
  I have fond memories of the fruit company taking the next step: removing the floppy drive bezel from the drive and instead having the floppy drive slot be part of the overall chassis front panel. Of course, their mechanical tolerances were nothing like they are today, so if you looked at the computer crosseyed, the front panel would fail to align to the actual internal disk path, and ejecting the disk would cause it to get stuck behind the front chassis panel. One could rescue it by careful wiggling with a tool to guide the disk through the slot or by removing the entire front panel. Meanwhile “PCs” had a functional but ugly rectangular opening the size of the entire drive, and the drive had its own bezel, and imperfect alignment between drive and case looked a bit ugly but had no effect on function. (I admit I’m suspicious that Apple’s approach was a cost optimization, not an aesthetic optimization.)
pgraf 2y ago
Sounds like a problem that should be (rather easily) fixable in the Operating System, no? If the emergency call doesn’t go through, try the call over a different network. This would also mitigate problems we see from time to time where emergency calls don’t work because the uplink to the emergency call center was impacted either physically or by a bad software update.
- Scoundreller 2y ago
  Yes, seems to be a deficiency in the gsm spec as this affected most/all devices. I didn’t think this failure mode was even possible.
- sidewndr46 2y ago
  Very few phones have the calls handled by the OS. It usually is handled by something called the baseband, which is more like firmware
throw0101d 2y ago
> (The CTO was roaming out-of-country, had zero bars and thought nothing of it... how they had no idea an enterprise-risking update was scheduled, we'll never know) How does one know ahead of time if any particular change is "enterprise-risking"? It appeared to be a fairly routine set of changes that were going just fine: > The report summary says that in the weeks leading up to the outage, Rogers was undergoing a seven-phase process to upgrade its network. The outage occurred during the sixth's phase of the upgrade. * https://www.cbc.ca/news/politics/rogers-outage-human-error-system-deficiencies-1.7255641 https://www.cbc.ca/news/politics/rogers-outage-human-error-s... It turns out that they self-DoSed certain components: > Staff at Rogers caused the shutdown, the report says, by removing a control filter that directed information to its appropriate destination. > Without the filter in place, a flood of information was sent into Rogers' core network, overloading and crashing the system within minutes of the control filter being removed. * Ibid > In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network. * https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage#Postmortem https://en.wikipedia.org/wiki/2022_Rogers_Communications_out... > Rogers staff removed the Access Control ListFootnote 5 policy filter from the configuration of the distribution routers. This consequently resulted in a flood of IP routing information into the core network routers, which triggered the outage. The core network routers allow Rogers wireline and wireless customers to access services such as voice and data. The flood of IP routing data from the distribution routers into the core routers exceeded their capacity to process the informationFootnote 6. The core routers crashed within minutes from the time the policy filter was removed from the distribution routers configuration. * https://crtc.gc.ca/eng/publications/reports/xona2024.htm https://crtc.gc.ca/eng/publications/reports/xona2024.htm These types of things happen: > In October, Facebook suffered a historic outage when their automation software mistakenly withdrew the anycasted BGP routes handling its authoritative DNS rendering its services unusable. Last month, Cloudflare suffered a 30-minute outage when they pushed a configuration mistake in their automation software which also caused BGP routes to be withdrawn. * https://www.kentik.com/blog/a-deeper-dive-into-the-rogers-outage/ https://www.kentik.com/blog/a-deeper-dive-into-the-rogers-ou...
- immibis 2y ago
sidewndr46 2y ago
It is likely no one at the C-Level knew the update had any risk
- Scoundreller 2y ago
  it was "high risk", and then downgraded to "low risk" as this several week and phase long project proceeded without initial issue.

wannacboatmovie 2y ago

> Supposedly you could remove your SIM card (who carries that tool doohickey with them at all times?) In sane handsets (ones where the battery is still removable), that tool was and still is a fingernail, which most have on their person. I believe the innovation of the need for a special SIM eject tool was bestowed upon us by the same fruit company that gave us floppy and optical drives without manual eject buttons over 30 years ago.

rzzzt 2y ago
You could operate the ejection mechanism by hand both on optical and floppy disk drives with an uncurled paperclip (or a SIM card ejection tool were they to exist at that point in time). But I wouldn't ascribe the introduction of the motorized tray to the fruit company, it was the wordmark company: https://youtu.be/bujOWWTfzWQ https://youtu.be/bujOWWTfzWQ
amluto 2y ago
I have fond memories of the fruit company taking the next step: removing the floppy drive bezel from the drive and instead having the floppy drive slot be part of the overall chassis front panel. Of course, their mechanical tolerances were nothing like they are today, so if you looked at the computer crosseyed, the front panel would fail to align to the actual internal disk path, and ejecting the disk would cause it to get stuck behind the front chassis panel. One could rescue it by careful wiggling with a tool to guide the disk through the slot or by removing the entire front panel. Meanwhile “PCs” had a functional but ugly rectangular opening the size of the entire drive, and the drive had its own bezel, and imperfect alignment between drive and case looked a bit ugly but had no effect on function. (I admit I’m suspicious that Apple’s approach was a cost optimization, not an aesthetic optimization.)

pgraf 2y ago

Sounds like a problem that should be (rather easily) fixable in the Operating System, no? If the emergency call doesn’t go through, try the call over a different network. This would also mitigate problems we see from time to time where emergency calls don’t work because the uplink to the emergency call center was impacted either physically or by a bad software update.

Scoundreller 2y ago
Yes, seems to be a deficiency in the gsm spec as this affected most/all devices. I didn’t think this failure mode was even possible.
sidewndr46 2y ago
Very few phones have the calls handled by the OS. It usually is handled by something called the baseband, which is more like firmware

throw0101d 2y ago

> (The CTO was roaming out-of-country, had zero bars and thought nothing of it... how they had no idea an enterprise-risking update was scheduled, we'll never know) How does one know ahead of time if any particular change is "enterprise-risking"? It appeared to be a fairly routine set of changes that were going just fine: > The report summary says that in the weeks leading up to the outage, Rogers was undergoing a seven-phase process to upgrade its network. The outage occurred during the sixth's phase of the upgrade. * https://www.cbc.ca/news/politics/rogers-outage-human-error-system-deficiencies-1.7255641 https://www.cbc.ca/news/politics/rogers-outage-human-error-s... It turns out that they self-DoSed certain components: > Staff at Rogers caused the shutdown, the report says, by removing a control filter that directed information to its appropriate destination. > Without the filter in place, a flood of information was sent into Rogers' core network, overloading and crashing the system within minutes of the control filter being removed. * Ibid > In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network. * https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage#Postmortem https://en.wikipedia.org/wiki/2022_Rogers_Communications_out... > Rogers staff removed the Access Control ListFootnote 5 policy filter from the configuration of the distribution routers. This consequently resulted in a flood of IP routing information into the core network routers, which triggered the outage. The core network routers allow Rogers wireline and wireless customers to access services such as voice and data. The flood of IP routing data from the distribution routers into the core routers exceeded their capacity to process the informationFootnote 6. The core routers crashed within minutes from the time the policy filter was removed from the distribution routers configuration. * https://crtc.gc.ca/eng/publications/reports/xona2024.htm https://crtc.gc.ca/eng/publications/reports/xona2024.htm These types of things happen: > In October, Facebook suffered a historic outage when their automation software mistakenly withdrew the anycasted BGP routes handling its authoritative DNS rendering its services unusable. Last month, Cloudflare suffered a 30-minute outage when they pushed a configuration mistake in their automation software which also caused BGP routes to be withdrawn. * https://www.kentik.com/blog/a-deeper-dive-into-the-rogers-outage/ https://www.kentik.com/blog/a-deeper-dive-into-the-rogers-ou...

immibis 2y ago

sidewndr46 2y ago

It is likely no one at the C-Level knew the update had any risk

Scoundreller 2y ago
it was "high risk", and then downgraded to "low risk" as this several week and phase long project proceeded without initial issue.