36 ms·
> I just always imagined the GSM spec was so resilient that you could always make a 9-1-1 call if a working network was available but this outage proved that wr
by kevin_nisbet 3y ago
> I just always imagined the GSM spec was so resilient that you could always make a 9-1-1 call if a working network was available but this outage proved that wrong.
As I recall it is slightly more nuanced than this and was particular to the failure mode, and has a couple of different things aligning to create the failure mode.
If you're phone is just blank, no sim card. To make an emergency call, it has to just start scanning all the supported frequencies. This is very slow, tune radio, wait for the scheduled information block that described the network on the radio protocol. See if it has the emergency services bit enabled. If not, tune to next frequency and try again. I used to remember all the timers, but almost a decade later I can't remember all the network timers for the information blocks.
The sim card interaction, is say you're at home and you boot up your phone with 100% clean state. You don't want to wait for this scan to complete, so the SIM card gives the phone hints about which frequencies the carrier uses, so start on frequency x to find the network. But if you roam internationally, it can take alot longer to find a partner network, and there are some other techs around steering to preferred partners, but I don't know that those come into play here. I don't know but would be surprised if there is a SIM option to try and pin the emergency calls to a network, I think it's more likely the interaction is this hint on where to start the scan.
The way the rogers network failed, it appears to me it caused the towers to stay in a state where they advertised in their radio block the network was there, and the 911 bit was enabled so the network could be used for emergency calls. This is where I don't really have the details since they haven't been public about it, how much of their network was still available internally. Maybe the cell towers could all see each other, that network layer was OK, and the signalling equipment was all talking to each other as well. That's the part I don't really know and have to speculate, as well as the tower side since I was a core person. So because the towers had enough service to never wilt themselves, they kept advertising the network, along with the 911 support. But then when you try to activate an emergency call, somewhere in the signalling path, as you get from tower to signalling system, to the voip equipment, to the circuits to the emergency center the outage knocked something out. Oh and for all these pieces of 911 equipment, there are two of everything for redundancy... two network paths, two pieces of equipment, etc.
And because they lost admin access to their management network, no one could go in manually and tell the towers to wilt themselves either.
If the towers had just stopped advertising 911 services, the phone would fall back into the network search mode as I described when you have no sim card. It just starts scanning the frequencies until it see's an information block for a network it can talk with the emergency support advertised to and does an emergency attach to the network that the carriers will all accept (An unauthenticated attach for the sole purpose of contacting an emergency center).
So my suspicion is because carriers are so used to we have two of everything, and all emergency calls are marked for priority handling at all layers of the equipment (they get high priority bits on all the network packets and priority CPU scheduling in all the equipment), this particular failure mode where there was a fault somewhere down the line, and they lost control of the towers to tell them to stop advertising 911 services all sort of played together to create the failure mode.
- mjevans 3y agoMulti-faceted failure mode. 0) At the network terminal level (mobile phone): at least for emergency calls if a given network fails to connect, fail over and try other networks. Even if the preferred networks claim to provide service. 1) At the network level: failure thresholds should be present. If those thresholds are crossed enter a fail-safe state. This should include entering a soft offline / overloaded response state. 2) Where possible critical data paths should cross-route. Infra Command and Control and Emergency calls in this case. Though if Roger's issue was expired certs or something the plans for handling that get complicated.
- Scoundreller 3y agoit’s that “0” level that surprised me the most here. Days later, Rogers said you might be able to pull out/disable your SIM card to call 9-1-1, but then it depends: if Rogers is the strongest network, you might end up in the same predicament anyway.
- mjevans 3y agoAgreed, 'zombie state' is a valid failure mode. Partners / Infra can think they're alive and respond but be non-functional. As can agents spoofing valid infra but always failing operations.