4 ms·
Nothing like this story of getting bullied to do something, but I once talked a director out of rebooting the cell phone switch that handled all cellular phone
by kevin_nisbet 2y ago
Nothing like this story of getting bullied to do something, but I once talked a director out of rebooting the cell phone switch that handled all cellular phone calls for I think it was 4 provinces in Canada. This is something I don't think we had ever done before, and was expected to be a 100% outage for all cell phones calls + 911 for like 20 - 25 minutes or something.
We were having elevated error rates due to an earlier outage, and had just run the next closest hail mary to recovery the switch by telling it to resync all temporary state. The problem was that resync didn't work the way anyone on the team thought it did and the director didn't give enough time for it to be effective. We later found out the resync for a particular record didn't happen until the cell phone itself checked in with the network for a periodic update, so we had to wait for however long that timer was to expire on all cell phones. Luckily there was an active 911 call on the switch which caused the team to delay long enough for me to get the right data (the real world failure rate was far below 1% and starting to trend towards normal after the resync command) and talk the director out of the reboot.
The funny thing is that equipment wasn't even mine, I worked on the internet side of cell phones, and I don't even remember why I was pulled onto that outage call since it didn't have anything to do with my part of the network.
- KMag 2y ago> The funny thing is that equipment wasn't even mine, Did you have a reputation for being good at debugging things? Expertise and experience in one area often has crossover in connected systems. I got pulled in to advise on an automated options market making system written in Java when the closest I'd done was equity execution algorithms in Slang and C++ (and integration with the options automated delta hedging facility).
- kevin_nisbet 2y agoYea I did alot of debugging of the networks hardest problems, and the crossover is especially true with telecom systems since all the inter-connectivity comes from the standards bodies, which share a lot of the same base and history. But some stuff differed as well, it was difficult for a bunch of the hardened circuit switched folks to move to an all IP world, and I only had a cursory view into the circuit switched world which was on it's way out when I was in industry.