5 ms·
This will be a post mortem to read. No details yet but I’m wondering where the broken links are? Is it a VOIP glitch or more than that? Edit: VOIP indeed. Cent
by _eht 8y ago
This will be a post mortem to read. No details yet but I’m wondering where the broken links are? Is it a VOIP glitch or more than that?
Edit:
VOIP indeed. Centurylink outage: https://komonews.com/news/local/centurylink-outage-knocks-out-911-calls-in-parts-of-w-wash https://komonews.com/news/local/centurylink-outage-knocks-ou...
- paganel 8y agoToday I’ve learned that vast swathes of the emergency call network depend on just one ISP. I wonder if the old, analogue and copper-only network wasn’t more resilient.
- ztjio 8y agoCenturyLink operates that, too.
- StudentStuff 8y agoIn Kitsap County (just across the water from Seattle) 911 traffic is delivered by Centurylink via SIP over a Wave Cable connection (think Docsis 3.0, not fiber). Why the county won't permit everyone to interconnect with them directly befuddles me, as all they're doing is adding a single point of failure by forcing all calls to go through Centurylink.
- iamdave 8y agoWhy the county won't permit everyone to interconnect with them directly befuddles me, as all they're doing is adding a single point of failure by forcing all calls to go through Centurylink. I've consulted with a few rural counties (I don't know a thing about the PNW, is Kitsap a rural or suburban county?) who got their circuits leased to them through a local MSP/IT Shop who had all the best intentions, but overlooked any sort of failover and you end up with an answering center taking calls over a single adtran going direct out to a CLEC. Just wondering if this is less county refusing to do something, and more like the county implementing services sold to them that probably could have benefited from better planning and integration.
- pm90 8y agoProbably the latter. How many rural county boards have people who actually understand any of this stuff? It seems more than likely that Centurylink Sales/Marketing convinced a few counties to use them and then sold itself to other counties as "the" provider for such services.
- oasisbob 8y agoIn NW WA you might be surprised. There's a long history of utility self sufficiency with the San Juan islands, and the area and neighboring counties has a lot of dark fiber available for lease from smaller independent providers. Many municipalities and community organizations light up their own WAN links. There was a always a tension when it comes to outsourcing and considering core competencies in the orgs I saw. Can imagine less appetite for creative solutions when critical things like 911 are involved.
- iamdave 8y agoHow many rural county boards have people who actually understand any of this stuff? Not enough, I would wager. It's something I spend a lot of time thinking about, living in one of the largest cities in the country now, with parents who live comfortably in the rural south-but have complained about the internet a couple of times. To their credit, they'll lose connectivity for hours and not even mind, since mom will just turn on the radio and listen to the local PBS station and tend to her garden while dad goes out to his woodshop. Others in their community probably aren't as content without connectivity, for such long periods as they are.
- pbhjpbhj 8y agoIf gov at that level can't afford to consult, decisions on that infrastructure should move up a level.
- ineedasername 8y agoYes, it does seem rather redundant to have each county responsible for rolling their own solution (even if that's just deciding how to outsource it)
- nostrebored 8y agoCenturyLink owns large swathes of telecoms infrastructure nationwide, and Level 3, a tfn/did resporg provider
- reaperducer 8y agoIt was more resilient, in a way. It had more points of failure, instead of one big point of failure. It was also less available nationwide due to the expense.
- wpietri 8y agoAnd what amazes me is that the ISP was content to be a single point of failure for emergency services. I'm no expert, but I'd think a primary design criterion for emergency systems is that they keep working in emergencies.
- komali2 8y agoMy friend and I were just talking about how they should put a bit of oil/natural gas at the top of all those little 911 repeater/tower/whatever things that are sprinkled throughout the city to turn them into a sort of "Beacons of Gondor" type thing, as an absolute-last-resort communication method. Mostly jokingly.
- toufiqbarhamov 8y agoWell... if you put the oil in a small emergency generator it might be more practical, if less romantic.
- kortilla 8y agoIt’s not the ISP’s responsibility to ensure the customers’ have adequate HA designs for their own systems.
- iamdave 8y agoNo, but if you are providing voice and telephony services that connect to the public switching network, as CenturyLink does, you absolutely have certain reliability requirements for 911 calls being relayed to local answering points. I await the results of all the inquiries into this to find out what took place here. Edit: Oh and "But $ISP doesn't offer traditional phone service, they sell VoIP", is as of 2005 no longer a valid excuse for non-compliance: https://www.fcc.gov/document/commission-requires-interconnected-voip-providers-provide-enhanced-911 https://www.fcc.gov/document/commission-requires-interconnec...
- tracker1 8y agoThis isn't an issue with servicing their direct clients to E911 services, it's cell phone access from various carriers that went out. It's still a bad thing, but not quite the same as what it looks like you're assuming.
- Animats 8y agoIt was. In the entire history of electromechanical switching in the Bell System, no central office was down for more than 30 minutes for any reason other than a natural disaster or major fire. There are books about Number Five Crossbar and how it worked, and they're worth reading if you design high-reliability systems. The only central office downtime from a major fire was in 1975 in New York City, resulting in 23 days of downtime.[1] All the incoming cables and the main distributing frame had to be replaced. The switching equipment, on upper floors, survived and just needed some maintenance and cleaning. That was still a crossbar office; it hadn't been converted to electronic switching yet. Worst disaster in the history of the Bell System. Bell switched to less flammable cable coverings, like plenum cable, after that. Widespread failure of 911 service suggests an overcentralized architecture. 911 requires a phone number to address lookup, so there's a database involved. Widespread failure indicates this was implemented as a remote query service ("in the cloud") rather than read-only database copies of the directory at each central office. [1] https://youtu.be/f_AWAmGi-g8?t=110 https://youtu.be/f_AWAmGi-g8?t=110
- tjohns 8y agoThere's an old AT&T video showing the careful coordination and huge amount of manpower needed to upgrade a single telco switch. They had rows of workers with cable cutters, in order to literally cut the entire service over to the new equipment. Total downtime was just 47 seconds. (And even during the cutover, they're careful not to interrupt any emergency calls.) https://www.youtube.com/watch?v=saRir95iIWk https://www.youtube.com/watch?v=saRir95iIWk
- toast0 8y agoCenturyLink is an incumbent local exchange carrier, as it merged with Qwest, which merged with US West, formed in the 1984 AT&T breakup. I wouldn't be surprised to find that most emergency call networks rely on the local ILEC, although I would have expected routing to be less centralized.
- armchairanalyst 8y agoActually, CenturyLink has sold their services to a very long list of municipalities which then place CL between the ILEC and emergency services.
- jijji 8y agothe decentralized nature of the e911 system is what made it fault tolerant. moving these systems to a more centralized topology is probably going to cost them alot of money in downtime and fines from the FCC in the long run...
- dbenhur 8y agoIf someone dies due to inability to summon emergency assistance, this will truly be a post-mortem instead of a badly named incident retrospective.