18 ms·

911 emergency services go down across the US after CenturyLink outage

• StillBored 8y ago
  "In this case, the outage affected only cellular calls to 911, and not landline calls." Which means its basically the system which determines which 911 center to route a cell phone based 911 call. That this is still centralized, rather than distributed to the geographical towers is crazy.
• _eht 8y ago
  This will be a post mortem to read. No details yet but I’m wondering where the broken links are? Is it a VOIP glitch or more than that? Edit: VOIP indeed. Centurylink outage: https://komonews.com/news/local/centurylink-outage-knocks-out-911-calls-in-parts-of-w-wash https://komonews.com/news/local/centurylink-outage-knocks-ou...
  • paganel 8y ago
    Today I’ve learned that vast swathes of the emergency call network depend on just one ISP. I wonder if the old, analogue and copper-only network wasn’t more resilient.
    • ztjio 8y ago
      CenturyLink operates that, too.
    • StudentStuff 8y ago
      In Kitsap County (just across the water from Seattle) 911 traffic is delivered by Centurylink via SIP over a Wave Cable connection (think Docsis 3.0, not fiber). Why the county won't permit everyone to interconnect with them directly befuddles me, as all they're doing is adding a single point of failure by forcing all calls to go through Centurylink.
      • iamdave 8y ago
        Why the county won't permit everyone to interconnect with them directly befuddles me, as all they're doing is adding a single point of failure by forcing all calls to go through Centurylink. I've consulted with a few rural counties (I don't know a thing about the PNW, is Kitsap a rural or suburban county?) who got their circuits leased to them through a local MSP/IT Shop who had all the best intentions, but overlooked any sort of failover and you end up with an answering center taking calls over a single adtran going direct out to a CLEC. Just wondering if this is less county refusing to do something, and more like the county implementing services sold to them that probably could have benefited from better planning and integration.
        • pm90 8y ago
          Probably the latter. How many rural county boards have people who actually understand any of this stuff? It seems more than likely that Centurylink Sales/Marketing convinced a few counties to use them and then sold itself to other counties as "the" provider for such services.
          • oasisbob 8y ago
            In NW WA you might be surprised. There's a long history of utility self sufficiency with the San Juan islands, and the area and neighboring counties has a lot of dark fiber available for lease from smaller independent providers. Many municipalities and community organizations light up their own WAN links. There was a always a tension when it comes to outsourcing and considering core competencies in the orgs I saw. Can imagine less appetite for creative solutions when critical things like 911 are involved.
          • iamdave 8y ago
            How many rural county boards have people who actually understand any of this stuff? Not enough, I would wager. It's something I spend a lot of time thinking about, living in one of the largest cities in the country now, with parents who live comfortably in the rural south-but have complained about the internet a couple of times. To their credit, they'll lose connectivity for hours and not even mind, since mom will just turn on the radio and listen to the local PBS station and tend to her garden while dad goes out to his woodshop. Others in their community probably aren't as content without connectivity, for such long periods as they are.
    • nostrebored 8y ago
      CenturyLink owns large swathes of telecoms infrastructure nationwide, and Level 3, a tfn/did resporg provider
    • reaperducer 8y ago
      It was more resilient, in a way. It had more points of failure, instead of one big point of failure. It was also less available nationwide due to the expense.
    • wpietri 8y ago
      And what amazes me is that the ISP was content to be a single point of failure for emergency services. I'm no expert, but I'd think a primary design criterion for emergency systems is that they keep working in emergencies.
      • komali2 8y ago
        My friend and I were just talking about how they should put a bit of oil/natural gas at the top of all those little 911 repeater/tower/whatever things that are sprinkled throughout the city to turn them into a sort of "Beacons of Gondor" type thing, as an absolute-last-resort communication method. Mostly jokingly.
        • toufiqbarhamov 8y ago
          Well... if you put the oil in a small emergency generator it might be more practical, if less romantic.
      • kortilla 8y ago
        It’s not the ISP’s responsibility to ensure the customers’ have adequate HA designs for their own systems.
        • iamdave 8y ago
          No, but if you are providing voice and telephony services that connect to the public switching network, as CenturyLink does, you absolutely have certain reliability requirements for 911 calls being relayed to local answering points. I await the results of all the inquiries into this to find out what took place here. Edit: Oh and "But $ISP doesn't offer traditional phone service, they sell VoIP", is as of 2005 no longer a valid excuse for non-compliance: https://www.fcc.gov/document/commission-requires-interconnected-voip-providers-provide-enhanced-911 https://www.fcc.gov/document/commission-requires-interconnec...
          • tracker1 8y ago
            This isn't an issue with servicing their direct clients to E911 services, it's cell phone access from various carriers that went out. It's still a bad thing, but not quite the same as what it looks like you're assuming.
    • Animats 8y ago
      It was. In the entire history of electromechanical switching in the Bell System, no central office was down for more than 30 minutes for any reason other than a natural disaster or major fire. There are books about Number Five Crossbar and how it worked, and they're worth reading if you design high-reliability systems. The only central office downtime from a major fire was in 1975 in New York City, resulting in 23 days of downtime.[1] All the incoming cables and the main distributing frame had to be replaced. The switching equipment, on upper floors, survived and just needed some maintenance and cleaning. That was still a crossbar office; it hadn't been converted to electronic switching yet. Worst disaster in the history of the Bell System. Bell switched to less flammable cable coverings, like plenum cable, after that. Widespread failure of 911 service suggests an overcentralized architecture. 911 requires a phone number to address lookup, so there's a database involved. Widespread failure indicates this was implemented as a remote query service ("in the cloud") rather than read-only database copies of the directory at each central office. [1] https://youtu.be/f_AWAmGi-g8?t=110 https://youtu.be/f_AWAmGi-g8?t=110
      • tjohns 8y ago
        There's an old AT&T video showing the careful coordination and huge amount of manpower needed to upgrade a single telco switch. They had rows of workers with cable cutters, in order to literally cut the entire service over to the new equipment. Total downtime was just 47 seconds. (And even during the cutover, they're careful not to interrupt any emergency calls.) https://www.youtube.com/watch?v=saRir95iIWk https://www.youtube.com/watch?v=saRir95iIWk
        • setquk 8y ago
          That was absolutely amazing. Thanks for posting.
        • DanBC 8y ago
          I love the "DON'T TOUCH THIS" post it at 3:55
        • the_duke 8y ago
          So, what exactly happened there? Where were the signals going after the cables have been cut?
    • toast0 8y ago
      CenturyLink is an incumbent local exchange carrier, as it merged with Qwest, which merged with US West, formed in the 1984 AT&T breakup. I wouldn't be surprised to find that most emergency call networks rely on the local ILEC, although I would have expected routing to be less centralized.
      • armchairanalyst 8y ago
        Actually, CenturyLink has sold their services to a very long list of municipalities which then place CL between the ILEC and emergency services.
    • jijji 8y ago
      the decentralized nature of the e911 system is what made it fault tolerant. moving these systems to a more centralized topology is probably going to cost them alot of money in downtime and fines from the FCC in the long run...
  • dbenhur 8y ago
    If someone dies due to inability to summon emergency assistance, this will truly be a post-mortem instead of a badly named incident retrospective.
• tybit 8y ago
  Is there any information available about how these systems are typically architected and run as well as what sort of availability they generally achieve?
  • de_watcher 8y ago
    Nothing fancy, just some PABX. They like to have an independent backup connection and a backup control room.
  • Aloha 8y ago
    I can tell you what 911 Call Taking systems look like, they are mostly a call presentation system, with some call handling features. This issue is upstream of any 911 center however.
• rococode 8y ago
  Interestingly the University of Washington put out an email notice nearly an hour before AlertSeattle sent one. You'd think the city would make it a higher priority to let everyone know that 911 is down. Not sure if they did a text message earlier though, currently overseas w/o service. https://i.imgur.com/2rlxi7f.png https://i.imgur.com/2rlxi7f.png More info on the outage here: https://www.king5.com/article/news/local/nationwide-centurylink-outage-affecting-911-calls-in-western-washington/281-1e461abb-8b3c-49e2-8a57-764e39466213 https://www.king5.com/article/news/local/nationwide-centuryl...
  • icebraining 8y ago
    It may just be that their email service is not particularly optimized for such an urgent case, and so only got to your email address one hour after the person sent the blast. I've seen CRMs take hours to push out a 40k email blast.
  • MaulingMonkey 8y ago
    There was a reverse 911 alert around here about the outage, loud alarm and everything.
  • reaperducer 8y ago
    Interestingly the University of Washington put out an email notice nearly an hour before AlertSeattle sent one. You'd think the city would make it a higher priority to let everyone know that 911 is down. How long do you think it takes to send out a few million alerts versus sending a few thousand?
    • wpietri 8y ago
      For an emergency response system, I hope the answer is "about the same time". Or, more precisely, "within the allowed latency window for emergency alerts".
    • deleted 8y ago
      [deleted]
    • lukeschlather 8y ago
• tarellel 8y ago
  Could this happen to have been part of the massive CenturyLink/Verizon outage that's been rolling across the midwest the last day or 2? I know it's been a major pain in the butt for me, but I've been super productive at work not having to worry about answering calls.
  • walrus01 8y ago
    network engineer for pacific NW ISP here: Yes this is absolutely centurylink related, from what I can observe that it's taken down, and I have my suspicions about Intrado. I think centurylink broke things further trying to fix this morning's DWDM/transport system outage. See the PDF file linked here: https://www.fcc.gov/document/april-2014-multistate-911-outage-report https://www.fcc.gov/document/april-2014-multistate-911-outag...
    • alexeldeib 8y ago
      Appreciate the link, that was a very informative document (didn't finish all 40+ pages in the ~2 minutes to post this comment, of course).
    • pishpash 8y ago
      I know it's not the same orgs, but Centurylink has had the most incompetent people I've dealt with as a retail customer.
      • walrus01 8y ago
        My totally subjective opinion: In US/Canadian telecom, any business formed from legacy near-monopoly ILEC (incumebent local exchange carrier) staff, they neither have a clue nor give a fuck. It's a corporate cultural attitude left over from days when ILECs truly were the last-mile monopoly carriers in many cities. In this case Centurylink is composed of former Embarq, Qwest (USWest) and other ILEC entities. My opinion also applies to entities like Bell, Telus, Verizon and Frontier. If you look at the people who run the core routers of their ASes, sure they know what they're doing, but they're so insulated from actually talking to a customer through myriad layers of corporate bureaucracy, that their clue level might as well be irrelevant.
• Animats 8y ago
  From CenturyLink's Twitter feed: "Update: On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership."
  • HocusLocus 8y ago
    [turns off Spanning Tree Protocol] "What does this switch do?"
  • walrus01 8y ago
    This info is related to their much earlier DWDM/transport system outage, which took down a huge number of 10Gbps lit transport circuits starting around 0700-0800 Pacific time on the 27th. The 911 outage for WA began 10+ hours later. see outages mailing list archives for more details: https://puck.nether.net/mailman/listinfo/outages https://puck.nether.net/mailman/listinfo/outages
    • ct520 8y ago
      This effected Washington, and its still very much on going 24hrs later. Geez is it a pain. https://downdetector.com/status/centurylink/map/ https://downdetector.com/status/centurylink/map/
• jrockway 8y ago
  I'm surprised cell towers can't tell connected phones what the local emergency number is, and then dial that instead when 911 isn't working correctly.
  • ThePadawan 8y ago
    That sounds like a MITM attack waiting to happen.
    • jkepler 8y ago
      Isn't that exactly what cops or others do when they use a Stingray phone tracker and intercept people's mobile phone connections? I still remember in late 2013/early 2014 when I lived in Kyiv, Ukraine, and my friends at the Maidan protest against the government said everybody within geographic proximity of the protest received SMS messages from an unlisted number telling them their protest wasn't 'legal'. Everybody assumed it was the Ukrainian government security forces using a Stingray or similar device as a MITM between their devices and the network.
      • Crosseye_Jack 8y ago
        If the government is in control of the network or can can order / request the network to do so it would be fairly trivial to send a message to anyone connected to a cell tower. The thing is if the tower is serving a lot of people you could end up sending the message to people not involved so you could use a IMSI catcher (you can even make them out of a TV Tuner turned SDR receiver[0]) to catch IMSI's in a smaller area or use triangulation to narrow down the pool of devices in an area and then use that list to send SMS from the network itself. The UK did trails of using location based text messages back in 2013 in order to be used for Public emergency alerts [1] When you can gain access to the network their is no real need to MITM it, but yeah I believe a sting ray type device could also do the same as they act as a tower and trick your phone into connecting to it. [0] https://github.com/Oros42/IMSI-catcher https://github.com/Oros42/IMSI-catcher [1] https://www.gov.uk/government/publications/mobile-alerting-trials-for-public-emergencies https://www.gov.uk/government/publications/mobile-alerting-t... (link to the info about the trial and the final report)
• BucketSort 8y ago
  The incompetence of the Roman government once became so great that they lost the ability to provide basic public utilities like the keeping of time. Is this that for us? How could 911 possibly go down? Crazy.
  • StudentStuff 8y ago
    911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate. There are thousands of (poorly run, uncoordinated) Public Safety Answering Points across the country, and most tend to contract with just the local incumbent telecom, ensuring any time there is an issue with said incumbent, every call to said PSAP fails. In the context of most of these PSAPs having already moved to VOIP, there isn't a coherent reason why these calls shouldn't go straight from the carrier originating the call to the PSAP, without middlemen like Centurylink standing by to misroute those calls.
    • JadeNB 8y ago
      > 911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate. I'm always interested in Chicago history, so I went looking, but couldn't find anything at, for example, https://www.nena.org/page/911overviewfacts https://www.nena.org/page/911overviewfacts . Do you know anywhere this history is written up?
      • JadeNB 8y ago
        Ah, I may have seen what you mean. According to https://www.countyofunion.org/site/cpage.asp?cpage_id=180009766&sec_id=180003667 https://www.countyofunion.org/site/cpage.asp?cpage_id=180009... : > 1976 Chicago claims to have had "the first enhanced 911 system of any major city" in the United States. … so I guess in > > 911 is a creaky old system put in place by a Chicago mayor looking to score some points with the electorate. you meant E911 (which, as the linked page points out, is a nebulously defined and variously implemented technology), not necessarily 911?
    • ecp9 8y ago
• vegardx 8y ago
  And this is why I've made it a habit to save the the local number to the police station where I live, and after moving around a little I seem to have accumulated a few. Always nice to have a backup should you need it.
  • riskable 8y ago
    Not just the police department... Fire, ambulance, poison control, dig safely, local traffic control tower (they're about ~9 miles away from me), pharmacy, and suicide prevention (every fridge needs one of those magnets).
    • tomcam 8y ago
      Smart. Noting these. Why traffic control tower?
      • techopoly 8y ago
        Maybe to call if there's a plane in trouble?
        • iamdave 8y ago
          Some airports also have public phone numbers to get weather information, could come in handy to know if you're-for example, living in tornado alley and a system rolls through that knocks out power and cellular service. If you have a working landline phone, calling a nearby airport could be your only way to know the latest conditions and if it's safe to go outside.
          • HocusLocus 8y ago
            My elderly father is living with my sister's family. To find out if it's safe to venture outside they tie a rope to him and push him out the door.
    • lukeschlather 8y ago
• Waterluvian 8y ago
  Is 911 in the U.S. still that wildly disparate patchwork of networks and operations centres?
  • deleted 8y ago
    [deleted]
  • minikites 8y ago
    >wildly disparate patchwork of networks This is true of pretty much all types of infrastructure in the USA, from roads to healthcare to electricity.
  • gwright 8y ago
    You say that as if it is a bad thing. Decentralizing key infrastructure and emergency response systems seems like a good idea.
    • wpietri 8y ago
      Absolutely. Especially given this widespread failure, the last thing I want is more central control.
      • minikites 8y ago
        There needs to be some central or regional oversight body to ensure standards are being met, equitable access is being maintained, etc. A lot of poor areas are undeserved by these patchwork networks and several infrastructure deficiencies can compound. Example: poorly maintained roads leads to potholes, those lead to drivers swerving or getting otherwise surprised leading to injuries, which are insufficiently addressed by local healthcare. Imagine if we had a tyrannical electric utility who cut your power for a bill paid one day late.
        • gwright 8y ago
          > There needs to be some central or regional oversight body to ensure standards are being met You mean something like your local, county, or state governments and their agencies?
• tzs 8y ago
  At least the systems to notify of the outage sure worked well, at least for me. Got an EAS alert while watching TV and two emergency alerts on my cell phone (one generic telling me to call local police or fire, and one telling me a specific number to call for my county). Furthermore, I now think that a call that came in on my landline at 10:42 PM was probably also a notification. The caller ID just said "wireless caller" and I didn't recognize the number so I ignored it, but now Googling that number turns it up as a cell contact number for my county's Department of Emergency Management.
• petee 8y ago
  Mass says they restored 911 by 9am, but I just got an alert at 12 noon saying it's down now...
• iask 8y ago
  So, now there is proof that our national emergency 911 system has a possible vulnerability. Hope this will be addressed with some prioritity.
• zipwitch 8y ago
  It wasn't just emergency calls. Most Centurylink service around the US was down intermittently yesterday.
  • rememberbitcoin 8y ago
    How would one distinguish this from any other day on CenturyLink?
  • aviv 8y ago
    A lot of Centurylink circuits are still down in Arizona, and if not down completely, have significant packet loss.
    • reaperducer 8y ago
      A friend texted me that his POS systems that rely on CenturyLink in Nevada have been down for two days. The stores can only do transactions in cash. Makes you think twice about the cashless utopia the HN crowd imagines.
      • lotsofpulp 8y ago
        Makes me think we should be investing in redundancies (one of which is cash, but also in our electronic networks).
      • smileysteve 8y ago
        Or to switch to crypto currency that is distributed and eventually consistant
        • dvtrn 8y ago
          I swear the swiftness with which people swoop into these kinds of threads to suggest crypto/blockchain is bordering on becoming a meme. If the network is fully down, I don't see how the scenario for this business owner changes with blockchain, they'd have the exact same problem with a register that can't close an ACH: backlogged transactions that need to catch up eventually, meaning some card types are going to decline if the POS can't reach a clearing house utilized by the card vendor.
• walrus01 8y ago
  Notes from their massive DWDM system outage earlier on the 27th, which preceded the 911 outage: 2018-12-28 13:35:00 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT remains. 2018-12-28 13:27:30 GMT - The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes. CenturyLink NOC management remains in contact with the equipment vendor to obtain updates as restoration efforts continue. 2018-12-28 11:04:24 GMT - CenturyLink continues to work with the Equipment Vendor to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Client services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes. 2018-12-28 10:05:18 GMT - CenturyLink NOC Management reports steady progression of node recovery and restoral of client services. In addition to the remote node recovery process, Field Operations continue to dispatch and assist the Equipment Vendor with local equipment login. 2018-12-28 08:51:29 GMT - CenturyLink NOC Management has advised that repair efforts are steadily progressing, and services are incrementally restoring. The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels at this time. There have been additional restoration steps identified for certain nodes, which includes either line card resets or Field Operations dispatches for local equipment login, that have impeded the restoration process. Various repair teams are working in tandem on these actions to ensure that services are restored in the most expeditious method available. Restoration efforts are ongoing. 2018-12-28 07:12:32 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. Additional information on repair progress will be available from the Equipment Vendor by 07:30 GMT. Information will be relayed as soon as it is obtained. 2018-12-28 06:00:01 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT remains. 2018-12-28 04:58:44 GMT - CenturyLink engineers in conjunction with the Equipment Vendor¿s Tier IV Technical Support team have identified the elements causing the impact to customer services. Through the filters being applied and the removal of the secondary communication channels, it is anticipated services will be fully restored within four hours.¿We apologize for any inconvenience this caused our customers. Additional details regarding details of the underlying cause will be relayed as available. 2018-12-28 04:09:31 GMT - The Equipment Vendor¿s Tier IV Technical Support team in conjunction with CenturyLink Tier III Technical Support continues to remotely work to remove the secondary communication channel tunnels across the network until full visibility can be restored, as well as applying the necessary polling filter to each of the reachable nodes. 2018-12-28 02:53:38 GMT - The Transport NOC has confirmed that cooperative efforts remain ongoing to remove the secondary communication channel tunnel across the network until full visibility can be restored, as well as applying the necessary filter to each of the reachable nodes. It has been confirmed that both of these actions are being performed remotely, but an estimated time to complete the activities is not available at this time. 2018-12-28 01:58:56 GMT - Once the card was removed in Denver, CO it was confirmed that there was no significant improvement. Additional packet captures, and logs will be pulled from the device with the card removed to further isolate the root cause. The Equipment vendor continues to work with CenturyLink Field Operations at multiple sites to remove the secondary communication channel tunnel across the network until full visibility can be restored. The equipment vendor has identified a number of additional nodes that visibility has been restored to, and their engineers are currently working to apply the necessary filter to each of the reachable nodes. 2018-12-28 00:59:04 GMT - Following the review of the logs and packet captures, the Equipment Vendor's Tier IV Support team has identified a suspected card issue in Denver, CO. Field Operations has arrived on site and are working in cooperation with the Equipment Vendor to remove the card. 2018-12-27 23:57:16 GMT - The Equipment Vendor is currently reviewing the logs and packet captures from devices that have been completed, while logs and packet captures continue to be pulled from additional devices. The necessary teams continue to remove a secondary communication channel tunnel across the network until visibility can be restored. All technical teams continue to diligently work to review the information obtained in an effort to isolate the root cause. 2018-12-27 22:52:43 GMT - Multiple teams continue work to pull additional logs and packet captures on devices that have had visibility restored, which will be scrutinized during root cause analysis. The Tier IV Equipment Vendor Technical Support team in conjunction with Field Operations are working to remove a secondary communication channel tunnel across the network until visibility can be restored. The Equipment Vendor Support team has dispatched their Field Operations team to the site in Chicago, IL and has been obtaining data directly from the equipment. 2018-12-27 21:35:55 GMT - It has been advised that visibility has been restored to both the Chicago, IL and Atlanta, GA sites. Engineering and Tier IV Equipment Vendor Technical Support are currently working to obtain additional logs from devices across multiple sites including Chicago and Atlanta to further isolate the root cause. 2018-12-27 21:01:26 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC was engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO and New Orleans, LA a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. Tier IV Equipment Vendor Technical Support continues to review equipment logs from the sites where visibility was previously restored. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.
• mekaj 8y ago
  There was an incident with similar consequences on April 10, 2014. The cause was a programmed threshold being breached and the impact was 6h of downtime. Source: "The Coming Software Apocalypse" published by The Atlantic (https://www.theatlantic.com/technology/archive/2017/09/saving-the-world-from-code/540393/ https://www.theatlantic.com/technology/archive/2017/09/savin...) walrus01 also linked to https://www.fcc.gov/document/april-2014-multistate-911-outage-report https://www.fcc.gov/document/april-2014-multistate-911-outag... in another comment.
  • PhasmaFelis 8y ago
    > Operated by a systems provider named Intrado, the server kept a running counter of how many calls it had routed to 911 dispatchers around the country. Intrado programmers had set a threshold for how high the counter could go. They picked a number in the millions. I'm really curious if there's some explanation that makes this sound less catastrophically stupid, particularly the part where they picked a threshold less than INT_MAX.
    • MacroChip 8y ago
      Great question. Maybe they did pick a sensible constant, but it still wasn't enough. I.e. INT_MAX wasn't big enough.
    • swiftcoder 8y ago
      What do you mean, VARCHAR(8) is a perfectly valid way of storing counters...
      • ineedasername 8y ago
        much better to future proof it by using a BLOB. You know, just in case you ever need to shove something else in there.
        • AdamJacobMuller 8y ago
          if you're going to use a BLOB how can you know the type and how to decode it? Make sure to serialize that data with something like protobuf.
• ergothus 8y ago
  I'm unable to read the article on my phone (ad?) so I apologize if it is answered there, but why is this a long lasting problem? My past two or three (not sure) workplaces have had to failover to alternate providers for office traffic when something broke in the primary, and both had contracts in place and the technical infrastructure to do so in relatively short order. (I don't want to ignore the pre- and intra- effort by ops, I'm just saying it paid off) Is this outage just breaking those places that didn't have such things (as a result of insufficient budget or skill)? Is there some now-regrettable contract(s) that prevented that from being in place? Why is a system that saves lives working without the backup that systems that dont dictate life are? While I understand that govt has both budget and hiring issues, i've worked in state govt and there are definitely some highly competant admins to be found, so I wouldn't assume that is the problem without word from someone with more detailed knowledge into this particular problem.
  • jpollock 8y ago
    Software is hard, and backbones tend to be mono-cultures. This means that there will be bugs which only become apparent when the full fleet has been deployed. This is why large phone networks have really long rollout periods - to hopefully catch these things before it affects everything. This is also why emergency services should not be using cell phones for reliable service during emergencies. Here's another example from 1990, where the phone network got stuck in a reset loop. https://catless.ncl.ac.uk/Risks/9/62#subj2 https://catless.ncl.ac.uk/Risks/9/62#subj2 Corporations make similar mistakes with their disaster planning. It doesn't matter if you purchase network connections from two suppliers if they go across the same bridge.
    • flexer2 8y ago
      My dad spent his career as a natural gas pipeline technician. For a long time they maintained their own radio towers and VHF or UHF (I forget which) radio network. Each district had their own comms guy who maintained the network, and it was very resilient and reliable, allowing the entire company to be in contact essentially anywhere. In the early 2000s as a cost cutting measure, the company decided to nix the radios for satellite phones on each truck. Of course this proved to be problematic, as the sat phones often had reception issues. They relied on cell phones as backups, which was also quite foolish as the remote compressor stations had very poor reception. They also had issues where someone would leave a voicemail and they wouldn’t get notified of it for days or weeks due to some issue with AT&T. After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies and dismantled the infrastructure. My dad retired not long after this, but the “corporate bean counter” trope rang quite true here, and in the long run we were all a little less safe because some executive with no field experience wanted to make a name for themself by saving a little money on something that proved to be mission critical.
• mirimir 8y ago
  context: https://news.ycombinator.com/item?id=18775352 https://news.ycombinator.com/item?id=18775352
  • dang 8y ago
    Since the current post adds information, we'll leave it up instead of marking it as a dupe, and have moved most of the non-time-sensitive comments here.
• EGreg 8y ago
  Why is it centralized??
  • JeremyBanks 8y ago
    Put it on the blockchain.
• drdeadringer 8y ago
  After reading about this earlier today I had the amusing experience of tweeting my local PD asking if "our" 911 was still operational given the situation; apparently it is given their reply tweet. I do still have the actual "direct" local phone numbers of dispatches for the areas I most frequent.
• tracker1 8y ago
  > Ajit Pai, chairman of the Federal Communications Commission, which regulates and monitors 911 services, said the commission is investigating the outage. Until they "resolve" the issue without disclosing any details...
  • josteink 8y ago
    Maybe they will open the case for public comments and insights, then “ignore” those comments because of a “DDOS attack”, and then finally not investigate the alleged attack despite earlier on outlining how it was able to cripple public infrastructure. Could there be a more mismanaged agency than the FCC?
    • clubm8 8y ago
      >Could there be a more mismanaged agency than the FCC? or maybe they're all just as mismangaged, but you noticed them in partiular becaue you are passionate about the FCC's mission? I'd be willing to bet if the average American interacted with fincen, FDA, IRS etc they'd be shocked at the ineptitude. America has been strangling their enforcement agencies for a long time - one could argue there's a direct path from a poorly funded IRS only taking on "slam dunk" cases and ignoring complex money laundering schemes to Trump's election.
      • iscrewyou 8y ago
        The fact that it takes a special prosecutor (and the subsequent reporting) to dig into these big money laundering schemes only cements your point even more.
      • uxp 8y ago
        If someone neglects to provide their children with food and water, it's rather inappropriate to get mad at the children for their lack of capacity to provide manual labor.
  • onetimemanytime 8y ago
    Ajit Pai will be so hard on them that Century Link might offer him a job after the government "service." 911 service is VITAL, pay die from heart attacks when ambulances are not sent and people are killed by psychos when cops are not notified. Make it 100% redundant, all the way. No one single provider should have all of this in one basket.
• herostratus101 8y ago
  lol, The Purge (2018)
• armchairanalyst 8y ago
  CenturyLink is critical to 9-1-1 services throughout the U.S. They have a 9-1-1 division to which a very large number of municipalities outsource. Even military installations outsource parts or all of their 9-1-1 operations to CenturyLink. This CenturyLink division has historically acted very carelessly in regards to securing these systems. Getting them to take ownership of their contractual obligations can be difficult. Criticism of their operations is very sensitive. Pointing out any vulnerabilities can be prosecuted as an act of terrorism. A post as simple as this one comes with significant risk.
  • stingraycharles 8y ago
    While I'm not refuting any of your claims, do you have a source for them? It sound like typical big corp behavior, albeit worse because of the importance of emergency services.
    • nickpsecurity 8y ago
      I at least found the 911 unit: https://www.centurylink.com/wholesale/pcat/911.html https://www.centurylink.com/wholesale/pcat/911.html
      • Operyl 8y ago
        To the best of my understanding, they aren’t providing the 911 call center here.
• clubm8 8y ago
  Is this outage for both mobile (cell phones) and landlines? Does dialing the direct number (XXX-XXX-X911) work? I've had issues on both when there was no widespread outage to the point I've given up on the utility of 911 and try to put the local police department's number in my phone for emergencies.
• JustSomeNobody 8y ago
  Is it me or did TechCrunch recently ruin their mobile experience. I tried to read the article but got so frustrated and distracted by the wonky scrolling I gave up. It’s just awful.
• redleggedfrog 8y ago
  “When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” said Pai in a statement. “The CenturyLink service outage is therefore completely unacceptable, and its breadth and duration are particularly troubling.” In response CenturyLink is now offering a new service, called "911 Fastlane", that guarantees that your 911 calls will get through and will be prioritized ahead of calls from standard CenturyLink accounts. This service is only an additional $10 a month. Of course, satire.
  • FlyMoreRockets 8y ago
    Absolutely brilliant! You win the internet.
    • redleggedfrog 8y ago
      Not sure if your comment is satire or not. Thanks anyway, though.
      • FlyMoreRockets 8y ago
        Judging by the number of downvotes my comment achieved, it seems a lot of folks couldn't tell it was satire. Seemed rather obvious to me.
        • dmd 8y ago
          No, you were downvoted because your comment was content-free and this isn't reddit.
          • deleted 8y ago
            [deleted]
  • ryanlol 8y ago
    Why did you feel the need to post this weird piece of NN propaganda in a completely unrelated thread?
• reasonablemann 8y ago
  Strange Apple doesn't push their own tech for this. They could include it in Messages and release on Android. For Android there would be a store button where you could see all the new stuff from Apple and buy an iPhone. Would do a service to the 50% of the world's population who's entire community communication network is owned and mined by FB.
  • emeraldd 8y ago
    Apple doesn't do this kind of infrastructure. Their stuff only works if the rest of the network works, which seems to be what's falling over here.
• smaili 8y ago
  I'm quite surprised no fallback/failover was put in place for a critical tier service like this.
• epynonymous 8y ago
  time to switch to aws
• shostack 8y ago
  Given the past and recent news about ongoing infrastructure attacks the US has been seeing, reminiscent of Ukraine, is there any possibility this is related?
• IronWolve 8y ago
  Not sure how common the problem still is, but some custom roms for android phones could not use shortcodes, and could not dial 911. I'd recommend having the full numbers programmed in your phones for emergencies if your playing with new roms. Also, federal alerts did not go out for the 911 outage, but major cities did. So subscribe to the local alert systems.
• zw123456 8y ago
  My two cents... When a company lays off their most experienced engineers year over year over year... this is what happens. Just my humble opinion of the real root cause.
• Trisell 8y ago
  From my experience for a lot of these 911 centers they are centered around counties(not all, but a lot). Back when they were first being set up with the internet capacity there were two issues. DSL was the only available service within the area, and the centers couldn’t afford large enough pipes. Talking early 2000s here. So a lot of these centers got pretty large grants from the feds. They then hired one of the companies at the time that were doing these upgrades in bulk around the country when the money was flying because the fcc wanted all of the emergency services to move to narrow band from wide band. These companies came in updated everything to modern and setup the contracts with the DSL providers back in the day(in my rural town I grew up in they just got 80mb cable internet in 2016, prior to that 10mb dsl was the best you could get there) In the west that was Qwest who became Century Link. But government being what it is. A lot of these 911 centers have languished because there is no desire to invest in updating these systems as they are expensive and hve a difficult upgrade process. So a lot of these centers are just paying to bills and trying to scrape by with minimal funding. And so upgrades and redundancy aren’t even considered. Until something like this happens. And the feds might throw some money at the problem. All the centers will get grants to have failover and then will go back into dormancy of just working day to day with no budget.
• ConcernedCoder 8y ago
  I was in a grocery store when my cellphone, in unison with all the other phones in the store gave the clarion call for attention... I thought Russia had finally launched a 1st strike...
• iheartpotatoes 8y ago
  From 1990, these guys were so ahead of their time (yea, I know, institutional racism != IT infrastructure, but still...): https://www.youtube.com/watch?v=CPNK0VspQ0M https://www.youtube.com/watch?v=CPNK0VspQ0M
• Schmazo 8y ago
  Should cell phone users be plugging in 360-693-3111 as an emergency backup in the event 911 goes down in the future?
• anticensor 8y ago
  Why does a congressman not step up and draft a law stipulating: * US telecom, an office of US federal government shall be founded and it shall serve all federal and state governments' institutions. US telecom shall be funded by federal sales tax. Government institutions themselves shall not be billed. * Emergency services number shall be 911. * 911 shall be operated by US telecom and US telecom shall pay private communication providers in international rate every time 911 is called. * Private providers shall not bill 911 calls to their subscribers.