16 ms·
Level 3 technician's misstep causes largest telephone outage ever reported
- empath75 9y agoI wouldn’t consider that to be a technicians misstep. That’s poor software design.
- bigiain 9y agoYeah, but nobody ever seems to throw the UX designers under the bus, right?
- davidkuhta 9y agoWell there was that one test missile notification system in Hawaii with stellar UX and UI...
- madeofpalk 9y ago...and it was the technician/operator that was fired.
- ashelmire 9y agoThat’s a funny way to spell project manager.
- Mtinie 9y agoTLDR; Ill-conceived design is only part of the problem. —— You are assuming the entry form was put together with the assistance of a UX designer. In my experience, back office or configuration software rarely is seen as important enough by the Powers-That-Be to justify all the “extra research and design effort” required. Usually work like that is tossed over to whichever mid-level software engineer has an extra cycle or two during the sprint. This is not to imply that software engineers are always bad at UX. In fact, most engineers I’ve worked closely with care a lot about the end users’ experiences, however, when push comes to shove, their leaders (or the overseeing Product teams they are accountable to) push for rapid delivery of features to reach market parity, rather than spending the extra few days to formally validate the right design decisions were made and if the implementation of those designs were solidly understandable. Going back to your original comment: We should, as an industry, hold designers accountable if their decisions lead to detrimental consequences, especially ones which could be anticipated, like this one. However, we should also recognize this is never the failure of solely the designer, but rather indicators of systematic issues of the extended team. A poorly designed feature/function which makes it way through... * pre-dev stakeholder reviews * development and implementation * quality assurance * acceptance testing ...before release has been vetted and signed-off by enough people to ensure everyone is complicit. It is a failure of culture and leadership if no one along the chain had been comfortable or able to raise a flag if they disagreed or foresaw a problem. Edit: Fixed formatting and this ended up to be longer than I expected when I stared typing. Also, typos.
- bigiain 9y ago100% agree with everything you say here. I strongly recommend: https://deardesignstudent.com/a-designers-code-of-ethics-f4a88aca9e95 https://deardesignstudent.com/a-designers-code-of-ethics-f4a... and: https://medium.com/@monteiro/designs-lost-generation-ac7289549017 https://medium.com/@monteiro/designs-lost-generation-ac72895... from someone who's done _way_ more thinking about these sorts of issues than I have...
- virtuowl 9y agoThis sounds more like a failed ui in the managing software than the technicians fault if noone there knew what that empty field would do
- mdip 9y agoI commented more extensively in the root of the post, but you can't even begin to imagine. Think about every script you've ever written for "some thing at home" and how you only cared that it worked for the very narrow, specific, circumstances you were looking for. Maybe you left out error handling and just let it crash when you failed to put in the right parameter. Who cares? It's just a script for your one, lonely, workstation/server. That's about the quality we're talking about. The companies that make these switches sell them to, maybe, five customers[0]. Software upgrades? Sure, if you replace that $30,000 card with the new version. Having trouble with the software? A support contract can be purchased for a similarly high fee[1]. A company producing this equipment doesn't put a lot of money into QA. In security, there was a general fear about these programs. It was so concerning that the management interfaces to the equipment was on "as close to air-gapped as you can be without being air-gapped" networks with the kinds of logging, auditing and the likes that you'd expect for a network holding government classified information[2]. [0] So few customers, in fact, that you can call them up with a serial number and find out who the purchaser is. I know this first hand due to someone propping the door to the switch site open resulting in, I think, 5 of what I was told were $30,000 a-piece cards being stolen. I was told they were effectively worthless to the thief, though, because nobody would buy them second hand in that manner and the moment they were offered for sale, if someone realized what they were, the thief would be caught. [1] To be fair, I know of one specific circumstance where the company only offered paid support but that was mainly because I didn't work on that team; I'd speculate that all of them functioned this way. [2] Well, maybe what you'd expect in an ideal world, anyway.
- indigochill 9y agoTo [0], would there be any value to stealing the cards not to sell them but to reverse engineer them to find vectors for attacks? "Effectively unsupported, never updated, and responsible for a massive amount of telecommunications" sounds like a great time for an attacker. Also seems like it would have value for industrial espionage.
- Shivetya 9y agothat is an impressive flaw. why would you ever assume a wild card across all parts of a phone number? I get a lot of spam calls where there area code + prefix matches my number and I just ignore them. I have a hard time justifying even wildcard values for the last four digits in whole, partial wild card yes to get a PBX or such
- mdip 9y agoMy wild, totally uninformed, guess that it wasn't an assumption that empty meant wildcard but rather a failure to sanitize input at all. My experience, though limited, with these management applications is that the developers assumed the operator would never, ever, ever enter an illegal or unexpected value and therefor implicitly trusted the input. It's possible that the empty value was considered illegal and whatever module that handles call routing on the phone switch failed to function, blocking traffic as a result of the value. It sounds crazy, but consider a similar circumstance of popping something illegal into an Apache or nginx configuration file. The service fails to start and anything hosted behind it is down. I'm not saying it's acceptable, just likely[0]. The difference here is that this software has an audience of very few people, is poorly developed to begin with, and usually outputs error messages similar to C++ compilers from the 90s. And the software was probably written in the 90s, too. [0] While a competent sysadmin expects that a failure to provide valid values in a configuration file will result in a service not functioning, our typical interaction with modern software comes with the expectation that an invalid value provided to a configuration form will result in a rejection of the value. Even in the cases of Apache/nginx, they provide a method to check your configuration before using it -- just to be safe to make sure you didn't leave out a semicolon/closing brace/</Something>
- paulie_a 9y agoI rarely answer calls from numbers I don't recognize anymore, unless it is to simply fuck with the scammer and waste their time.
- nkrisc 9y agoI never even answer to mess with them because then they know it's a good number.
- exikyut 9y agoOh wow. > The technician left empty a field that would normally contain a target telephone number. The network management software interpreted the empty field as a 'wildcard,' ... Exactly the same type of technical error happened nine years ago at Google! > We maintain a list of [malicious] sites through both manual and automated methods. We periodically update that list and released one such update to the site this morning. Unfortunately (and here's the human error), the URL of '/' was mistakenly checked in as a value to the file and '/' expands to all URLs. What it looked like: https://i.imgur.com/W5ICyVq.png https://i.imgur.com/W5ICyVq.png, https://i.imgur.com/LrtLceN.png https://i.imgur.com/LrtLceN.png Source: https://googleblog.blogspot.com.au/2009/01/this-site-may-harm-your-computer-on.html https://googleblog.blogspot.com.au/2009/01/this-site-may-har...
- Mtinie 9y agoThis is a painful reminder for those of us who routinely design form-based interfaces to ensure ambiguous fields are explicitly understood by the user before entry or called out during verification of the submission.
- mdip 9y agoAn "Are you sure?" prompt goes a long way and sometimes takes a really long time to get added, even when it's historically been a problem[0]. Particularly in these kinds of cases -- this is software that is used by very few people at very few companies and at those companies, it's used very rarely. That nobody at Level 3 knew leaving that field blank would cause that issue doesn't surprise me at all. We had management applications for some switch software that we had to run on Windows 98 using ThinkPad laptops[1]. [0] https://en.wikipedia.org/wiki/Rm_(Unix)#Protection_of_the_filesystem_root https://en.wikipedia.org/wiki/Rm_(Unix)#Protection_of_the_fi... [1] I was a Frontier, then Global Crossing and ultimately Level 3 employee for around 17 years. The story about the ThinkPad Laptops is detailed in another comment in this post.
- Mtinie 9y agoI agree completely. As a designer I fear I’ve missed something like this any time a feature or function I’ve worked on is released.
- mdip 9y agoI was an employee of Level 3 for a very long time (in IT, various roles -- most of my career was there). A small disclaimer: I left before this incident happened (about a year prior) and have spoken with nobody about it, so I have no insider knowledge specific to this. I was also an employee of Global Crossing that was acquired by Level 3 and this incident appears to have happened on the Level 3 side of the network (though it's not immediately clear; GC operated a SONET network and it's entirely possible it was from there). The article and at least one other commenter mentioned this being a UI problem, and all I can say is "bingo". They didn't identify the vendor, but the article called out Cisco. I am a little skeptical of that, personally[0]. I lean toward it being something else, mainly because of the statement that followed "no one at Level 3 was aware of the consequences of leaving that field empty". We had a lot of very knowledgeable Cisco folks there. There are a few folks that I knew personally that were probably among the top folks on administering that equipment outside of Cisco. In addition, if a problem like this arose, they're accessible and helpful. It was almost certainly one of the many, many, outdated software applications that make up the vast array of management interfaces into the equipment. I worked inside one of the phone switches (technically, one of the test switches). About a year prior to my leaving, the room next to my desk was filled floor to ceiling with cards that had been there on the day I started. I was told these cards ran along the lines of $30,000 a piece. If you wanted to upgrade the software, you had to replace the card. So we had machines in our operations center that were on isolated networks running Windows 98 in order to run the executable required to configure the switches[1]. We had devices made by Pirelli[3] that had similarly awful software. I knew of about 5 different devices with 5 different problem management tools but there were several more. And there were the handful of devices that were well known as ones "you don't even breathe on" (and I've never heard those words said about the Cisco products ... not that people spoke terribly highly of them, but never quite that negatively). Telecom went through a really bad time in the early 2000s. At Global Crossing, every 6 months about 10% of the staff was laid off ... this happened for 10 years. Hardware didn't get upgraded, and therefore software didn't get upgraded. Sofware support contracts and maintenance were allowed to lapse. The quality of the software -- since its audience was a small handful of companies, many of which in the same financial state as Global Crossing -- was awful. After that many layoffs, the one or two guys who knew the myriad of corner cases involved in operating some of the management interfaces were on to other jobs at other companies, or retired. That this sort of thing didn't happen somewhat regularly is still surprising to me to this day. At the time that I left, Level 3 was doing better from a financial standpoint and money was being invested in modernising a lot of these problems, but it's a huge network with many switch sites[4] in any major city that Level 3 terminated wires. Each of those sites had an array of equipment, some of it common, some of it from companies that went under shortly after the dot-com bust. [0] It could easily have been a Cisco product, but there are so many -- far worse -- products out there that nobody outside of telecom has ever dealt with. I lean more toward that. [1] Before someone says "Why didn't you virtualize them?". That was done for some machines with both running inside the isolated network, but it became more of a hassle than it was worth since these machines had to occasionally connect to the devices via serial port. The very buggy software included a very temperamental driver that only worked with a few models of older PCs and IBM (not Lenovo) Thinkpad laptop serial ports ... and only then if one of the wires was cut on the cable. [2] Earlier than the style recently featured in a post about enthusiasts for the brand upgrading the motherboards. [3] https://www.pirelli.com/global/en-ww/homepage https://www.pirelli.com/global/en-ww/homepage ... Yes, that Pirelli. There's crossover between the rubber tires and fiber optics, apparently ... that was news to me. And they had some of the worst software I've ever encountered. [4] We had, I think, 4 in Detroit. Some are not in terribly convenient locations, either. One in the area required passing an expensive OSHA certification, wearing steel toed boots and a hard hat if you wished to access it due to its proximity to a rail yard. I'd been there once and couldn't imagine the need for the restrictions.
- greenleafjacob 9y agoThis is just a failure in change management. They detected the issue in 4 minutes but it took over an hour and a half to mitigate?
- gonzo41 9y ago1 minutes to fix. 30 minutes rationalizing that the 1 minute fix your thinking of is going to work. 45 minutes convincing others. 14 minutes of terrible self doubt and worry. And we're back up. Ops life!
- greenleafjacob 9y agoMitigate first. If you are confident in your change management system then rolling back is always safe.
- SlowBro 9y agoAnd quite possibly a failure in disaster recovery methods. If it was at all possible to have a replicated site it should have been failed over to it immediately. Then resolution and root cause analysis could have been carried over at their leisure. Perhaps that would not have been possible with this style of equipment, I don’t know. But at least yes, change management procedures should have been in place and obeyed. However, I hope they don’t fire the guy. I once read of an employee of some company who created a $600k mistake. They asked the CEO if he would fire the employee. “I just paid $600k to educate one of my employees. Do you think I’m going to give him away to my competitors?” That has stuck with me. Unless there is clearly malicious intent I want to give second chances.
- bonesss 9y agoThat's the conclusion of the report: no one was previously aware that an "unrelated" field in an "unrelated" activity would have production-wide consequences. The detected an issue within 4 minutes, but it took an hour and a half to diagnose the issue, find the unwanted and unforeseen change, and revert it. That's not an absurd amount of time for a complex system with millions of users. I would hazard to guess that this long-running system isn't even taken in as part of the companies 'change management' routines, in so far as it's job was number filtering and the operation was 'routine'. At my work we produce production change reports for production changes, but we don't fill them out every time we run a stable application...
- scotty79 9y agoI always thought that it's a very strange default in SQL: if you don't specify which rows you want, the operation affects everything.
- GrumpyNl 9y agoIts is exactly doing what you tell it to do, update without filter, it updates all.
- yjftsjthsd-h 9y agoOne would expect to have to specify that with an asterisk or something and have blank just give a syntax error.
- mannykannot 9y ago...until one thinks it through and realizes that this would require that special value as an extra condition for every attribute in every relation mentioned in the query, which in turn would eliminate both the theoretical elegance of the relational model and the many very practical benefits that arise from it. A secondary problem is what to do if '*' is a valid value... Ad-hoc solutions are often more difficult than they first seem.
- aexaey 9y agoPercent, rather than asterisk. UPDATE t SET foo='bar' WHERE baz LIKE '$INPUT'; This will do no changes when $INPUT is empty, and only with $INPUT set to '%' will it update every record.
- woliveirajr 9y ago> The FCC report said Level 3 subsequently adopted measures to prevent a recurrence of the problem - measures in accord with best practices. What is the "Best practices" to prevent someone from leaving a blank field, since this field was interpreted as a "*" and blocked everyone ? Would a "send an e-mail telling to never leave this field blank again" enought ?
- organsnyder 9y agoI would hope that they actually added input validation to prevent this from occurring in the future. But it wouldn't surprise me if someone managed to consider sending a "don't do this" email with the high-priority flag set as a "best practice"...
- grecy 9y agoWell, I dunno. When I worked at the Telco that serves all of Northern Canada - the telco that has the largest operating area of any telco in the world, in fact - we had an outage that took out everything for between 1 and 3 days. When I say everything, I mean if you picked up your phone you didn't get a dial tone. Or cell phone. Or internet. Even people that still have 2-way radios to use as phones were out of luck. There is no other provider for every single one of those customers, so not a single person had a single shred of connectivity. In the larger cities they posted Police and EMT Personnel on street corners in case people needed 911. That was a rather big outage, caused by a backup generator not starting during a power outage. We were still trying to bring systems back online a week later that had never been shutdown in 20+ years.
- avs733 9y agoThis type of stuff is terrifying to me. When these incidents occur, people freak out and get angry and demand action...for a while. Then it gets forgotten about. As I've gotten older I've started to separate process driven organizations from progress driven organizations. Process driven ones tend to be very boring, and it can be abused, but good processes trump progress to me everyday.
- abandonliberty 9y ago>Then it gets forgotten about. Then consumers don't want to pay for it and action ceases. You can play those odds for quite a while.
- avs733 9y agounfortunately. I forgot where I read it, but there was a story/quote/interview about how politicians can never run on maintenance because maintenance isn't sexy, they need to run on big projects.
- occams_chainsaw 9y ago> Even people that still have 2-way radios to use as phones were out of luck How would a telecom outage impact two-way radios?
- elil17 9y agoTitle seems misleading - a poorly chosen default behavior caused the outage