5 ms·
All Circuits are Busy Now: The 1990 AT&T Long Distance Network Collapse (1995)
- hexbus 4y agoBreak statements contained in an if-then-else loop are a bad idea.
- phkahler 4y agoSkipping the full test suite is also a bad idea.
- acadiel 4y agoI think the article mentioned they were fanatical about their test suite - but this somehow still slipped through. I wonder if peer review would have caught it.
- gumby 4y agoI don’t think that was the case in this situation
- DangitBobby 4y agoIt can be hard to test what you didn't anticipate!
- carl_sandland 4y agowould complete code coverage from tests of this line found the problem, or was it temporal somehow ?
- gumby 4y agoI remember that outage. It was finally blamed (as described in this brief) on phone switch manufacturer DSC. IIRC this killed the company. Their SLA with their customers was something like three minutes of downtime per decade. DSC was our customer at Cygnus. They were interesting as a customer (tough requirements but they paid a lot for them). For example if they reported a bug and got a fix they diffed the binaries and looked at every difference to be sure that the change was a result of the fix, and nothing else (no, they didn’t want upgrades).
- JosephRedfern 4y ago> For example if they reported a bug and got a fix they diffed the binaries and looked at every difference to be sure that the change was a result of the fix, and nothing else (no, they didn’t want upgrades). That sounds pretty diligent!
- yunohn 4y ago> Their SLA with their customers was something like three minutes of downtime per decade. That is insane. I really feel like modern SLAs are only getting worse - so much so that most companies fudge them, and try their hardest to never declare any sort of outage.
- wincy 4y agoWhat do you mean, Microsoft 358 is an excellent product that has uptime for exactly what they’ve got on the label!
- deleted 4y ago[deleted]
- gumby 4y agoThe Bell System standard was an extremely high Erlang number, which of course they strove to provide at the lowest cost which meant extremely high utilization rate on hardware which in turn meant extreme uptime (compare to the then contemporary PTT QoS even in major economies like France). This is also why the software itself was designed with so many internal defenses and what I would consider an “immune system”. I’ve never seen anything like it even on an aircraft control system. That is mentioned in passing in the brief article but is easily missed if you don’t know what it’s referring to. Most of what is done on the Internet at, say, “layer 5 or above” isn’t at all important so there’s no need for this level of SLA, but the actual backbone carriers do still carry SLAs at around that level. With packet switching it’s easier for them to provide than it was in the days of the 4ESS and 5ESS.
- twisteriffic 4y ago> Clearly, the use of C programs and compilers contributed to the breakdown. A more structured programming language with stricter compilers would have made this particular defect much more obvious. Nice to see that "should have used Rust" has been a thing since before Rust existed.
- EGreg 4y agoShould have used Q !
- conjectureproof 4y agoWhy? I briefly had an interest in learning Q, then looked at some code: https://github.com/KxSystems/cookbook/blob/master/start/buildhdb.q https://github.com/KxSystems/cookbook/blob/master/start/buil... Why not just build what you need with C/arrow/parquet?
- EGreg 4y agoNot that Q. The one I am talking about hasn’t been released yet!!
- bee_rider 4y agoI’m sure people have been saying “shouldn’t have used C” for longer than most of us here have been alive.
- kccqzy 4y agoDo you seriously think this kind of logic error would have been prevented in Rust?
- IntelMiner 4y agoTheir quip was a playful jab at the "rewrite every possible thing in Rust" mentality
- shrubble 4y ago
- mdmglr 4y agoSo what was the fix? Remove lines 9-10? Or do “set up pointers to optional parameters” before break?
- vhold 4y agoThis is an example of why you want interoperable diversity in complex distributed systems. By having everything so standardized and consistent, they had the exact same failure mode everywhere and lost redundant fault tolerance. If they had different interoperable switches, running different software, the outage wouldn't have been absolute. When large complex distributed systems grow organically over time, they tend to wind up with diversity. It usually takes a big centralized project focused on efficiency to destroy that property.
- yusyusyus 4y agoI appreciate this comment. In my world of packet pushing, I try to promote vendor diversity for this reason. The practical downsides of this diversity live in the complexity of the interop (often slowing feature velocity), operations, and procurement/support. But issues like the AT&T 4ESS outage have occurred before in IP networks, as an example, in some BGP bug. Diversity alleviates some of the global impact.
- vlovich123 4y agoThere are other ways of accomplishing this like doing staged rollouts without giving up the cost efficiencies of implementing your own network only once and avoiding a combinatorial explosion in testing complexity. You can sometimes play this game with vendors because you want them to give you an interoperable interface so that you avoid vendor lock-in and have better pricing, but that’s a secondary benefit and staged rollouts should still be performed even if you have heterogenous software.
- kortilla 4y agoStaged rollouts do not protect you from long lurking bugs. Even in this ATT case they most certainly did do a staged rollout just because they couldn’t just shut off the entire phone network to run an update across all systems.
- a-dub 4y agothese days i think the remediation would be "fuzz test the timing of critical network protocols to find nonobvious edge cases and state machine implementation faults"
- pacificmint 4y agoIf some one wants to read a lot more details an out this incident, there is book about it. It’s been a decade or two since I read it, but I remember it being well written. ‘The Day the Phones Stopped Ringing’ by Leonard Lee
- acadiel 4y ago> ‘The Day the Phones Stopped Ringing’ by Leonard Lee Definitely going to look into this! The whole *ESS architecture still underpins a lot of the telephony system. There are quite a few still running, even though other TDM equipment is being phased out.
- pifm_guy 4y agoObviously you do everything possible to stop an outage like this happening... But when it inevitably does, you should be prepared for a full system simultaneous restart. Ie. So that no 'bad' signals or data from the old system can impact the new. That is the sort of thing you should practice in the staging environment from time to time, just for when it might be needed. It could have taken this outage from many hours down to just many minutes.
- pifm_guy 4y agoYou should also design all your code to be rollbackable... But for the very rare case that a rollback won't solve the problem (eg. An outage is caused by changes outside your organisation's control), you also need to be able to do a rapid code change, recompile and push. Many companies aren't able to do this for example their release process involves multiple days worth of interlocked manual steps. Don't get yourself in that position.
- anonymousiam 4y agoNot all that different from the root cause of this (more costly) disaster: http://sunnyday.mit.edu/nasa-class/Ariane5-report.html http://sunnyday.mit.edu/nasa-class/Ariane5-report.html https://news.ycombinator.com/item?id=30556601 https://news.ycombinator.com/item?id=30556601
- segfaultbuserr 4y agoIt's funny that what caused the AT&T telephone switch system to break was a "break" statement in a "switch" control structure.