5 ms·

All Circuits Are Busy Now: The 1990 AT&T Long Distance Network Collapse (1995)

132 points by beagle3 6y ago

macintux 6y ago
This presumably was pulled from the Level 3 outage discussion. Relevant comment: https://news.ycombinator.com/item?id=24323402 https://news.ycombinator.com/item?id=24323402 phkahler's addendum should be painfully familiar to many: > Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. https://news.ycombinator.com/item?id=24323904 https://news.ycombinator.com/item?id=24323904
wolfgang42 6y ago
Longer-form article from 1992, which talks about the history of the phone system and covers this event: https://stuff.mit.edu/hacker/part1.html https://stuff.mit.edu/hacker/part1.html > On January 15, 1990, AT&T's long-distance telephone switching system crashed. > This was a strange, dire, huge event. Sixty thousand people lost their telephone service completely. During the nine long hours of frantic effort that it took to restore service, some seventy million telephone calls went uncompleted. > Losses of service, known as "outages" in the telco trade, are a known and accepted hazard of the telephone business. Hurricanes hit, and phone cables get snapped by the thousands. Earthquakes wrench through buried fiber-optic lines. Switching stations catch fire and burn to the ground. These things do happen. There are contingency plans for them, and decades of experience in dealing with them. But the Crash of January 15 was unprecedented. It was unbelievably huge, and it occurred for no apparent physical reason.
- vimy 6y ago
  ‘Outage’ wasn’t a commonly known word?
  - bobbylarrybobby 6y ago
    Oddly, no. I’m as surprised as you are. https://www.etymonline.com/word/outage#etymonline_v_9966 https://www.etymonline.com/word/outage#etymonline_v_9966
  - adrianmonk 6y ago
    I don't think it was uncommon, at least not for power outages. Here's a Washington Post article from 1977 that uses it: https://www.washingtonpost.com/archive/politics/1977/07/15/blackout-paralyzes-new-york-city-for-day/eb6f9906-db2c-4561-9ca9-f3d054f01c08/ https://www.washingtonpost.com/archive/politics/1977/07/15/b... And a NY Times article from later in 1977: https://www.nytimes.com/1977/10/17/archives/around-the-nation-fire-at-electricity-plant-blacks-out-san-antonio.html https://www.nytimes.com/1977/10/17/archives/around-the-natio...
at-fates-hands 6y ago
Interestingly enough, this was initially blamed on the Legion of Doom hacker group: It was a frightening outage that rang alarm bells in more places than just AT&T headquarters. Months earlier, the Secret Service had arrested a 16-year-old hacker nicknamed Fry Guy who had told them, under interrogation, that a hacker gang he was associated with had been planning to crash the system on a national holiday. That gang, the Legion of Doom, was suddenly on the law enforcement radar screen in a big way. But they weren’t alone. The Texas-centric group was in the midst of an online feud with a New York-based rival, the Masters of Deception. LOD and MOD, as they were known, were at each others throats and the battleground happened to run through the phone switches and computer systems of America. From the article: https://www.cybersecuritymastersdegree.org/legion-of-doom-vs-the-masters-of-deception/ https://www.cybersecuritymastersdegree.org/legion-of-doom-vs...
alrs 6y ago
It felt that monumental at the time. I recorded the evening news to VHS that night.
- rglover 6y ago
  Do you still have the tape?
  - alrs 6y ago
    Unlikely.
pgrote 6y ago
It was front page of the NY Times. https://www.nytimes.com/1990/01/17/us/at-t-pinpoints-cause-of-phone-disruption.html?searchResultPosition=1 https://www.nytimes.com/1990/01/17/us/at-t-pinpoints-cause-o... "Company executives said the malfuction occurred in a program at a giant computer switching station in the New York metropolitan area, causing the computer to send out alarm messages to other switching stations." Interesting to see MCI taking the opportunity to talk about backup providers.
monkpit 6y ago
Why are there so many typos...?
- empthought 6y ago
  Optical character recognition.
  - monkpit 6y ago
    Makes sense! Thanks!
jimmaswell 6y ago
I'm not sure I really get the bug with the break statement. What should the code have looked like instead?

132 points by beagle3 6y ago

This presumably was pulled from the Level 3 outage discussion. Relevant comment: https://news.ycombinator.com/item?id=24323402 https://news.ycombinator.com/item?id=24323402 phkahler's addendum should be painfully familiar to many: > Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. https://news.ycombinator.com/item?id=24323904 https://news.ycombinator.com/item?id=24323904

wolfgang42 6y ago

Longer-form article from 1992, which talks about the history of the phone system and covers this event: https://stuff.mit.edu/hacker/part1.html https://stuff.mit.edu/hacker/part1.html > On January 15, 1990, AT&T's long-distance telephone switching system crashed. > This was a strange, dire, huge event. Sixty thousand people lost their telephone service completely. During the nine long hours of frantic effort that it took to restore service, some seventy million telephone calls went uncompleted. > Losses of service, known as "outages" in the telco trade, are a known and accepted hazard of the telephone business. Hurricanes hit, and phone cables get snapped by the thousands. Earthquakes wrench through buried fiber-optic lines. Switching stations catch fire and burn to the ground. These things do happen. There are contingency plans for them, and decades of experience in dealing with them. But the Crash of January 15 was unprecedented. It was unbelievably huge, and it occurred for no apparent physical reason.

vimy 6y ago
‘Outage’ wasn’t a commonly known word?
- bobbylarrybobby 6y ago
  Oddly, no. I’m as surprised as you are. https://www.etymonline.com/word/outage#etymonline_v_9966 https://www.etymonline.com/word/outage#etymonline_v_9966
- adrianmonk 6y ago
  I don't think it was uncommon, at least not for power outages. Here's a Washington Post article from 1977 that uses it: https://www.washingtonpost.com/archive/politics/1977/07/15/blackout-paralyzes-new-york-city-for-day/eb6f9906-db2c-4561-9ca9-f3d054f01c08/ https://www.washingtonpost.com/archive/politics/1977/07/15/b... And a NY Times article from later in 1977: https://www.nytimes.com/1977/10/17/archives/around-the-nation-fire-at-electricity-plant-blacks-out-san-antonio.html https://www.nytimes.com/1977/10/17/archives/around-the-natio...

at-fates-hands 6y ago

Interestingly enough, this was initially blamed on the Legion of Doom hacker group: It was a frightening outage that rang alarm bells in more places than just AT&T headquarters. Months earlier, the Secret Service had arrested a 16-year-old hacker nicknamed Fry Guy who had told them, under interrogation, that a hacker gang he was associated with had been planning to crash the system on a national holiday. That gang, the Legion of Doom, was suddenly on the law enforcement radar screen in a big way. But they weren’t alone. The Texas-centric group was in the midst of an online feud with a New York-based rival, the Masters of Deception. LOD and MOD, as they were known, were at each others throats and the battleground happened to run through the phone switches and computer systems of America. From the article: https://www.cybersecuritymastersdegree.org/legion-of-doom-vs-the-masters-of-deception/ https://www.cybersecuritymastersdegree.org/legion-of-doom-vs...

alrs 6y ago

It felt that monumental at the time. I recorded the evening news to VHS that night.

rglover 6y ago
Do you still have the tape?
- alrs 6y ago
  Unlikely.

pgrote 6y ago

It was front page of the NY Times. https://www.nytimes.com/1990/01/17/us/at-t-pinpoints-cause-of-phone-disruption.html?searchResultPosition=1 https://www.nytimes.com/1990/01/17/us/at-t-pinpoints-cause-o... "Company executives said the malfuction occurred in a program at a giant computer switching station in the New York metropolitan area, causing the computer to send out alarm messages to other switching stations." Interesting to see MCI taking the opportunity to talk about backup providers.

monkpit 6y ago

Why are there so many typos...?

empthought 6y ago
Optical character recognition.
- monkpit 6y ago
  Makes sense! Thanks!

jimmaswell 6y ago

I'm not sure I really get the bug with the break statement. What should the code have looked like instead?