6 ms·

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "e

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for the original exchange to come back up, receive the crashy event back, and crash again. The whole network was full of nodes crashing, causing their peers to crash, ad infinitum. In order to bring the network back up, they needed to either take everything down at the same time (and make sure all the queues are emptied), but even that wouldn't have made it stable, because a similar "patient 0" event would have brought the whole network down. Once the problem was understood, they reverted to an earlier version which didn't have the bug, and the network re-stabilized. The lore I grew up on is that this specific event was very significant in pushing and funding research into robust distributed systems, of which the best known result is Erlang and its ecosystem - originally built, and still mostly used, to make sure that phone exchanges don't break. [0] https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...

immigrantsheep 6y ago
Is that related to the hacker's crackdown?
chrisweekly 6y ago
Fascinating. Thanks for sharing! :)
swinglock 6y ago
Indeed. In 2018 an Erlang telco software did break, bringing down the UK and Japan.
- pcc 6y ago
  If memory serves that also involved an expired certificate
  - swinglock 6y ago
    That matches my memory.
mcspiff 6y ago
There was something similar a few years ago on a large US mobile network. You could watch the ‘storm’ rolling across the map. Fascinating stuff
- throw0101a 6y ago
  Are you referring to CenturyLink’s 37-hour, nationwide outage? > In this instance, the malformed packets [Ethernet frames?] included fragments of valid network management packets that are typically generated. Each malformed packet shared four attributes that contributed to the outage: 1) a broadcast destination address, meaning that the packet was directed to be sent to all connected devices; 2) a valid header and valid checksum; 3) no expiration time, meaning that the packet would not be dropped for being created too long ago; and 4) a size larger than 64 bytes. * https://arstechnica.com/information-technology/2019/08/centurylinks-37-hour-outage-blocked-911-service-for-17-million-people/ https://arstechnica.com/information-technology/2019/08/centu...
phkahler 6y ago
Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...
- pacificmint 6y ago
  Probably not the book you are thinking off, since it’s just about the AT&T incident, but “The Day the Phones Stopped Ringing” by Leonard Lee is a detailed description of the event. It’s been many years since I read it, but I recall it being a very interesting read.
- tinco 6y ago
  For some reason in my university almost every CS class would start with an anecdote about the Therac 25, Ariane V, and/or a couple others as a motivation on why we the class existed. It was sort of a meme. The lessons are definitely still taught, I don't know if they're actually learned of course.. And who knows who actually taught the 737-Max software devs, I don't suppose they're fresh out of uni.
  - TheSpiceIsLife 6y ago
    Do management typical typically study Computer Science?
    - tinco 6y ago
      Typically yes actually, the director of engineering should always be an engineer. Of course, these are hardware companies so it would probably be some kind of hardware engineer.
      - TheSpiceIsLife 6y ago
        Should. Sure.
peterwwillis 6y ago
I have spent hours and hours banging my head against Erlang distributed system bugs in production. I am absolutely mystified why anyone thought just using a particular programming language would prevent these scenarios. If it's Turing-complete, expect the unexpected.
- BoorishBears 6y ago
  The idea isn't that Erlang is infallible in the design of distributed systems. The idea is it takes away enough foot-guns that if you're banging your head against systems written it in, you'd be banging your head even harder and more often if the same implementor had used another language
hinkley 6y ago
I think we used to call that a poison pill message (still bring it up routinely when we talk about load balancing and why infinite retries are a very, very bad idea).
- mikelward 6y ago
  Some queue processing systems I've seen have infinite retries. At least they have exponential backoff I guess.
  - hinkley 6y ago
    But your queue will grow and grow and the fraction of time you spend servicing old messages grows and grows. Not a terribly big fan of these queueing systems. People always seem to bung things up in ways they are not quite equipped to fix (in the “you are not smart enough to debug the code you wrote” sense). Last time I had to help someone with such a situation, we discovered that the duplicate processing problem had existed for >3 months prior to the crisis event, and had been consuming 10% of the system capacity, which was just low enough that nobody noticed.
    - mikelward 6y ago
      We also alert if any message is in the queue too long. If anything, the alert is too sensitive.
cat199 6y ago
> best known result is Erlang and its ecosystem not expert but erlang is listed as 1986, so that would seem not directly related https://en.wikipedia.org/wiki/Erlang_(programming_language) https://en.wikipedia.org/wiki/Erlang_(programming_language)
twic 6y ago
This reminds me of an incident on the early internet (perhaps ARPANET at that point) where a routing table got corrupted so it had a negative-length route which routers then propagated to each other, even after the original corrupt router was rebooted. As with AT&T, they had to reboot all the routers at once to get rid of the corruption. I can't remember where i read about this, but i recall the problem was called "The Creeping Crud from California". Sadly, this phrase apparently does not appear anywhere on the internet. Did i imagine this?
- ficklepickle 6y ago
  I can't find anything by that name either, but the details do match the major ARPANET outage of Oct 27, 1980. The incident is detailed in RFC 789: http://www.faqs.org/rfcs/rfc789.html#b http://www.faqs.org/rfcs/rfc789.html#b
  - twic 6y ago
    Interesting, thanks! That is different to the story i remember, but it's possible that i remember incorrectly, or read an incorrect explanation. I believe that i read about this episode in Hans Moravec's book 'Mind Children'. I can see in Google Books that chapter 5 is on 'Wildlife', and there is a section 'Spontaneous Generation', which promises to talk about a "software parasite" which emerged naturally in the ARPAnet - but of which the bulk is not available: https://books.google.co.uk/books?id=56mb7XuSx3QC&lpg=PA133&dq=moravec%20arpanet&pg=PA133#v=onepage&q=spontaneous%20generation&f=false https://books.google.co.uk/books?id=56mb7XuSx3QC&lpg=PA133&d...
dadver 6y ago
This sounds like the event that is described in the book Masters of Deception: The gang that ruled cyberspace. The way I remember it the book attributes the incident to MoD, while of course still being the result of a bug/faulty design.
doctorshady 6y ago
The thing with feature group D trunks to the long distance network is you could (and still can on non-IP/mobile networks) manually route to another long distance carrier like Verizon, and sidestep the outage from the subscriber end, full stop. That's certainly not possible with any of the contemporary internet outages.
- p_l 6y ago
  you can inject changes in routing, but if the other other carrier doesn't route around the affected network, you're back to square one. That's part of why Level3/CenturyLink was depeered and why several prefixes that are normally announced through it were quickly rerouted by owners.
  - doctorshady 6y ago
    That's my point; as a subscriber, you can prefix a long distance call with a routing code to avoid, for example, a shut down long distance network without any administrator changes. Routing to the long distance networks is done independently through the local network, so if AT&T's long distance network was having issues, it'd have no impact on your ability to access Verizon's long distance network.
    - a1369209993 6y ago
      There's actually no technical reason why you couldn't do that with IP (4 or 6); although you'd need a approriately located host to be running a relay daemon[0]. 0: ie something that takes, say, a UDP packet on port NNNN containing a whole raw IPv4 packet, throws away the wrapping, and drops the IPv4 packet onto its own network interface. This is safe - the packet must shrink by a dozen or two bytes with each retransmission - but usually not actually set up anywhere. Edit: It probably wouldn't work for TCP though - maybe try TOR?
      - p_l 6y ago
        There is, technically, a way for IP packets to signify preferred routes, but due to other (security) reasons it's disabled.
sbmthakur 6y ago
A thread discussing that event: https://news.ycombinator.com/item?id=24323412 https://news.ycombinator.com/item?id=24323412

immigrantsheep 6y ago

Is that related to the hacker's crackdown?

chrisweekly 6y ago

Fascinating. Thanks for sharing! :)

swinglock 6y ago

Indeed. In 2018 an Erlang telco software did break, bringing down the UK and Japan.

pcc 6y ago
If memory serves that also involved an expired certificate
- swinglock 6y ago
  That matches my memory.

mcspiff 6y ago

There was something similar a few years ago on a large US mobile network. You could watch the ‘storm’ rolling across the map. Fascinating stuff

throw0101a 6y ago
Are you referring to CenturyLink’s 37-hour, nationwide outage? > In this instance, the malformed packets [Ethernet frames?] included fragments of valid network management packets that are typically generated. Each malformed packet shared four attributes that contributed to the outage: 1) a broadcast destination address, meaning that the packet was directed to be sent to all connected devices; 2) a valid header and valid checksum; 3) no expiration time, meaning that the packet would not be dropped for being created too long ago; and 4) a size larger than 64 bytes. * https://arstechnica.com/information-technology/2019/08/centurylinks-37-hour-outage-blocked-911-service-for-17-million-people/ https://arstechnica.com/information-technology/2019/08/centu...

phkahler 6y ago

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

pacificmint 6y ago
Probably not the book you are thinking off, since it’s just about the AT&T incident, but “The Day the Phones Stopped Ringing” by Leonard Lee is a detailed description of the event. It’s been many years since I read it, but I recall it being a very interesting read.
tinco 6y ago
For some reason in my university almost every CS class would start with an anecdote about the Therac 25, Ariane V, and/or a couple others as a motivation on why we the class existed. It was sort of a meme. The lessons are definitely still taught, I don't know if they're actually learned of course.. And who knows who actually taught the 737-Max software devs, I don't suppose they're fresh out of uni.
- TheSpiceIsLife 6y ago
  Do management typical typically study Computer Science?
  - tinco 6y ago
    Typically yes actually, the director of engineering should always be an engineer. Of course, these are hardware companies so it would probably be some kind of hardware engineer.
    - TheSpiceIsLife 6y ago
      Should. Sure.

peterwwillis 6y ago

I have spent hours and hours banging my head against Erlang distributed system bugs in production. I am absolutely mystified why anyone thought just using a particular programming language would prevent these scenarios. If it's Turing-complete, expect the unexpected.

BoorishBears 6y ago
The idea isn't that Erlang is infallible in the design of distributed systems. The idea is it takes away enough foot-guns that if you're banging your head against systems written it in, you'd be banging your head even harder and more often if the same implementor had used another language

hinkley 6y ago

I think we used to call that a poison pill message (still bring it up routinely when we talk about load balancing and why infinite retries are a very, very bad idea).

mikelward 6y ago
Some queue processing systems I've seen have infinite retries. At least they have exponential backoff I guess.
- hinkley 6y ago
  But your queue will grow and grow and the fraction of time you spend servicing old messages grows and grows. Not a terribly big fan of these queueing systems. People always seem to bung things up in ways they are not quite equipped to fix (in the “you are not smart enough to debug the code you wrote” sense). Last time I had to help someone with such a situation, we discovered that the duplicate processing problem had existed for >3 months prior to the crisis event, and had been consuming 10% of the system capacity, which was just low enough that nobody noticed.
  - mikelward 6y ago
    We also alert if any message is in the queue too long. If anything, the alert is too sensitive.

cat199 6y ago

> best known result is Erlang and its ecosystem not expert but erlang is listed as 1986, so that would seem not directly related https://en.wikipedia.org/wiki/Erlang_(programming_language) https://en.wikipedia.org/wiki/Erlang_(programming_language)

twic 6y ago

This reminds me of an incident on the early internet (perhaps ARPANET at that point) where a routing table got corrupted so it had a negative-length route which routers then propagated to each other, even after the original corrupt router was rebooted. As with AT&T, they had to reboot all the routers at once to get rid of the corruption. I can't remember where i read about this, but i recall the problem was called "The Creeping Crud from California". Sadly, this phrase apparently does not appear anywhere on the internet. Did i imagine this?

ficklepickle 6y ago
I can't find anything by that name either, but the details do match the major ARPANET outage of Oct 27, 1980. The incident is detailed in RFC 789: http://www.faqs.org/rfcs/rfc789.html#b http://www.faqs.org/rfcs/rfc789.html#b
- twic 6y ago
  Interesting, thanks! That is different to the story i remember, but it's possible that i remember incorrectly, or read an incorrect explanation. I believe that i read about this episode in Hans Moravec's book 'Mind Children'. I can see in Google Books that chapter 5 is on 'Wildlife', and there is a section 'Spontaneous Generation', which promises to talk about a "software parasite" which emerged naturally in the ARPAnet - but of which the bulk is not available: https://books.google.co.uk/books?id=56mb7XuSx3QC&lpg=PA133&dq=moravec%20arpanet&pg=PA133#v=onepage&q=spontaneous%20generation&f=false https://books.google.co.uk/books?id=56mb7XuSx3QC&lpg=PA133&d...

dadver 6y ago

This sounds like the event that is described in the book Masters of Deception: The gang that ruled cyberspace. The way I remember it the book attributes the incident to MoD, while of course still being the result of a bug/faulty design.

doctorshady 6y ago

The thing with feature group D trunks to the long distance network is you could (and still can on non-IP/mobile networks) manually route to another long distance carrier like Verizon, and sidestep the outage from the subscriber end, full stop. That's certainly not possible with any of the contemporary internet outages.

p_l 6y ago
you can inject changes in routing, but if the other other carrier doesn't route around the affected network, you're back to square one. That's part of why Level3/CenturyLink was depeered and why several prefixes that are normally announced through it were quickly rerouted by owners.
- doctorshady 6y ago
  That's my point; as a subscriber, you can prefix a long distance call with a routing code to avoid, for example, a shut down long distance network without any administrator changes. Routing to the long distance networks is done independently through the local network, so if AT&T's long distance network was having issues, it'd have no impact on your ability to access Verizon's long distance network.
  - a1369209993 6y ago
    There's actually no technical reason why you couldn't do that with IP (4 or 6); although you'd need a approriately located host to be running a relay daemon[0]. 0: ie something that takes, say, a UDP packet on port NNNN containing a whole raw IPv4 packet, throws away the wrapping, and drops the IPv4 packet onto its own network interface. This is safe - the packet must shrink by a dozen or two bytes with each retransmission - but usually not actually set up anywhere. Edit: It probably wouldn't work for TCP though - maybe try TOR?
    - p_l 6y ago
      There is, technically, a way for IP packets to signify preferred routes, but due to other (security) reasons it's disabled.

sbmthakur 6y ago

A thread discussing that event: https://news.ycombinator.com/item?id=24323412 https://news.ycombinator.com/item?id=24323412