4 ms·

The full telephone system, which the author starts with, may not be restartable. Sure, you could restart the SS7 databases and computers, but the control plane

by gumby 3y ago

The full telephone system, which the author starts with, may not be restartable. Sure, you could restart the SS7 databases and computers, but the control plane runs over the data plane, which is configured via...the control plane. Originally the network controls were literally operators (humans), but bit by bit parts were incrementally automated, pulling the system slowly (over decades) by its bootstraps, which were gradually decommissioned as they weren't needed any more. I have a friend who knows a lot about the phone system (he has a security clearance for some of his telephone work). One time we had a long conversation about this topic, until at one point he said "and let's talk about something else" -- I guess from that point some of the details are classified. So maybe there is a plan, or maybe they just designed the system in such a way that they could convince themselves that it would not go down unless things were so severe that loss of the phone system would not be your chief worry. --- In September 2001 there was a full standdown of US airspace. That was accomplished pretty quickly: "you are ordered to land immediately on the closest airport that can handle your aircraft, or be shot down". Undoing that, however, took some careful planning! Fortunately the standdown lasted several days so there was time to work it out. Even if you had a plan for this (and I assume FAA had one), figuring out what the realities on the ground were and matching them up with the plan was nontrivial. Apparently some of the planes landed where they could not take of again unless they were empty with a small amount of fuel to get to an airport designed for them. I don't believe I heard that any planes landed where they could never leave.

protastus 3y ago
> So maybe there is a plan, or maybe they just designed the system in such a way that they could convince themselves that it would not go down unless things were so severe that loss of the phone system would not be your chief worry. My belief from working in very large companies, and (previously) in mission critical systems is that a clean bootstrap and recovery process is extremely unlikely, almost impossible. Because in complex systems full of legacy parts and people who have long retired, the stars won't align. The only way to truly know is to design and periodically test for disaster scenarios (emphasis on the plural). But due to the scale in time and space, cost and bureaucracy, this planning and rehearsing is not going to happen with the desired detail and intensity. People do not seriously plan for things that have never happened. If it does happen, there will be a small group of extremely capable people that will find a way to bootstrap the system. It won't be according to some previously laid out plans -- they will make the plan in real time. They're not famous and probably never will be.
- colechristensen 3y ago
  Eh, would not surprise me in the slightest if there was a secret billion dollar program that specifically practiced disaster recovery for the phone network. The government doesn't have the same motivations as a business and they spend a lot of money in a lot of places just to be prepared for unlikely events. Like we spend billions on military hardware we don't forsee ever needing _to keep the engineering capacity to design and build military hardware_.
  - fluoridation 3y ago
    >Like we spend billions on military hardware we don't forsee ever needing _to keep the engineering capacity to design and build military hardware_. Well, more because the military industrial complex lines the pockets of your politicians, who in turn decide how to spend the budget.
    - runlaszlorun 3y ago
      The two are not mutually exclusive. In fact they reinforce each other.
donalhunt 3y ago
In Ireland, we just build a runway to allow the plane to take off again. https://www.rte.ie/archives/2018/0521/965058-mexican-lands-plane-at-mallow-racecourse/ https://www.rte.ie/archives/2018/0521/965058-mexican-lands-p...
- wongarsu 3y ago
  Clear evidence that news was more entertaining in the 80s: In the merry month of May Just before the dawn of day A plane flew in for Shannon to refuel. Because Shannon is fogged out, Their are the rite of ought To touch down in Cork Airport as a rule. As he flew towards Mallow town His supply of fuel was down But the pilot was as cool as cool could be. In a racetrack west of town He made a safe touch down Just beside the Mallow sugar factory.
  - unmole 3y ago
    Hunt the hare and turn her down the rocky road... Wait, what?
- LoganDark 3y ago
  this is truly one of the takeoffs of all time
- joncrane 3y ago
  And in the meantime they had the pilot judge a beauty contest. What a story!
- myself248 3y ago
  I've long had a fascination with aviation incidents (the Gimli Glider happened on my birthday), but I hadn't heard of this one before! What a great story. Thank you for posting it.
  - psd1 3y ago
    Gimli Glider incident: https://simpleflying.com/gimli-glider/ https://simpleflying.com/gimli-glider/
walrus01 3y ago
If you dig deep enough into the SS7 stuff running in a modern regional ILEC it's way more fragile than you might think. Mostly because it's no longer treated as an absolutely cannot fail thing that is also a primary source of revenue like back in the days when everyone has a POTS line and tons of money came in from long distance bills. Many operators are decommissioning stuff like 5ESS and Nortel switches and moving to modern soft switches as quickly as they can. The network stuff underpinning a lot of critical tdm phone traffic these days is like a collection of 23 year old Cisco 15454 held together by spare parts and a few people who care about them.
- myself248 3y ago
  I've been out of the industry just long enough to remember that Cerent 454's got rebranded as Cisco 15454's right near the end of my tenure... Yow. Way to make a guy feel old! :P
  - walrus01 3y ago
    Mostly I was using the 15454 as an example, there's lots of other 20 to 30 year old stuff out there in the TDM transport sector that's only available on ebay, through weird used equipment dealers, or by finding a decom from another ISP/telco.. Stuff like T1 (or DS0!) to DS3 mux/demux to attach to a 15454, or similar. There's literally 911 call center transport circuits being held together by the telco equivalent of duct tape and string right now, nobody notices until it breaks. One of the weird challenges in building a new state-of-the-art inter city DWDM transport network now is dealing with things like legacy customers that have one OC48 and are unlikely to drop it any time soon, it's a considerable monthly revenue source, and have to deal with stuffing that into the system along with 100Gbps and greater coherent circuits. Also from a customer relations perspective sometimes the customer literally forgets that they have this extremely exensive DS3 or OC48 or something in monthly recurring billing, and you don't want to bring it to the attention of management, because they might go "are we still using this?" and cancel it.
    - LeoPanthera 3y ago
adityaathalye 3y ago
OP here. Thanks for the remarks! That tale is apocryphal to me. I found it amusing, and telling in the sense that disasters are the last thing people think of, and for large enough systems (especially ones that have accreted over human generations) the organisational knowledge has probably been lost to retirement and death. And if you're lucky, maybe a scrap of it has survived. Then one's job is to do the requisite software archaeology to figure out what one's people might have entirely forgotten. Also, if that script remained valid at the time, I doubt it would do any critical actions. It might have been a sort of literal script to follow --- run the script, see what it says, do a thing, run the script, and so forth. Its supposed job was to help humans solve a bootstrap problem. I see how my wording of that passage makes it sound like the be-all end-all of cold booting a telco. But that's what we get when we wall-of-text in our Slacks :D (edit: clarifying remarks)
bobthepanda 3y ago
The 2001 shutdown is even more crazy when you consider that for the FAA administrator who ordered it, it was his first day on the job. Hell of a first day.
- Arrath 3y ago
  Hell of a day to quit sniffing glue.
kotaKat 3y ago
> I don't believe I heard that any planes landed where they could never leave. This has happened before outside of 2001, albeit not really a DR issue but more a political issue -- if we look to the Meigs Field destruction by former Chicago mayor Richard Daley, multiple aircraft were left stranded with a now destroyed runway. (The solution was to just give them special clearance to take off on a taxiway, but still.) https://web.archive.org/web/20110720045652/http://www.aopa.org/whatsnew/newsitems/2003/03-2-014x.html https://web.archive.org/web/20110720045652/http://www.aopa.o...
tinus_hn 3y ago
You know you can actually bring fuel to a plane and refuel it wherever it is? And that’s probably the answer to the first question, it can be done but you you have to have someone who knows what he’s doing think outside the box a little.

by gumby 3y ago

protastus 3y ago

> So maybe there is a plan, or maybe they just designed the system in such a way that they could convince themselves that it would not go down unless things were so severe that loss of the phone system would not be your chief worry. My belief from working in very large companies, and (previously) in mission critical systems is that a clean bootstrap and recovery process is extremely unlikely, almost impossible. Because in complex systems full of legacy parts and people who have long retired, the stars won't align. The only way to truly know is to design and periodically test for disaster scenarios (emphasis on the plural). But due to the scale in time and space, cost and bureaucracy, this planning and rehearsing is not going to happen with the desired detail and intensity. People do not seriously plan for things that have never happened. If it does happen, there will be a small group of extremely capable people that will find a way to bootstrap the system. It won't be according to some previously laid out plans -- they will make the plan in real time. They're not famous and probably never will be.

colechristensen 3y ago
Eh, would not surprise me in the slightest if there was a secret billion dollar program that specifically practiced disaster recovery for the phone network. The government doesn't have the same motivations as a business and they spend a lot of money in a lot of places just to be prepared for unlikely events. Like we spend billions on military hardware we don't forsee ever needing _to keep the engineering capacity to design and build military hardware_.
- fluoridation 3y ago
  >Like we spend billions on military hardware we don't forsee ever needing _to keep the engineering capacity to design and build military hardware_. Well, more because the military industrial complex lines the pockets of your politicians, who in turn decide how to spend the budget.
  - runlaszlorun 3y ago
    The two are not mutually exclusive. In fact they reinforce each other.

donalhunt 3y ago

In Ireland, we just build a runway to allow the plane to take off again. https://www.rte.ie/archives/2018/0521/965058-mexican-lands-plane-at-mallow-racecourse/ https://www.rte.ie/archives/2018/0521/965058-mexican-lands-p...

wongarsu 3y ago
Clear evidence that news was more entertaining in the 80s: In the merry month of May Just before the dawn of day A plane flew in for Shannon to refuel. Because Shannon is fogged out, Their are the rite of ought To touch down in Cork Airport as a rule. As he flew towards Mallow town His supply of fuel was down But the pilot was as cool as cool could be. In a racetrack west of town He made a safe touch down Just beside the Mallow sugar factory.
- unmole 3y ago
  Hunt the hare and turn her down the rocky road... Wait, what?
LoganDark 3y ago
this is truly one of the takeoffs of all time
joncrane 3y ago
And in the meantime they had the pilot judge a beauty contest. What a story!
myself248 3y ago
I've long had a fascination with aviation incidents (the Gimli Glider happened on my birthday), but I hadn't heard of this one before! What a great story. Thank you for posting it.
- psd1 3y ago
  Gimli Glider incident: https://simpleflying.com/gimli-glider/ https://simpleflying.com/gimli-glider/

walrus01 3y ago

If you dig deep enough into the SS7 stuff running in a modern regional ILEC it's way more fragile than you might think. Mostly because it's no longer treated as an absolutely cannot fail thing that is also a primary source of revenue like back in the days when everyone has a POTS line and tons of money came in from long distance bills. Many operators are decommissioning stuff like 5ESS and Nortel switches and moving to modern soft switches as quickly as they can. The network stuff underpinning a lot of critical tdm phone traffic these days is like a collection of 23 year old Cisco 15454 held together by spare parts and a few people who care about them.

myself248 3y ago
I've been out of the industry just long enough to remember that Cerent 454's got rebranded as Cisco 15454's right near the end of my tenure... Yow. Way to make a guy feel old! :P
- walrus01 3y ago
  Mostly I was using the 15454 as an example, there's lots of other 20 to 30 year old stuff out there in the TDM transport sector that's only available on ebay, through weird used equipment dealers, or by finding a decom from another ISP/telco.. Stuff like T1 (or DS0!) to DS3 mux/demux to attach to a 15454, or similar. There's literally 911 call center transport circuits being held together by the telco equivalent of duct tape and string right now, nobody notices until it breaks. One of the weird challenges in building a new state-of-the-art inter city DWDM transport network now is dealing with things like legacy customers that have one OC48 and are unlikely to drop it any time soon, it's a considerable monthly revenue source, and have to deal with stuffing that into the system along with 100Gbps and greater coherent circuits. Also from a customer relations perspective sometimes the customer literally forgets that they have this extremely exensive DS3 or OC48 or something in monthly recurring billing, and you don't want to bring it to the attention of management, because they might go "are we still using this?" and cancel it.
  - LeoPanthera 3y ago

adityaathalye 3y ago

OP here. Thanks for the remarks! That tale is apocryphal to me. I found it amusing, and telling in the sense that disasters are the last thing people think of, and for large enough systems (especially ones that have accreted over human generations) the organisational knowledge has probably been lost to retirement and death. And if you're lucky, maybe a scrap of it has survived. Then one's job is to do the requisite software archaeology to figure out what one's people might have entirely forgotten. Also, if that script remained valid at the time, I doubt it would do any critical actions. It might have been a sort of literal script to follow --- run the script, see what it says, do a thing, run the script, and so forth. Its supposed job was to help humans solve a bootstrap problem. I see how my wording of that passage makes it sound like the be-all end-all of cold booting a telco. But that's what we get when we wall-of-text in our Slacks :D (edit: clarifying remarks)

bobthepanda 3y ago

The 2001 shutdown is even more crazy when you consider that for the FAA administrator who ordered it, it was his first day on the job. Hell of a first day.

Arrath 3y ago
Hell of a day to quit sniffing glue.

kotaKat 3y ago

> I don't believe I heard that any planes landed where they could never leave. This has happened before outside of 2001, albeit not really a DR issue but more a political issue -- if we look to the Meigs Field destruction by former Chicago mayor Richard Daley, multiple aircraft were left stranded with a now destroyed runway. (The solution was to just give them special clearance to take off on a taxiway, but still.) https://web.archive.org/web/20110720045652/http://www.aopa.org/whatsnew/newsitems/2003/03-2-014x.html https://web.archive.org/web/20110720045652/http://www.aopa.o...

tinus_hn 3y ago

You know you can actually bring fuel to a plane and refuel it wherever it is? And that’s probably the answer to the first question, it can be done but you you have to have someone who knows what he’s doing think outside the box a little.