5 ms·

Why 40-Year-Old Tech Is Still Running America's Air Traffic Control

58 points by ghosh 12y ago

SocksCanClose 12y ago
ugh: "THE UPGRADED SYSTEM WILL GO ONLINE THIS SPRING, FIVE YEARS LATE AND AT LEAST $500M OVER BUDGET." #
- cmurf 12y ago
  Before Lockhead Martin's work trying to modernize, IBM had a billions dollar contract in the 80's. http://sebokwiki.org/wiki/FAA_Advanced_Automation_System_%28AAS%29_Vignette http://sebokwiki.org/wiki/FAA_Advanced_Automation_System_%28... http://articles.latimes.com/1988-08-11/business/fi-288_1_air-traffic-control-equipment http://articles.latimes.com/1988-08-11/business/fi-288_1_air... Basically, none of these huge companies want to do this kind of work unless the pay is obscene. And then once it's obscene, the companies all fight over it and throw each other under the bus. Court battles aren't even the half of it. They hire lobbyists, contribute to campaigns, and every 2-8 years there's some amount of government shuffling. So nothing is actually ever very consistent within the government. New politicians, new bureaucrats, new policies, new ideas (they think anyway), and they basically have been scrapping previously done work and starting over multiple times. And it's been longer than 40 years. Perversely, running on vacuum tubes makes the old system a more security hardened system than what we're likely to end up with. We should hope the NSA has hacked the FAA and ATC as a defensive measure.
technofiend 12y ago
I worked on the shuttle program for a while and was debugging some fortran code that called honeywell assembly as part of a rewrite of the DOLILU program. It calculated stress on the shuttles frame based on winds aloft data collected by balloon on launch day. The code passed in a jump parameter and the assembly jumped back to the appropriate fortran label. Over the years, all the exits except one had been coded out. I asked my boss if we could just make the fortran code jump directly from point A to point B since that's all the assembly did. He answered my 6 months contract would be done before the change made it through committee. The thing about process is people rarely lessen or remove it once it's in place. But it's possible to do a classic waterfall project on time and on budget, look at any bank release of ATM software. Of course losing an orbiter or losing a plane isn't the same class of problem as losing a transaction, but it is possible to do the projects properly. The fact the Obama administration had to bring in outside help to rescue their most recent major rollout (Obamacare insurance info websites) tells you there's no incentive to get it right or on time in these contracts. You just keep working and the bills keep getting paid. These contracts need real oversight and financial penalties for failure to meet deadlines. Assuming of course they also include some iron-clad agreements about requirements not changing.
- dasil003 12y ago
  I'm not saying your wrong, but shuttle software and even ATMs are different from Obamacare. The former has specific operational requirements that are relatively clear. Health care by contrast is a clusterfuck of legislation piled on top a multitude of tangential legacy systems. I don't doubt that the talent to get it right exists, but it requires more juice than just an engineer saying "this thing won't fly". The problem is that legislation is itself code, but it is written by people who have no nuanced knowledge of the downstream implications, and perhaps more importantly, no incentive to ever refactor.
  - dba7dba 12y ago
    Agree. Physics doesn't lie. But Politicians lie (or don't know what they are talking about OR pretend to not know what they are talking about).
    - cmurf 12y ago
      Or both at the same time. They answer the question they wish you had asked. http://jezebel.com/male-politician-thinks-that-your-vagina-and-stomach-are-1687558660 http://jezebel.com/male-politician-thinks-that-your-vagina-a...
- bsder 12y ago
  Obamacare and shuttle software are completely different political beasts. Obamacare is a standard government contract. The contractor is piling money as fast as possible into today so that when the government pulls the plug tomorrow he's already made a profit. The shuttle software is a safety critical system with a defined budget and schedule. There is no overall incentive to do things immediately and badly as the political fallout for failure will be very severe.
qiqing 12y ago
I had worked on this a few years back, and I quit to start a startup. Sorry. (Not sorry.)
alricb 12y ago
"Lockheed Martin began developing the software for it in 2002" I see your problem right there.
damian2000 12y ago
One of the core components of the NextGen system, ADS-B, is already mandatory for airlines in some countries (such Australia). Its how flightradar24 and other sites manage to get real time info. http://www.flightradar24.com/about http://www.flightradar24.com/about http://en.wikipedia.org/wiki/Automatic_dependent_surveillance_%E2%80%93_broadcast#Implementations_by_country http://en.wikipedia.org/wiki/Automatic_dependent_surveillanc...
peferron 12y ago
Reminds me of 7 years back when I was working at a major power generation company. There was a team working on restarting production of completely obsolete analog control equipment for nuclear reactors. The operator wanted to extend the life of their reactors by a decade or two and it just wasn't worth it for them to have to re-certify a modern system. The existing system had been running for decades without any serious accident, so with stakes that high and red tape that thick, they just went for "if it ain't broke, don't fix it". It was funny in a steampunk / Fallout kind of way.
- bdamm 12y ago
  Seems totally reasonable, provided the system is maintained and operated within design limits.
cmurf 12y ago
"FOR THE FORESEEABLE FUTURE, IF YOU PURCHASE WI-FI IN COACH, YOU'RE PRETTY MUCH BETTER OFF THAN THE PILOT." So why does ATC not having GPS tracking make the pilot worse off? The pilot does have GPS, and even more precision navigation in certain cases like instrument landings. If the FAA and the companies they keep hiring weren't so f'n sclerotic, I'd say they need to look at a totally decentralized system that account for autonomous drones, so all positive control flights (non-VFR) can be integrated into the same system; but obviously starting over yet again is a great way to just burn another $5 billion. No actually burning the cash would at least have residual value as potash.
- jrockway 12y ago
  NextGen gives the pilots (with "ADS-B in") the same view of the airspace that the controllers get. Right now, they only have a limited view, via the TCAS.
  - cmurf 12y ago
    Yes, but WiFi doesn't give me the same view of the airspace the controllers get, or pilots will eventually get. WiFi gets me something like flightware tracking which is a lot slower and more delayed than primary/secondary radar, TCAS or ADS-B. My point is I think the article's author thought they were making a funny. And it wasn't correct or funny.
- Animats 12y ago
  It's unsafe for aviation to be entirely dependent on GPS and self-reporting systems like ADS-B. Planes should still have VORs available, even though they take up a lot of acreage next to airports, the ground should still have primary radar, and aircraft should still have their dumb old AM VHF radios. Large aircraft typically have at least three completely different navigational sources - VOR/DME, GPS, and inertial guidance. Over- reliance on GPS is dangerous.
cbd1984 12y ago
"If it ain't costing me money in a way I can see on a simple spreadsheet, don't fix it." There. That's the mature way of maintaining systems. That's how grown-ups approach the world.
dkarapetyan 12y ago
The problem is that most people confuse bureaucracy with process and structure. Those are not the same thing. It is possible to have process and structure without bureaucracy.
pjtr 12y ago
Did I miss it or do they not mention what tech it is? Are they talking about this? http://en.wikipedia.org/wiki/IBM_9020 http://en.wikipedia.org/wiki/IBM_9020
- wglb 12y ago
  They did not mention it. I do believe that is the technology.
ryannevius 12y ago
As a pilot myself, I wouldn't be so fast to blame the FAA (as much as we pilots don't like them at times). The FAA is using 40 year old tech for the same reason we (especially general aviation pilots) are flying 40+ year old planes: there used to be tons of money pouring into aviation, and now there's not. It's prohibitively expensive to do just about anything in aviation nowadays.
- rwc 12y ago
  But that's in large part because of the FAA. Look no further than the LSA category, which never took off because of the onerous regulatory environment. I think it's much more likely the 40 year old tech is a product of risk-aversion than a lack of money pouring into aviation.
na85 12y ago
There's a lot of negativity and disdain in this thread, but something is being overlooked: You simply can't apply "startup-style" development to something like air traffic control. This isn't your latest javascript app where you can push an update to fix localization bugs after the client reports an issue. This is air traffic control, and thousands of lives are at risk every day. This stuff has to work. It has to be bulletproof. The FAA doesn't refactor existing codebases into Haskell on a whim. A lot of this code and the hardware it runs on has been thoroughly vetted, not to mention battle-tested over decades of use. "Move fast and break stuff" is not an attitude I want FAA-certified gear to be built with.
- jmspring 12y ago
  This makes me wonder where the threads (beyond server scaling) around resiliency on HN are. I'm doing some non-critical embedded programming at the moment (typically do full stack stuff), but embedded and critical systems are much different than the typical audience here....
  - na85 12y ago
    It's easier, flashier, and more immediately gratifying to slap together a new javascript framework. Embedded is hard. Fault-tolerant is hard. Debugging is hard and also very tedious.
    - davidw 12y ago
      It's way easier to build a web company than something that involves difficult programming, may well involve hardware, and probably doesn't have the upside potential that a web thing does. That's why there isn't as much of that kind of thing. That said, I love to read about embedded stuff, because some of those guys are really sharp. I'm also a user and fan of Erlang.
- dkarapetyan 12y ago
  I think the negativity is warranted. There is too much bureaucratic overhead and even it wasn't purely bureaucratic it should not take a decade to design a resilient and fault-tolerant software/hardware system. Margaret Hamilton (http://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%29 http://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%...) might have a few things to say on the matter.
solutionyogi 12y ago
> THE UPGRADED SYSTEM WILL GO ONLINE THIS SPRING, FIVE YEARS LATE AND AT LEAST $500M OVER BUDGET. If they are over budget by half a billion, what was the original budget number? I completely understand that building such system is complicated, not just in terms of technology but in mainly in terms of coordinating all the moving pieces involved. But at no point, I can fathom how you need to spend close to a billion dollars to implement it. As Milton Friedman famously said, you are least careful when you are spending other's money. (i.e. Government spending taxpayer's money)
- panzagl 12y ago
  Twitter's worth $38 billion, so I guess ATC for the US should be worth 40 million billion gajillion dollars. Sounds like an extra $500 million is chump change.
- tezzer 12y ago
  It takes close to half a billion dollars just to test it. Stuff has to work right the first time, and every time after that, for 40+ years. They (and by they I mean we) test the human factors of the displays the controllers work at, the communications links between the ground and the air, the changing of routes for thunderstorms or wind changes, the interoperation of communications and navigation systems built across several decades, the algorithms for deconflicting merging aircraft and each type of aircraft's ability to follow the deconflicting instructions... stuff's complicated, yo.
Osiris 12y ago
The current system is 40 years old, but the new system will be almost 20 years old by the time it's fully deployed (2020).
chopealavu 12y ago
There's a lot that goes into ICAO certification of telecom technologies. You can't just use contention-based protocols, you need some sort of deterministic bound on latency and performance and availability requirements are insane. Different worlds with different requirements.
chupy 12y ago
Every time I see that critical systems like these are running on very old tech I am wondering whether or not the programmers that built these a couple of decades ago are way more capable than the ones from today.

58 points by ghosh 12y ago

SocksCanClose 12y ago

ugh: "THE UPGRADED SYSTEM WILL GO ONLINE THIS SPRING, FIVE YEARS LATE AND AT LEAST $500M OVER BUDGET." #

cmurf 12y ago
Before Lockhead Martin's work trying to modernize, IBM had a billions dollar contract in the 80's. http://sebokwiki.org/wiki/FAA_Advanced_Automation_System_%28AAS%29_Vignette http://sebokwiki.org/wiki/FAA_Advanced_Automation_System_%28... http://articles.latimes.com/1988-08-11/business/fi-288_1_air-traffic-control-equipment http://articles.latimes.com/1988-08-11/business/fi-288_1_air... Basically, none of these huge companies want to do this kind of work unless the pay is obscene. And then once it's obscene, the companies all fight over it and throw each other under the bus. Court battles aren't even the half of it. They hire lobbyists, contribute to campaigns, and every 2-8 years there's some amount of government shuffling. So nothing is actually ever very consistent within the government. New politicians, new bureaucrats, new policies, new ideas (they think anyway), and they basically have been scrapping previously done work and starting over multiple times. And it's been longer than 40 years. Perversely, running on vacuum tubes makes the old system a more security hardened system than what we're likely to end up with. We should hope the NSA has hacked the FAA and ATC as a defensive measure.

technofiend 12y ago

I worked on the shuttle program for a while and was debugging some fortran code that called honeywell assembly as part of a rewrite of the DOLILU program. It calculated stress on the shuttles frame based on winds aloft data collected by balloon on launch day. The code passed in a jump parameter and the assembly jumped back to the appropriate fortran label. Over the years, all the exits except one had been coded out. I asked my boss if we could just make the fortran code jump directly from point A to point B since that's all the assembly did. He answered my 6 months contract would be done before the change made it through committee. The thing about process is people rarely lessen or remove it once it's in place. But it's possible to do a classic waterfall project on time and on budget, look at any bank release of ATM software. Of course losing an orbiter or losing a plane isn't the same class of problem as losing a transaction, but it is possible to do the projects properly. The fact the Obama administration had to bring in outside help to rescue their most recent major rollout (Obamacare insurance info websites) tells you there's no incentive to get it right or on time in these contracts. You just keep working and the bills keep getting paid. These contracts need real oversight and financial penalties for failure to meet deadlines. Assuming of course they also include some iron-clad agreements about requirements not changing.

dasil003 12y ago
I'm not saying your wrong, but shuttle software and even ATMs are different from Obamacare. The former has specific operational requirements that are relatively clear. Health care by contrast is a clusterfuck of legislation piled on top a multitude of tangential legacy systems. I don't doubt that the talent to get it right exists, but it requires more juice than just an engineer saying "this thing won't fly". The problem is that legislation is itself code, but it is written by people who have no nuanced knowledge of the downstream implications, and perhaps more importantly, no incentive to ever refactor.
- dba7dba 12y ago
  Agree. Physics doesn't lie. But Politicians lie (or don't know what they are talking about OR pretend to not know what they are talking about).
  - cmurf 12y ago
    Or both at the same time. They answer the question they wish you had asked. http://jezebel.com/male-politician-thinks-that-your-vagina-and-stomach-are-1687558660 http://jezebel.com/male-politician-thinks-that-your-vagina-a...
bsder 12y ago
Obamacare and shuttle software are completely different political beasts. Obamacare is a standard government contract. The contractor is piling money as fast as possible into today so that when the government pulls the plug tomorrow he's already made a profit. The shuttle software is a safety critical system with a defined budget and schedule. There is no overall incentive to do things immediately and badly as the political fallout for failure will be very severe.

qiqing 12y ago

I had worked on this a few years back, and I quit to start a startup. Sorry. (Not sorry.)

alricb 12y ago

"Lockheed Martin began developing the software for it in 2002" I see your problem right there.

damian2000 12y ago

One of the core components of the NextGen system, ADS-B, is already mandatory for airlines in some countries (such Australia). Its how flightradar24 and other sites manage to get real time info. http://www.flightradar24.com/about http://www.flightradar24.com/about http://en.wikipedia.org/wiki/Automatic_dependent_surveillance_%E2%80%93_broadcast#Implementations_by_country http://en.wikipedia.org/wiki/Automatic_dependent_surveillanc...

peferron 12y ago

Reminds me of 7 years back when I was working at a major power generation company. There was a team working on restarting production of completely obsolete analog control equipment for nuclear reactors. The operator wanted to extend the life of their reactors by a decade or two and it just wasn't worth it for them to have to re-certify a modern system. The existing system had been running for decades without any serious accident, so with stakes that high and red tape that thick, they just went for "if it ain't broke, don't fix it". It was funny in a steampunk / Fallout kind of way.

bdamm 12y ago
Seems totally reasonable, provided the system is maintained and operated within design limits.

cmurf 12y ago

"FOR THE FORESEEABLE FUTURE, IF YOU PURCHASE WI-FI IN COACH, YOU'RE PRETTY MUCH BETTER OFF THAN THE PILOT." So why does ATC not having GPS tracking make the pilot worse off? The pilot does have GPS, and even more precision navigation in certain cases like instrument landings. If the FAA and the companies they keep hiring weren't so f'n sclerotic, I'd say they need to look at a totally decentralized system that account for autonomous drones, so all positive control flights (non-VFR) can be integrated into the same system; but obviously starting over yet again is a great way to just burn another $5 billion. No actually burning the cash would at least have residual value as potash.

jrockway 12y ago
NextGen gives the pilots (with "ADS-B in") the same view of the airspace that the controllers get. Right now, they only have a limited view, via the TCAS.
- cmurf 12y ago
  Yes, but WiFi doesn't give me the same view of the airspace the controllers get, or pilots will eventually get. WiFi gets me something like flightware tracking which is a lot slower and more delayed than primary/secondary radar, TCAS or ADS-B. My point is I think the article's author thought they were making a funny. And it wasn't correct or funny.
Animats 12y ago
It's unsafe for aviation to be entirely dependent on GPS and self-reporting systems like ADS-B. Planes should still have VORs available, even though they take up a lot of acreage next to airports, the ground should still have primary radar, and aircraft should still have their dumb old AM VHF radios. Large aircraft typically have at least three completely different navigational sources - VOR/DME, GPS, and inertial guidance. Over- reliance on GPS is dangerous.

cbd1984 12y ago

"If it ain't costing me money in a way I can see on a simple spreadsheet, don't fix it." There. That's the mature way of maintaining systems. That's how grown-ups approach the world.

dkarapetyan 12y ago

The problem is that most people confuse bureaucracy with process and structure. Those are not the same thing. It is possible to have process and structure without bureaucracy.

pjtr 12y ago

Did I miss it or do they not mention what tech it is? Are they talking about this? http://en.wikipedia.org/wiki/IBM_9020 http://en.wikipedia.org/wiki/IBM_9020

wglb 12y ago
They did not mention it. I do believe that is the technology.

ryannevius 12y ago

As a pilot myself, I wouldn't be so fast to blame the FAA (as much as we pilots don't like them at times). The FAA is using 40 year old tech for the same reason we (especially general aviation pilots) are flying 40+ year old planes: there used to be tons of money pouring into aviation, and now there's not. It's prohibitively expensive to do just about anything in aviation nowadays.

rwc 12y ago
But that's in large part because of the FAA. Look no further than the LSA category, which never took off because of the onerous regulatory environment. I think it's much more likely the 40 year old tech is a product of risk-aversion than a lack of money pouring into aviation.

na85 12y ago

There's a lot of negativity and disdain in this thread, but something is being overlooked: You simply can't apply "startup-style" development to something like air traffic control. This isn't your latest javascript app where you can push an update to fix localization bugs after the client reports an issue. This is air traffic control, and thousands of lives are at risk every day. This stuff has to work. It has to be bulletproof. The FAA doesn't refactor existing codebases into Haskell on a whim. A lot of this code and the hardware it runs on has been thoroughly vetted, not to mention battle-tested over decades of use. "Move fast and break stuff" is not an attitude I want FAA-certified gear to be built with.

jmspring 12y ago
This makes me wonder where the threads (beyond server scaling) around resiliency on HN are. I'm doing some non-critical embedded programming at the moment (typically do full stack stuff), but embedded and critical systems are much different than the typical audience here....
- na85 12y ago
  It's easier, flashier, and more immediately gratifying to slap together a new javascript framework. Embedded is hard. Fault-tolerant is hard. Debugging is hard and also very tedious.
  - davidw 12y ago
    It's way easier to build a web company than something that involves difficult programming, may well involve hardware, and probably doesn't have the upside potential that a web thing does. That's why there isn't as much of that kind of thing. That said, I love to read about embedded stuff, because some of those guys are really sharp. I'm also a user and fan of Erlang.
dkarapetyan 12y ago
I think the negativity is warranted. There is too much bureaucratic overhead and even it wasn't purely bureaucratic it should not take a decade to design a resilient and fault-tolerant software/hardware system. Margaret Hamilton (http://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%29 http://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%...) might have a few things to say on the matter.

solutionyogi 12y ago

> THE UPGRADED SYSTEM WILL GO ONLINE THIS SPRING, FIVE YEARS LATE AND AT LEAST $500M OVER BUDGET. If they are over budget by half a billion, what was the original budget number? I completely understand that building such system is complicated, not just in terms of technology but in mainly in terms of coordinating all the moving pieces involved. But at no point, I can fathom how you need to spend close to a billion dollars to implement it. As Milton Friedman famously said, you are least careful when you are spending other's money. (i.e. Government spending taxpayer's money)

panzagl 12y ago
Twitter's worth $38 billion, so I guess ATC for the US should be worth 40 million billion gajillion dollars. Sounds like an extra $500 million is chump change.
tezzer 12y ago
It takes close to half a billion dollars just to test it. Stuff has to work right the first time, and every time after that, for 40+ years. They (and by they I mean we) test the human factors of the displays the controllers work at, the communications links between the ground and the air, the changing of routes for thunderstorms or wind changes, the interoperation of communications and navigation systems built across several decades, the algorithms for deconflicting merging aircraft and each type of aircraft's ability to follow the deconflicting instructions... stuff's complicated, yo.

Osiris 12y ago

The current system is 40 years old, but the new system will be almost 20 years old by the time it's fully deployed (2020).

chopealavu 12y ago

There's a lot that goes into ICAO certification of telecom technologies. You can't just use contention-based protocols, you need some sort of deterministic bound on latency and performance and availability requirements are insane. Different worlds with different requirements.

chupy 12y ago

Every time I see that critical systems like these are running on very old tech I am wondering whether or not the programmers that built these a couple of decades ago are way more capable than the ones from today.