4 ms·

I was at a conference in Northern Virginia some months ago and saw a presentation from the folks at Upside, a startup specializing in booking business travel. T

I was at a conference in Northern Virginia some months ago and saw a presentation from the folks at Upside, a startup specializing in booking business travel. They described the legacy system which handles pretty much all booking in the US, a system called SABRE. They described it as an ancient 6-bit computer system in Texas with no modern API. Everything they do tech-wise is a modern wrapper around that system. So I'm not at all surprised by any air travel computer failures if tech like that is central to the system.

dhimes 9y ago
Wow. I remember sabre from genie- General Electric Network Information Exchange. In the 1980s. I’m not sure compuserve was invented yet. My father and I could use “electronic mail” to stay in touch. For you youngsters: I hsd never yet had a “remote control” for a tv yet. Yep. Had to get up to sdjust the volume or change the channel (don’t get me started on adjusting the antenna). Wish I didn’t know this.
- mikeash 9y ago
  And it was already ancient then! Sabre started operating in 1960, and traces its origins to a chance meeting of an IBM salesman and the CEO of American Airlines on a flight in 1953.
  - dhimes 9y ago
    holy crap! I had no idea.
userbinator 9y ago
I've heard of enough misguided "modernisations" (and failures thereof) that I think the "legacy system" was the part that stayed working throughout, and it's the newer stuff added around it that failed. The old stuff may be old but there's a reason it's old... it's outlasted any attempts at replacing it.
- tyingq 9y ago
  I can confirm this. It's usually the Java/Tomcat boxes that front the TPF box that cause this type of huge meltdown. It's almost never the TPF core. It happens, of course, but they've offloaded most of the functionality to more modern technology. So there isn't much code change in the TPF core. And code change is usually what drives outages. Update: Seems this incident wasn't the reservation system at all. It was ETFMS. Government run system. http://www.eurocontrol.int/articles/enhanced-tactical-flow-management-system-etfms http://www.eurocontrol.int/articles/enhanced-tactical-flow-m...
- AceyMan 9y ago
  >it's outlasted any attempt to replace it See, https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect (my fave citation from everyone's fave source for non-academic citations)
  - mseebach 9y ago
    The counteracting force to that will be that they complexity of the surrounding environment increases and becomes more brittle, or is just simply in the way, as constantly increasing demands for new features drifts further from the capabilities of the old system in the middle. A few years ago, SAS introduced a new status tier. It took 18 months to introduce into the system (Amadeus). The system may be stable, but those kinds of turn around times for a minor customer service change simply isn't feasible. I don't have numbers, but I wouldn't be surprised if one of the reasons upstart airlines such as EasyJet are competitive is that their IT is comparatively modern and can actively support the organisation while IT is more of a millstone around the legacy airlines' necks.
tyingq 9y ago
You're talking about TPF[1]. Many smart people and organizations have tried and failed to build something that could match it, including a company Google paid $700 million to buy. I personally know of at least 10 failed attempts :) Not sure where "6 bit" is coming from though, and you can use gcc/c++ now, not just assembler[2]. And it's in Tulsa, Oklahoma, not Texas. Sabre's HDQ is in Texas, the data center is not. The hardware is very modern and new Z series mainframes in big loosely coupled clusters. Amadeus, Sabre's main competition, also still has TPF at the core. There is one notable non TPF reservation system. http://www.navitaire.com/ http://www.navitaire.com/ Last I checked, it couldn't scale well enough to handle a large airline. Both Sabre and Amadeus are replacing TPF, but one function at a time (shopping, fare engine, booking, check in, etc). And very slowly. [1]https://en.m.wikipedia.org/wiki/Transaction_Processing_Facility https://en.m.wikipedia.org/wiki/Transaction_Processing_Facil... [2]https://www.ibm.com/support/knowledgecenter/en/SSB23S_1.1.0.14/gtpm1/m1chclg.html https://www.ibm.com/support/knowledgecenter/en/SSB23S_1.1.0.... Fwiw, TPF is basically a huge, distributed, and transactionally consistent nosql database. Most orgs still using it have extracted most of the business logic that was in it, out to Linux boxes that front it. Not for stability reasons, but faster time to market with new features. To date, attempts to replace the high contention and high transaction rate nosql type traffic haven't scaled well enough. Just in the US, 2.5 million people fly each day. And the processes to sell, board, etc each passenger are lots of transactions each. It's a pretty big scale. I think it's fairly close to Amazon sales per day, but with more contention and sub transactions. All Visa credit card transactions are also still on TPF.
ubernostrum 9y ago
In the air-travel world, a lot of seemingly-random limitations on systems that interface with reservations comes from the fact that they were originally built with telephone interfaces intended for use only by trained staff. Every few years, for example, someone digs up and reposts one of the articles explaining why some airlines didn't allow 'Q' and 'Z' in account passwords (they were passing things directly through to SABRE on the backend, and so only allowed letters that could be "dialed" on the 1960s rotary phones SABRE was designed to interface with).

dhimes 9y ago

Wow. I remember sabre from genie- General Electric Network Information Exchange. In the 1980s. I’m not sure compuserve was invented yet. My father and I could use “electronic mail” to stay in touch. For you youngsters: I hsd never yet had a “remote control” for a tv yet. Yep. Had to get up to sdjust the volume or change the channel (don’t get me started on adjusting the antenna). Wish I didn’t know this.

mikeash 9y ago
And it was already ancient then! Sabre started operating in 1960, and traces its origins to a chance meeting of an IBM salesman and the CEO of American Airlines on a flight in 1953.
- dhimes 9y ago
  holy crap! I had no idea.

userbinator 9y ago

I've heard of enough misguided "modernisations" (and failures thereof) that I think the "legacy system" was the part that stayed working throughout, and it's the newer stuff added around it that failed. The old stuff may be old but there's a reason it's old... it's outlasted any attempts at replacing it.

tyingq 9y ago
I can confirm this. It's usually the Java/Tomcat boxes that front the TPF box that cause this type of huge meltdown. It's almost never the TPF core. It happens, of course, but they've offloaded most of the functionality to more modern technology. So there isn't much code change in the TPF core. And code change is usually what drives outages. Update: Seems this incident wasn't the reservation system at all. It was ETFMS. Government run system. http://www.eurocontrol.int/articles/enhanced-tactical-flow-management-system-etfms http://www.eurocontrol.int/articles/enhanced-tactical-flow-m...
AceyMan 9y ago
>it's outlasted any attempt to replace it See, https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect (my fave citation from everyone's fave source for non-academic citations)
- mseebach 9y ago
  The counteracting force to that will be that they complexity of the surrounding environment increases and becomes more brittle, or is just simply in the way, as constantly increasing demands for new features drifts further from the capabilities of the old system in the middle. A few years ago, SAS introduced a new status tier. It took 18 months to introduce into the system (Amadeus). The system may be stable, but those kinds of turn around times for a minor customer service change simply isn't feasible. I don't have numbers, but I wouldn't be surprised if one of the reasons upstart airlines such as EasyJet are competitive is that their IT is comparatively modern and can actively support the organisation while IT is more of a millstone around the legacy airlines' necks.

tyingq 9y ago

You're talking about TPF[1]. Many smart people and organizations have tried and failed to build something that could match it, including a company Google paid $700 million to buy. I personally know of at least 10 failed attempts :) Not sure where "6 bit" is coming from though, and you can use gcc/c++ now, not just assembler[2]. And it's in Tulsa, Oklahoma, not Texas. Sabre's HDQ is in Texas, the data center is not. The hardware is very modern and new Z series mainframes in big loosely coupled clusters. Amadeus, Sabre's main competition, also still has TPF at the core. There is one notable non TPF reservation system. http://www.navitaire.com/ http://www.navitaire.com/ Last I checked, it couldn't scale well enough to handle a large airline. Both Sabre and Amadeus are replacing TPF, but one function at a time (shopping, fare engine, booking, check in, etc). And very slowly. [1]https://en.m.wikipedia.org/wiki/Transaction_Processing_Facility https://en.m.wikipedia.org/wiki/Transaction_Processing_Facil... [2]https://www.ibm.com/support/knowledgecenter/en/SSB23S_1.1.0.14/gtpm1/m1chclg.html https://www.ibm.com/support/knowledgecenter/en/SSB23S_1.1.0.... Fwiw, TPF is basically a huge, distributed, and transactionally consistent nosql database. Most orgs still using it have extracted most of the business logic that was in it, out to Linux boxes that front it. Not for stability reasons, but faster time to market with new features. To date, attempts to replace the high contention and high transaction rate nosql type traffic haven't scaled well enough. Just in the US, 2.5 million people fly each day. And the processes to sell, board, etc each passenger are lots of transactions each. It's a pretty big scale. I think it's fairly close to Amazon sales per day, but with more contention and sub transactions. All Visa credit card transactions are also still on TPF.

ubernostrum 9y ago

In the air-travel world, a lot of seemingly-random limitations on systems that interface with reservations comes from the fact that they were originally built with telephone interfaces intended for use only by trained staff. Every few years, for example, someone digs up and reposts one of the articles explaining why some airlines didn't allow 'Q' and 'Z' in account passwords (they were passing things directly through to SABRE on the backend, and so only allowed letters that could be "dialed" on the 1960s rotary phones SABRE was designed to interface with).