6 ms·

One customer was told that the root cause was a "lightning strike on a datacentre", although it sounds pretty unlikely (there are storms today in the UK but sur

One customer was told that the root cause was a "lightning strike on a datacentre", although it sounds pretty unlikely (there are storms today in the UK but surely there'd be a DR plan?)

tyingq 9y ago
They do have DR plans, but the reality is that whole thing is a huge distributed system with parts from different vendors, in different locations, etc, with decades of legacy. A power outage the reboots a whole datacenter would screw any major airline for at least half a day. Thus far, nobody has found the economics of a modern, truly HA setup worth the cost. Outsiders greatly underestimate the complexity too. Think something like 40 disparate applications from different vendors, or some homegrown systems, in different geographical locations. Then, all the client applications are in buildings you don't own (airports) where you aren't allowed to control the infrastructure. If it were a high margin business, perhaps things would be different. It's not.
- jacques_chester 9y ago
  > If it were a high margin business, perhaps things would be different. It's not. Most of the loss here is not from lost bookings; a lot of people who prefer BA will probably just wait and book once the systems are restored. It's going to be from secondary losses -- lawsuits, ding to the reputation and so on. These numbers are estimatable, even for low-margin businesses.
  - tyingq 9y ago
    I agree, but I've been around this sort of thing and seen the post incident analysis, been in the discussions, etc. Despite the huge costs of these outages, they pale in comparison to the costs of a real HA solution for all "needed to fly" applications. To give you some idea, ITA Software was bought by Google for $700 million. They were some of the best and brightest minds in this space, on par with any Silicon Valley darling. They successfully wrote a modern replacement for one popular airline function...shopping. They failed, however, at delivering a modern reservation system, despite tons of money and talent invested.
    - jacques_chester 9y ago
      > Despite the huge costs of these outages, they pale in comparison to the costs of a real HA solution for all "needed to fly" applications. Then, quite honestly, it's the rational decision for airlines to take. Even Google accepts that perfect reliability is impossible, and they're sailing in a pillow-strewn gold-plated yacht down a Mississippi of money.
    - mnutt 9y ago
      I thought that they successfully delivered a passenger scheduling system, but it failed to gain market traction. I don't know the details, but I'd imagine it would be a huge migration cost even if the new system was much better.
deleted 9y ago
[deleted]
mtkd 9y ago
that would make a lot of sense - there was a heavy storm that moved north from south coast right at that time you can have all the DR in the world but if you never fully test it - you never know if it's going to be 100% on the day - the only way to fully test is to do a full failover from the primary and see what happens - but it could be a real career low if you do that on something so commercially sensitive and it all goes very wrong interestingly it's a bank holiday weekend - so every possibility some people who knew some of the special incantations to get things back up on days like this are out camping somewhere remote - and will have 500+ messages when they get back in to signal tomorrow and wonder why they ever try to go away - and if they ever do go away again - wonder how they will ever be able to relax
rorykoehler 9y ago
Surely they have multiple datacentres as redundancy.
- robjan 9y ago
  The two data centres are geographically very close to each other, straddling the runway.
  - tyingq 9y ago
    It's also not generally that easy. There's more than one system required to operate flights, usually more than 10. Some would be in BA data centers (UK), some in Amadeus/Altea data centers(DE), etc. Some applications are vendor provided, some in-house. So, you can have two of everything. But when you have something big, like a power outage, getting everything talking to the "surviving/dr" instances of all the different sytems isn't easy. And, the outage itself causes a huge spike in client driven transactions. Every passenger and employee is hitting whatever the equivalent of a reload button is. The fix is more expensive than any major airline has been willing to spend thus far.

One customer was told that the root cause was a "lightning strike on a datacentre", although it sounds pretty unlikely (there are storms today in the UK but surely there'd be a DR plan?)

tyingq 9y ago

They do have DR plans, but the reality is that whole thing is a huge distributed system with parts from different vendors, in different locations, etc, with decades of legacy. A power outage the reboots a whole datacenter would screw any major airline for at least half a day. Thus far, nobody has found the economics of a modern, truly HA setup worth the cost. Outsiders greatly underestimate the complexity too. Think something like 40 disparate applications from different vendors, or some homegrown systems, in different geographical locations. Then, all the client applications are in buildings you don't own (airports) where you aren't allowed to control the infrastructure. If it were a high margin business, perhaps things would be different. It's not.

jacques_chester 9y ago
> If it were a high margin business, perhaps things would be different. It's not. Most of the loss here is not from lost bookings; a lot of people who prefer BA will probably just wait and book once the systems are restored. It's going to be from secondary losses -- lawsuits, ding to the reputation and so on. These numbers are estimatable, even for low-margin businesses.
- tyingq 9y ago
  I agree, but I've been around this sort of thing and seen the post incident analysis, been in the discussions, etc. Despite the huge costs of these outages, they pale in comparison to the costs of a real HA solution for all "needed to fly" applications. To give you some idea, ITA Software was bought by Google for $700 million. They were some of the best and brightest minds in this space, on par with any Silicon Valley darling. They successfully wrote a modern replacement for one popular airline function...shopping. They failed, however, at delivering a modern reservation system, despite tons of money and talent invested.
  - jacques_chester 9y ago
    > Despite the huge costs of these outages, they pale in comparison to the costs of a real HA solution for all "needed to fly" applications. Then, quite honestly, it's the rational decision for airlines to take. Even Google accepts that perfect reliability is impossible, and they're sailing in a pillow-strewn gold-plated yacht down a Mississippi of money.
  - mnutt 9y ago
    I thought that they successfully delivered a passenger scheduling system, but it failed to gain market traction. I don't know the details, but I'd imagine it would be a huge migration cost even if the new system was much better.

deleted 9y ago

[deleted]

mtkd 9y ago

that would make a lot of sense - there was a heavy storm that moved north from south coast right at that time you can have all the DR in the world but if you never fully test it - you never know if it's going to be 100% on the day - the only way to fully test is to do a full failover from the primary and see what happens - but it could be a real career low if you do that on something so commercially sensitive and it all goes very wrong interestingly it's a bank holiday weekend - so every possibility some people who knew some of the special incantations to get things back up on days like this are out camping somewhere remote - and will have 500+ messages when they get back in to signal tomorrow and wonder why they ever try to go away - and if they ever do go away again - wonder how they will ever be able to relax

rorykoehler 9y ago

Surely they have multiple datacentres as redundancy.

robjan 9y ago
The two data centres are geographically very close to each other, straddling the runway.
- tyingq 9y ago
  It's also not generally that easy. There's more than one system required to operate flights, usually more than 10. Some would be in BA data centers (UK), some in Amadeus/Altea data centers(DE), etc. Some applications are vendor provided, some in-house. So, you can have two of everything. But when you have something big, like a power outage, getting everything talking to the "surviving/dr" instances of all the different sytems isn't easy. And, the outage itself causes a huge spike in client driven transactions. Every passenger and employee is hitting whatever the equivalent of a reload button is. The fix is more expensive than any major airline has been willing to spend thus far.