5 ms·

Incredible to see that 14 months ago there was an almost duplicate outage. Clearly the operations teams are running too lean and don't have enough slack to hand

Incredible to see that 14 months ago there was an almost duplicate outage. Clearly the operations teams are running too lean and don't have enough slack to handle predictable weather/sickness events. Seems like something you'd plan for. Even at the 98%+ uptime range https://news.ycombinator.com/item?id=28823774 https://news.ycombinator.com/item?id=28823774

chasd00 4y ago
There's no slack at an airline, it's basically a real time system. Imagine a bottling plant, if one part of the line malfunctions the rest of the line just keeps throwing bottles at the malfunctioning unit until the stop button is hit. There's no way to build in artificial delay to those kinds of systems. Even if you could do that at an airline the margins are so razor thin. I bet there's a dollar value assigned to every second a plane is not in the air.
- kibwen 4y ago
  > Even if you could do that at an airline the margins are so razor thin. I bet there's a dollar value assigned to every second a plane is not in the air. You either pay the predictable, ongoing cost or you pay the massive, unpredictable cost like they are currently. There's no free lunch. That is, unless you're an executive looking to boost short-term profits by eliminating "redundancy" and then peacing out before the whole thing collapses from your short-sighted, greedy ineptitude.
  - dehrmann 4y ago
    People don't like to hear this now, but airlines chose the right thing to optimize for. Air travel is rarely that important to have these redundancies, and they vote with their wallets every time they buy economy tickets. Given that safe is a baseline, most people value price over reliability.
    - ThunderSizzle 4y ago
      In my experience, every airline will screw you if your not their frequent flyer. Fly once a year? No one is going to waive a fee or refund you anything. Might as well go for cheapest price. My wife's experience shows SWA as an exception in that.
    - Uvix 4y ago
      Whereas I'd rather have reliability, which is why I'll take a 10-12 hour drive over a 2 hour flight any day.
  - pkulak 4y ago
    Is this really a massive cost, though? What does it cost to release some statements to the media and maybe reimburse some hotel receipts in 6 months? Especially balanced against the fuel savings of grounding half your fleet for 6 days. It's absolutely cheaper to run without enough staff and then just deal with the fallout every couple years. No one will remember this. Even the people effected will pick the cheapest flight next time, not matter the airline.
    - noirbot 4y ago
      That does bring up an interesting question - are they actually saving on fuel? I would have assumed, given the relatively planned-out nature of most airline operations, that fuel would be something contracted out in advance, and that they may have already bought the fuel whether they use it or not.
      - pc86 4y ago
        Yes and no, they certainly buy fuel in bulk and several weeks/months out, but they're still not flying for a week, so their yearly fuel cost is going to be less than projected.
  - HDThoreaun 4y ago
    Airlines are too competing to have extra labor for situations like this. Everyone will book on the other airline that works 99% of the time and you’ll be out of business before a catastrophe hits. “Just add more slack” is not an option that also allows these businesses to be profitable.
- bronson 4y ago
  This is not entirely true. Lots of airlines have had events where they need to ground all takeoffs for a couple hours while they put their systems back together. People don't die when airline operations go down so artificial delays are used occasionally. Lots of air travelers have experienced them. Yes, there's a dollar value attached to everything and everything is being optimized. Southwest was among the pioneers. See the story of the ten minute turn.
- howinteresting 4y ago
  I think the way to do it is how the EU does it, where slack is built in by law (large amounts of compensation for delayed flights). You pay more for each flight in return for predictability, and the race to the bottom is prevented. Though I'm not sure it would have helped in this case.
  - ghaff 4y ago
    And Aer Lingus had a meltdown just last September because of network issues. https://simpleflying.com/aer-lingus-data-center-meltdown-compensation/ https://simpleflying.com/aer-lingus-data-center-meltdown-com...
    - howinteresting 4y ago
      While that is definitely unfortunate, it looks like the system is working correctly. Aer Lingus is on the hook for millions of euros thanks to EU law, and then it goes after the IT provider to be made whole.
      - noirbot 4y ago
        I guess let's wait to see how it works out in the US? The US government seems to be saying they're going to try to do something about this, so it doesn't seem like SWA is going to pay nothing here. Actually, it appears that Southwest is already saying they'll reimburse tickets for canceled flights and potentially pay out for necessary hotels and alternate arrangements: https://www.southwest.com/html/air/travel-disruption https://www.southwest.com/html/air/travel-disruption
        howinteresting 4y ago
        EU compensation for cancelled flights is all of that plus a monetary sum: https://europa.eu/youreurope/citizens/travel/passenger-rights/air/index_en.htm#compensation-cancellation-1 https://europa.eu/youreurope/citizens/travel/passenger-right... edit: though there is an extraordinary circumstances exception below. I'm not sure if this would qualify as that given that it's a combination of an adverse weather event and a technical system collapse.
- dendrite9 4y ago
  There are things called accumulation systems for lines that serve as a buffer to make the line more resilient and able to function if there is a malfunction somewhere in the line. It took me a bit of time to think of the name but I have seen them on large production lines down to small brewery bottling/canning systems. I had a partner who worked on airline optimization several years ago, as I recall there were standby aircraft in some places that could be deployed to fix problems like a plane needing unexpected maintenance. Or even needing a single label required to be flightworthy. That Southwest doesn't use hubs likely makes it more difficult to recover from this kind of disruption before breaking. https://www.kinexcappers.com/accumulation-table/ https://www.kinexcappers.com/accumulation-table/ https://www.nerconconveyors.com/Nercon/Documents/White-Papers/Accumulating-Equipment-White-Paper https://www.nerconconveyors.com/Nercon/Documents/White-Paper...
hinkley 4y ago
Could be lean, could be a bus number problem. They flock together but are separate issues.

chasd00 4y ago

There's no slack at an airline, it's basically a real time system. Imagine a bottling plant, if one part of the line malfunctions the rest of the line just keeps throwing bottles at the malfunctioning unit until the stop button is hit. There's no way to build in artificial delay to those kinds of systems. Even if you could do that at an airline the margins are so razor thin. I bet there's a dollar value assigned to every second a plane is not in the air.

kibwen 4y ago
> Even if you could do that at an airline the margins are so razor thin. I bet there's a dollar value assigned to every second a plane is not in the air. You either pay the predictable, ongoing cost or you pay the massive, unpredictable cost like they are currently. There's no free lunch. That is, unless you're an executive looking to boost short-term profits by eliminating "redundancy" and then peacing out before the whole thing collapses from your short-sighted, greedy ineptitude.
- dehrmann 4y ago
  People don't like to hear this now, but airlines chose the right thing to optimize for. Air travel is rarely that important to have these redundancies, and they vote with their wallets every time they buy economy tickets. Given that safe is a baseline, most people value price over reliability.
  - ThunderSizzle 4y ago
    In my experience, every airline will screw you if your not their frequent flyer. Fly once a year? No one is going to waive a fee or refund you anything. Might as well go for cheapest price. My wife's experience shows SWA as an exception in that.
  - Uvix 4y ago
    Whereas I'd rather have reliability, which is why I'll take a 10-12 hour drive over a 2 hour flight any day.
- pkulak 4y ago
  Is this really a massive cost, though? What does it cost to release some statements to the media and maybe reimburse some hotel receipts in 6 months? Especially balanced against the fuel savings of grounding half your fleet for 6 days. It's absolutely cheaper to run without enough staff and then just deal with the fallout every couple years. No one will remember this. Even the people effected will pick the cheapest flight next time, not matter the airline.
  - noirbot 4y ago
    That does bring up an interesting question - are they actually saving on fuel? I would have assumed, given the relatively planned-out nature of most airline operations, that fuel would be something contracted out in advance, and that they may have already bought the fuel whether they use it or not.
    - pc86 4y ago
      Yes and no, they certainly buy fuel in bulk and several weeks/months out, but they're still not flying for a week, so their yearly fuel cost is going to be less than projected.
- HDThoreaun 4y ago
  Airlines are too competing to have extra labor for situations like this. Everyone will book on the other airline that works 99% of the time and you’ll be out of business before a catastrophe hits. “Just add more slack” is not an option that also allows these businesses to be profitable.
bronson 4y ago
This is not entirely true. Lots of airlines have had events where they need to ground all takeoffs for a couple hours while they put their systems back together. People don't die when airline operations go down so artificial delays are used occasionally. Lots of air travelers have experienced them. Yes, there's a dollar value attached to everything and everything is being optimized. Southwest was among the pioneers. See the story of the ten minute turn.
howinteresting 4y ago
I think the way to do it is how the EU does it, where slack is built in by law (large amounts of compensation for delayed flights). You pay more for each flight in return for predictability, and the race to the bottom is prevented. Though I'm not sure it would have helped in this case.
- ghaff 4y ago
  And Aer Lingus had a meltdown just last September because of network issues. https://simpleflying.com/aer-lingus-data-center-meltdown-compensation/ https://simpleflying.com/aer-lingus-data-center-meltdown-com...
  - howinteresting 4y ago
    While that is definitely unfortunate, it looks like the system is working correctly. Aer Lingus is on the hook for millions of euros thanks to EU law, and then it goes after the IT provider to be made whole.
    - noirbot 4y ago
      I guess let's wait to see how it works out in the US? The US government seems to be saying they're going to try to do something about this, so it doesn't seem like SWA is going to pay nothing here. Actually, it appears that Southwest is already saying they'll reimburse tickets for canceled flights and potentially pay out for necessary hotels and alternate arrangements: https://www.southwest.com/html/air/travel-disruption https://www.southwest.com/html/air/travel-disruption
      - howinteresting 4y ago
        EU compensation for cancelled flights is all of that plus a monetary sum: https://europa.eu/youreurope/citizens/travel/passenger-rights/air/index_en.htm#compensation-cancellation-1 https://europa.eu/youreurope/citizens/travel/passenger-right... edit: though there is an extraordinary circumstances exception below. I'm not sure if this would qualify as that given that it's a combination of an adverse weather event and a technical system collapse.
dendrite9 4y ago
There are things called accumulation systems for lines that serve as a buffer to make the line more resilient and able to function if there is a malfunction somewhere in the line. It took me a bit of time to think of the name but I have seen them on large production lines down to small brewery bottling/canning systems. I had a partner who worked on airline optimization several years ago, as I recall there were standby aircraft in some places that could be deployed to fix problems like a plane needing unexpected maintenance. Or even needing a single label required to be flightworthy. That Southwest doesn't use hubs likely makes it more difficult to recover from this kind of disruption before breaking. https://www.kinexcappers.com/accumulation-table/ https://www.kinexcappers.com/accumulation-table/ https://www.nerconconveyors.com/Nercon/Documents/White-Papers/Accumulating-Equipment-White-Paper https://www.nerconconveyors.com/Nercon/Documents/White-Paper...

hinkley 4y ago

Could be lean, could be a bus number problem. They flock together but are separate issues.