6 ms·

American Airlines is boarding flights again after nationwide groundstop

60 points by jbegley 2y ago

lxgr 2y ago
A problem with the weight/balance calculation software can indeed be serious, so the ground stop seems warranted. Previously: https://news.ycombinator.com/item?id=34895061 https://news.ycombinator.com/item?id=34895061
- ericcumbee 2y ago
  That seems like a process that could have eaisly been reverted to a paper based worksheet of some type. from my understanding thats how they did it before the software.
  - bpicolo 2y ago
    Doesn't seem easy at all if you've trained people, and they have practiced, on software and not on paper. Mistakes can be fatal.
    - deleted 2y ago
      [deleted]
    - ericcumbee 2y ago
      those are called contingencies....you train people for those.
      - lxgr 2y ago
        Training people for something they basically never have to do, and keeping that training current, is more expensive than you might imagine.
      - SteveNuts 2y ago
        The contingency in this case is just to shut down apparently. They’ve probably figured out that training on this issue is more expensive than shutting down for a few hours. Or maybe their risk management teams have never thought of this happening, who knows.
  - lxgr 2y ago
    Generally? Definitely. On short notice and without pilots having any experience in doing it? That’s another hole in the Swiss cheese. You either do something manually all the time, or you automate it and declare it a critical system. Half measures of automation kill people in Aviation.
  - potato3732842 2y ago
    You can't simply roll back because the old ways required the use of phone lines and fax systems and record types and workflows that literally no longer exist. That said, the critical plane flying and ground handling stuff (like loading cargo for proper balance) is well practiced and understood enough to run fine with ad-hoc paper records albeit it would reduce operational tempo. It's putting the right luggage in the right place, making sure the right plane gets its poo pumped, making sure the cleaning crew knows what to do and where to go, recording maintenance needs and activities on anything that doesn't fly, etc, etc that doesn't have redundancy or good fallback plans or people/managers who understand how their jobs fit into the context well enough to improvise effectively and even if they could the B rate people you had design all your corporate process didn't account for that sort of stuff so people would be risking their jobs if they couldn't do sign offs on the company ipads and whatnot.
  - Etheryte 2y ago
    Weight and balance isn't simply a matter of add up the weight of all cargo on paper. You need to load the plane in a reasonably laid out way and the whole result dictates how you take off. Being wrong or winging it could easily mean a catastrophic failure when trying to take off.
    - potato3732842 2y ago
      Weight and balance calculations are something that anyone in aviation will wind up memorizing early in their career. You don't "need" bespoke software for it. You don't need software at all. It just makes it way easier when you have hundreds of individual items. What's most likely to happen is that they will punch it all into an excel sheet and get their primary objective of weight distribution correct. What will suffer is cargo organization so the crew at the layover is likely to find that they need to unload a bunch of stuff in order to get at some cargo that needs to come off at that stop. A plane "could" fall out of the sky but that would require multiple people not doing their job but there's no point in pearl clutching over that because even if the software works the same people not doing their job can still cause that outcome.
      - Etheryte 2y ago
        This vastly understates the seriousness of problems you can run into. It was just last year that all of Alaska Airlines was grounded [0] because of issues with weight and balance. While those takeoffs resulted in tail scrapes, they could've just as easily lead to running out of runway if the bug had been slightly different [1]. [0] https://www.adn.com/alaska-news/aviation/2023/02/20/after-alaska-airlines-planes-bump-runway-a-scramble-to-pull-the-plug/ https://www.adn.com/alaska-news/aviation/2023/02/20/after-al... [1] https://news.ycombinator.com/item?id=34898361 https://news.ycombinator.com/item?id=34898361
        potato3732842 2y ago
        Comments like yours really drive me up the wall. Nobody doubts that a plane needs to be at least somewhat balanced to fly. It also needs fuel. It needs some semblance of a maintenance schedule. It needs many things. Everyone with a brain knows all this. So then what means beyond in-group signaling is served by hand wringing over any one of the dozens of critical links in the chain? The bug you cite, which you would not likely even know about if not for other commenters, was only caught because people asked for an old school manual calculation. Had the software process been more transparent it perhaps could have been caught earlier. The fact that someone failed to properly implement the calculation in software does not change the efficacy of doing the calculation manually. Furthermore, the fact that there were people capable of eyeballing the accuracy of the software outputs seems to back up my point that there are tons of people capable of doing these calculations kicking around.
  - c22 2y ago
    Around 25 years ago I worked the PoS at a retail establishment. We ran all credit card payments using a machine that would dial-up our payment processor for each transaction. It used the same line as the fax, so if we were receiving or sending a many-page fax this operation would fail. Sometimes it failed for other reasons. As a backup we had the old carbon slips that went into a plastic rig and made an impression of the customer's card. We also had a printed out sales tax table and some more carbon slips we could use to hand-write a receipt. I had to resort to the paper based system on many occasions. A few years back I was in a store shopping when the power went out. The staff didn't know what to do and closed the store early. They had no way to sell me the items in my basket. I asked if they could take an impression of my card and got met with blank stares. They couldn't even take cash because they didn't know how to calculate the total amount, had nothing to write a receipt on, and their cash drawers wouldn't open anyway. Nowadays credit cards don't even come with raised numbers...
    - guerrilla 2y ago
      > Nowadays credit cards don't even come with raised numbers... So that's what the raised numbers were for. I just got my first card without raised numbers last January. It still feels fake and "insecure" to me, even though I know better. Funny how things work.
      - diggan 2y ago
        My latest card doesn't even have the card number, expiry or CVC printed on it at all. And besides that, the CVC code is dynamically generated every time I need it, it doesn't even have a static CVC. Seems relatively new, as some people look at funnily when I hand it over when they need it for "deposits" or whatever.
        lxgr 2y ago
        Apple Card, by any chance? I've always wondered how that works for businesses that still prefer to compare the last four digits of a physical card with those of a booking made online, like some hotels or airlines still do.
- xyst 2y ago
  There should be a business continuity plan in case of “glitches”. But these airlines are cheap and greedy to develop playbooks, train airline staff, and airport personnel on the ground.
  - sokoloff 2y ago
    Much of the reason that commercial aviation is as safe as it is is the prioritization of safety over mission completion in the face of uncertainty. If you have uncertainty in any of the dispatch software or other dispatch items, as a passenger, I'm perfectly happy to wait until they get things sorted out rather than having this hypothetically trained staff manually executing alternative procedures that are unusual to them in order to get me to my destination a couple hours earlier...
  - orev 2y ago
    I guarantee you that they have numerous continuity plans that probably get activated often, and because they have them, you don’t see any issues. It’s only when something happens far outside of those plans that it becomes visible. It’s confirmation bias.
    - bitmasher9 2y ago
      I don’t think this is confirmation bias. Confirmation bias is putting more weight on information that support preexisting beliefs. If a continuity plan is activated and the passengers are unaware then that is a totally different thing. If the passengers are made aware of a continuity plan being implemented then Negativity Bias or the Von Restorff Effect would be more accurate descriptions, in which negative stimuli or unusual stimuli are more frequently recalled.
      - orev 2y ago
        You’re right. I meant survivorship bias, where the survivors are the events that are bad enough for them to be visible to the public. Other events that are addressed by a continuity plan are “killed” and therefore aren’t visible.
- metalman 2y ago
  can you imagine bieng in the room when they saw a problem with the weight/balance software?, planes in the air! realy realy hope this was a false positive or a reporting glitch, rather than ANY flaw in what was shown to pilots
  - M3L0NM4N 2y ago
    W/B is something calculated before takeoff and wouldn't affect planes already in the air or with the W/B already loaded into the FMS.
    - lxgr 2y ago
      But what if incorrect values have been loaded? I think the biggest risks are indeed during takeoff, but I'm not sure if there are no others at all.
      - woleium 2y ago
        i assume it would be within bounds during “normal” atmospheric conditions. In the air with the wrong data and extreme conditions would be where it gets dicey?
      - xvinci 2y ago
        Incorrect w&b can affect a lot of things. From longer ground roll during landing, to wrong brake settings, to worse handling of the aircraft and increased stall speeds. Under normal circumstances it should not be a problem due to huge safety margins, but if other things go wrong it starts adding up. One more hole in the swiss cheese model.
  - stronglikedan 2y ago
    I presume planes are in the air any time a problem is discovered with any system.
Someone1234 2y ago
While far from ideal, a 90-minute delay isn't really comparable to what happened to Southwest[0] or Delta[1], that resulted dozens of cancelled flights and almost a week of disruption. [0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu... [1] https://en.wikipedia.org/wiki/2024_Delta_Air_Lines_disruption https://en.wikipedia.org/wiki/2024_Delta_Air_Lines_disruptio...
- Dalewyn 2y ago
  I happened to be flying right in the midst and worst of Delta's incident, and honestly: Kudos to them for trying to make the best of a shitty situation. I still got to where I wanted to go, and the worst I experienced was my checked bag missed my flight and arrived a day late on the return leg. Looking over the flights that were cancelled while I was at the airport (Hartsfield-Jackson Atlanta International, oddly enough), Delta was clearly prioritizing cancelling flights that had alternatives so as many travellers had the best shot at getting to their destination. I've liked them ever since the Northwest days, and I don't expect that to change any time soon.
  - deleted 2y ago
    [deleted]
  - lolinder 2y ago
    I also like Delta and have forgiven them for that episode, but many of us flying that day didn't make it to our destinations until several days later (in many cases too late for the purpose of the trip). I was flying home from a work conference and was lucky that I woke up early, saw my flight was cancelled within an hour of the incident starting, and immediately booked on another airline. I got home 6 hours later than intended, but at least I got home the same day. The alternatives that Delta booked for me were already going to be the next morning (Saturday), and I followed them after getting home: while the first leg worked fine, the second leg was cancelled late Saturday morning. It's likely that I'd have received yet another rebooking, but if that one also involved two legs it could easily have pushed my return date to Sunday, and by that point in the weekend I doubt I could have found openings on other airlines if I'd changed my mind. I had coworkers who didn't make it back home from that conference until Monday night. So yes, I think they tried hard to avoid inconveniencing people, but if you made it the same day you were lucky. Many didn't.
ilaksh 2y ago
My flight has been delayed only about 20 minutes so far. Edit: now increased to 1 hour delay.
tiffanyh 2y ago
OT: just curious, is AA still run on a mainframe?
- knorker 2y ago
  Don't all airlines, banks, and insurance companies run on mainframes?
  - tiffanyh 2y ago
    I know Sabre, one of the largest GDS (and originally a spinoff of AA), has fully migrated off mainframe. Curious if AA and other airlines have as well.
emanuer 2y ago
As being part of a software development team working on RPS (runway performance) I can tell you, it is stories like this which keep us up at night. The amount of business logic involved is staggering, but miraculously there are experts who truly seem to understand almost all of it. Understanding “almost all of it” is not enough, so there are several different experts who in their entirety hopefully capture 99.99…% But you don’t know what you don’t know… One of the “calming” facts is, that in the tens of thousands of test cases we compared our results to existing solutions, we did discover errors in their calculations. The “calming” aspect is, due to the many safe-guards these errors seemingly never surfaced. Coming from a machine-learning background I quickly understood that there is little appetite for AI solutions, as any optimization which would reduce fuel consumption, is easily offset by all the safety margins. In my mind the goal of the software is to make everything explainable and proof that you truly did think of every eventuality. I am very curious to learn what major f*-up lead to planes being grounded.
cisrockandroll 2y ago
Serious question: Dont software changes with companies dealing with human lives have to go through a rigorous lab testing environment? And then maybe a beta group?
- eddsolves 2y ago
  I’ve been to a talk from the chief test engineer at Saab talking about all the Hardware in the Loop labs they use to test at unit, integration, and full system levels. Then there’s additional testing after that phase… It’s not a CD world!
- finnthehuman 2y ago
  normies on consumer software: they take quality seriously, right? software dev on field they haven't worked in yet: they actually take quality seriously, right?

60 points by jbegley 2y ago

lxgr 2y ago

A problem with the weight/balance calculation software can indeed be serious, so the ground stop seems warranted. Previously: https://news.ycombinator.com/item?id=34895061 https://news.ycombinator.com/item?id=34895061

ericcumbee 2y ago
That seems like a process that could have eaisly been reverted to a paper based worksheet of some type. from my understanding thats how they did it before the software.
- bpicolo 2y ago
  Doesn't seem easy at all if you've trained people, and they have practiced, on software and not on paper. Mistakes can be fatal.
  - deleted 2y ago
    [deleted]
  - ericcumbee 2y ago
    those are called contingencies....you train people for those.
    - lxgr 2y ago
      Training people for something they basically never have to do, and keeping that training current, is more expensive than you might imagine.
    - SteveNuts 2y ago
      The contingency in this case is just to shut down apparently. They’ve probably figured out that training on this issue is more expensive than shutting down for a few hours. Or maybe their risk management teams have never thought of this happening, who knows.
- lxgr 2y ago
  Generally? Definitely. On short notice and without pilots having any experience in doing it? That’s another hole in the Swiss cheese. You either do something manually all the time, or you automate it and declare it a critical system. Half measures of automation kill people in Aviation.
- potato3732842 2y ago
  You can't simply roll back because the old ways required the use of phone lines and fax systems and record types and workflows that literally no longer exist. That said, the critical plane flying and ground handling stuff (like loading cargo for proper balance) is well practiced and understood enough to run fine with ad-hoc paper records albeit it would reduce operational tempo. It's putting the right luggage in the right place, making sure the right plane gets its poo pumped, making sure the cleaning crew knows what to do and where to go, recording maintenance needs and activities on anything that doesn't fly, etc, etc that doesn't have redundancy or good fallback plans or people/managers who understand how their jobs fit into the context well enough to improvise effectively and even if they could the B rate people you had design all your corporate process didn't account for that sort of stuff so people would be risking their jobs if they couldn't do sign offs on the company ipads and whatnot.
- Etheryte 2y ago
  Weight and balance isn't simply a matter of add up the weight of all cargo on paper. You need to load the plane in a reasonably laid out way and the whole result dictates how you take off. Being wrong or winging it could easily mean a catastrophic failure when trying to take off.
  - potato3732842 2y ago
    Weight and balance calculations are something that anyone in aviation will wind up memorizing early in their career. You don't "need" bespoke software for it. You don't need software at all. It just makes it way easier when you have hundreds of individual items. What's most likely to happen is that they will punch it all into an excel sheet and get their primary objective of weight distribution correct. What will suffer is cargo organization so the crew at the layover is likely to find that they need to unload a bunch of stuff in order to get at some cargo that needs to come off at that stop. A plane "could" fall out of the sky but that would require multiple people not doing their job but there's no point in pearl clutching over that because even if the software works the same people not doing their job can still cause that outcome.
    - Etheryte 2y ago
      This vastly understates the seriousness of problems you can run into. It was just last year that all of Alaska Airlines was grounded [0] because of issues with weight and balance. While those takeoffs resulted in tail scrapes, they could've just as easily lead to running out of runway if the bug had been slightly different [1]. [0] https://www.adn.com/alaska-news/aviation/2023/02/20/after-alaska-airlines-planes-bump-runway-a-scramble-to-pull-the-plug/ https://www.adn.com/alaska-news/aviation/2023/02/20/after-al... [1] https://news.ycombinator.com/item?id=34898361 https://news.ycombinator.com/item?id=34898361
      - potato3732842 2y ago
        Comments like yours really drive me up the wall. Nobody doubts that a plane needs to be at least somewhat balanced to fly. It also needs fuel. It needs some semblance of a maintenance schedule. It needs many things. Everyone with a brain knows all this. So then what means beyond in-group signaling is served by hand wringing over any one of the dozens of critical links in the chain? The bug you cite, which you would not likely even know about if not for other commenters, was only caught because people asked for an old school manual calculation. Had the software process been more transparent it perhaps could have been caught earlier. The fact that someone failed to properly implement the calculation in software does not change the efficacy of doing the calculation manually. Furthermore, the fact that there were people capable of eyeballing the accuracy of the software outputs seems to back up my point that there are tons of people capable of doing these calculations kicking around.
- c22 2y ago
  Around 25 years ago I worked the PoS at a retail establishment. We ran all credit card payments using a machine that would dial-up our payment processor for each transaction. It used the same line as the fax, so if we were receiving or sending a many-page fax this operation would fail. Sometimes it failed for other reasons. As a backup we had the old carbon slips that went into a plastic rig and made an impression of the customer's card. We also had a printed out sales tax table and some more carbon slips we could use to hand-write a receipt. I had to resort to the paper based system on many occasions. A few years back I was in a store shopping when the power went out. The staff didn't know what to do and closed the store early. They had no way to sell me the items in my basket. I asked if they could take an impression of my card and got met with blank stares. They couldn't even take cash because they didn't know how to calculate the total amount, had nothing to write a receipt on, and their cash drawers wouldn't open anyway. Nowadays credit cards don't even come with raised numbers...
  - guerrilla 2y ago
    > Nowadays credit cards don't even come with raised numbers... So that's what the raised numbers were for. I just got my first card without raised numbers last January. It still feels fake and "insecure" to me, even though I know better. Funny how things work.
    - diggan 2y ago
      My latest card doesn't even have the card number, expiry or CVC printed on it at all. And besides that, the CVC code is dynamically generated every time I need it, it doesn't even have a static CVC. Seems relatively new, as some people look at funnily when I hand it over when they need it for "deposits" or whatever.
      - lxgr 2y ago
        Apple Card, by any chance? I've always wondered how that works for businesses that still prefer to compare the last four digits of a physical card with those of a booking made online, like some hotels or airlines still do.
xyst 2y ago
There should be a business continuity plan in case of “glitches”. But these airlines are cheap and greedy to develop playbooks, train airline staff, and airport personnel on the ground.
- sokoloff 2y ago
  Much of the reason that commercial aviation is as safe as it is is the prioritization of safety over mission completion in the face of uncertainty. If you have uncertainty in any of the dispatch software or other dispatch items, as a passenger, I'm perfectly happy to wait until they get things sorted out rather than having this hypothetically trained staff manually executing alternative procedures that are unusual to them in order to get me to my destination a couple hours earlier...
- orev 2y ago
  I guarantee you that they have numerous continuity plans that probably get activated often, and because they have them, you don’t see any issues. It’s only when something happens far outside of those plans that it becomes visible. It’s confirmation bias.
  - bitmasher9 2y ago
    I don’t think this is confirmation bias. Confirmation bias is putting more weight on information that support preexisting beliefs. If a continuity plan is activated and the passengers are unaware then that is a totally different thing. If the passengers are made aware of a continuity plan being implemented then Negativity Bias or the Von Restorff Effect would be more accurate descriptions, in which negative stimuli or unusual stimuli are more frequently recalled.
    - orev 2y ago
      You’re right. I meant survivorship bias, where the survivors are the events that are bad enough for them to be visible to the public. Other events that are addressed by a continuity plan are “killed” and therefore aren’t visible.
metalman 2y ago
can you imagine bieng in the room when they saw a problem with the weight/balance software?, planes in the air! realy realy hope this was a false positive or a reporting glitch, rather than ANY flaw in what was shown to pilots
- M3L0NM4N 2y ago
  W/B is something calculated before takeoff and wouldn't affect planes already in the air or with the W/B already loaded into the FMS.
  - lxgr 2y ago
    But what if incorrect values have been loaded? I think the biggest risks are indeed during takeoff, but I'm not sure if there are no others at all.
    - woleium 2y ago
      i assume it would be within bounds during “normal” atmospheric conditions. In the air with the wrong data and extreme conditions would be where it gets dicey?
    - xvinci 2y ago
      Incorrect w&b can affect a lot of things. From longer ground roll during landing, to wrong brake settings, to worse handling of the aircraft and increased stall speeds. Under normal circumstances it should not be a problem due to huge safety margins, but if other things go wrong it starts adding up. One more hole in the swiss cheese model.
- stronglikedan 2y ago
  I presume planes are in the air any time a problem is discovered with any system.

Someone1234 2y ago

While far from ideal, a 90-minute delay isn't really comparable to what happened to Southwest[0] or Delta[1], that resulted dozens of cancelled flights and almost a week of disruption. [0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_scheduling_crisis https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu... [1] https://en.wikipedia.org/wiki/2024_Delta_Air_Lines_disruption https://en.wikipedia.org/wiki/2024_Delta_Air_Lines_disruptio...

Dalewyn 2y ago
I happened to be flying right in the midst and worst of Delta's incident, and honestly: Kudos to them for trying to make the best of a shitty situation. I still got to where I wanted to go, and the worst I experienced was my checked bag missed my flight and arrived a day late on the return leg. Looking over the flights that were cancelled while I was at the airport (Hartsfield-Jackson Atlanta International, oddly enough), Delta was clearly prioritizing cancelling flights that had alternatives so as many travellers had the best shot at getting to their destination. I've liked them ever since the Northwest days, and I don't expect that to change any time soon.
- deleted 2y ago
  [deleted]
- lolinder 2y ago
  I also like Delta and have forgiven them for that episode, but many of us flying that day didn't make it to our destinations until several days later (in many cases too late for the purpose of the trip). I was flying home from a work conference and was lucky that I woke up early, saw my flight was cancelled within an hour of the incident starting, and immediately booked on another airline. I got home 6 hours later than intended, but at least I got home the same day. The alternatives that Delta booked for me were already going to be the next morning (Saturday), and I followed them after getting home: while the first leg worked fine, the second leg was cancelled late Saturday morning. It's likely that I'd have received yet another rebooking, but if that one also involved two legs it could easily have pushed my return date to Sunday, and by that point in the weekend I doubt I could have found openings on other airlines if I'd changed my mind. I had coworkers who didn't make it back home from that conference until Monday night. So yes, I think they tried hard to avoid inconveniencing people, but if you made it the same day you were lucky. Many didn't.

ilaksh 2y ago

My flight has been delayed only about 20 minutes so far. Edit: now increased to 1 hour delay.

tiffanyh 2y ago

OT: just curious, is AA still run on a mainframe?

knorker 2y ago
Don't all airlines, banks, and insurance companies run on mainframes?
- tiffanyh 2y ago
  I know Sabre, one of the largest GDS (and originally a spinoff of AA), has fully migrated off mainframe. Curious if AA and other airlines have as well.

emanuer 2y ago

As being part of a software development team working on RPS (runway performance) I can tell you, it is stories like this which keep us up at night. The amount of business logic involved is staggering, but miraculously there are experts who truly seem to understand almost all of it. Understanding “almost all of it” is not enough, so there are several different experts who in their entirety hopefully capture 99.99…% But you don’t know what you don’t know… One of the “calming” facts is, that in the tens of thousands of test cases we compared our results to existing solutions, we did discover errors in their calculations. The “calming” aspect is, due to the many safe-guards these errors seemingly never surfaced. Coming from a machine-learning background I quickly understood that there is little appetite for AI solutions, as any optimization which would reduce fuel consumption, is easily offset by all the safety margins. In my mind the goal of the software is to make everything explainable and proof that you truly did think of every eventuality. I am very curious to learn what major f*-up lead to planes being grounded.

cisrockandroll 2y ago

Serious question: Dont software changes with companies dealing with human lives have to go through a rigorous lab testing environment? And then maybe a beta group?

eddsolves 2y ago
I’ve been to a talk from the chief test engineer at Saab talking about all the Hardware in the Loop labs they use to test at unit, integration, and full system levels. Then there’s additional testing after that phase… It’s not a CD world!
finnthehuman 2y ago
normies on consumer software: they take quality seriously, right? software dev on field they haven't worked in yet: they actually take quality seriously, right?