15 ms·

After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

242 points by dlgeek 4y ago

ww520 4y ago
If weight calculation is so crucial for take off, it seems like redundancy of the weight calculation is needed - a different weight calculation from a different company using a different method to verify against the main one.
- tzs 4y ago
  Would it be possible to actually weigh the loaded plane to check the calculations, say with something like a bigger version of truck scales that the plane could taxi over?
  - pqdbr 4y ago
    Why doesn't the plane weight itself? Couldn't there be sensors in the landing gear or something?
Aloha 4y ago
This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground stop * The problem was figured out quickly and a work-around was developed * Flights Resumed after successfully deploying the work around to the 'production process' * A patch was quickly developed and deployed once the underlying bug was uncovered. I think everyone (but perhaps the developers) comes out looking like a champ. Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable - particularly if I was held to account for undetected failures. The other part that I'd note here, is process is just as important as software could process alone have caught this failure without the tail strikes? possibly, and its something worth looking into further - but only if it doesn't add an unacceptable workload burden to pilots workload that could otherwise compromise safety further.
- basch 4y ago
  The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes. >Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. > Both planes headed down the runway with less power and at lower speed than they should have. And with the jets judged lighter than they actually were, the pilots rotated too early. >The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins. Both aircraft got airborne well within safety limits despite the lower thrust. It’s also probably lucky that two Hawai’i bound flights both didn’t notice the discrepancy, or two bumps wouldn’t have bee as suspicious. Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human. Like all complex failures at least 5 things had to happen. Software update that morning, some kind of hard to simulate resource issue in processing, overweight flights, pilot not pricing plane data appeared light, pilot turning early despite low speed, two in a row.
  - pxmpxm 4y ago
    >Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human. Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.
    - hooverd 4y ago
      I thought a lot of safety thinking was built around acknowledging that humans don't do everything perfectly all the time.
fortran77 4y ago
[flagged]
- JamesonNetworks 4y ago
  The article says the reason the weights were wrong was due to high load on the DynamicSource software. Seems more like a concurrency issue
- deleted 4y ago
  [deleted]
- loeg 4y ago
  Both aircraft seat about 180 pax. To be 20,000 lbs off in your hypothesis, each passenger would have to be 110 lbs heavier than modeled, on average. For 30,000 lbs, 165 lbs.
  - fortran77 4y ago
    See: https://airinsight.com/the-pending-new-faa-weight-balance-rules/ https://airinsight.com/the-pending-new-faa-weight-balance-ru... There were recent changes to the weight and balance rules because of increased weight. It's certainly possible this was a factor.
    - loeg 4y ago
      > The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively. This is a lot smaller in magnitude than +110-165 lb.
hodgesrm 4y ago
That was an impressively quick call based on ability to reason about unexpected behavior of the system. It should give pause to anyone who wants to take the human out of the loop in safety critical software.
- guitarbill 4y ago
  Agreed, although it takes an individual with a huge amount of integrity or a culture that values and promotes integrity to make a call like this: > “At that point, two in a row like that, that’s when I said, ‘No, we’re done,'” said Peyton. “That’s when I stopped things.” Hats off to Peyton, and possibly Alaska. I would also love to hear about times when something similar happened, and it was discovered that nothing was wrong.
  - CPLX 4y ago
    Amazingly, this really is the approach that has characterized all aspects of aviation for decades. It’s incredibly admirable and worth emulating in many other situations. It’s also fragile, and the fact that it strikes many as an uncommon level of integrity for a business decision is precisely why aviation needs to be shielded from having the prevailing business culture sneak in, like we have seen happen at Boeing in a few highly notable instances.
    - hodgesrm 4y ago
      This type of judgment call by a single person has also helped avoid nuclear war on at least two documented occasions. [0,1] Again, food for thought. [0] https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alarm_incident https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar... [1] https://en.wikipedia.org/wiki/Able_Archer_83 https://en.wikipedia.org/wiki/Able_Archer_83
neonate 4y ago
https://archive.md/XOgpi https://archive.md/XOgpi
sudhirj 4y ago
> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.
- cratermoon 4y ago
  Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs
  - djanogo 4y ago
    This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them. "the bug was missed because it only presented when many aircraft at the same time were using the system"
    - hn_throwaway_99 4y ago
      There is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/ https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare.... Obviously sessions should be independent and not sharing data, but that's why it was a bug.
    - usea 4y ago
      I believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.
  - howinteresting 4y ago
Tempest1981 4y ago
Curious about the actual bug: > the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system > the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings. Multithreading/contention issue? But how would that alter the weights? > The software code was permanently repaired about five hours later That's surprisingly fast, isn't it?
- sidlls 4y ago
  Multithreading/contention issue? But how would that alter the weights? Off the top of my head: failure to sum values correctly, misreporting the weight to the target plane (reading values from another plane’s weight data).
- hansvm 4y ago
  Could be something as simple as the first concurrency 101 example: atomic addition. The += operator behaves poorly on concurrent access.
  - Jtsummers 4y ago
    We had that error in an application at work. Essentially data was coming in on multiple threads, but there was a shared value between them that was being incremented and decremented. Without a lock and without using atomics, the tally could be incorrect. But under low load this didn't appear because the odds of interrupting +=/-=/++/-- was low. Under high load, however, the odds of a thread being interrupted mid-operation increased and so it appeared in production. At least in our case it was just an annoyance (no critical decisions were based on the value, it was merely informative). If there are multiple data streams feeding in weight data per aircraft, under a low aircraft load (the scenarios they apparently tested) the odds of interruption during an operation like += can be low enough to not see the issue. Under high load, though, the odds of interruption and incorrect tallying increases substantially.
deleted 4y ago
[deleted]
justinclift 4y ago
With Boeing's recent 737 family history (MAX fiasco, etc), it's very concerning to see "software errors" still happening in operation. :(
- ceejayoz 4y ago
  How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to account for the root cause.
  - sopooneo 4y ago
    I'm with you that a couple layers of the "safety in depth" worked impressively well here. But a bug of this types points to egregiously bad process and standards with the software vendor. And the airlines or FAA should have audits to catch that well upstream of this incident.
  - abudabi123 4y ago
    The SpaceX rockets launch routinely with automation at takeoff and landing. There is no need for human heroic drama in the loop when the software is AAA grade.
    - ceejayoz 4y ago
      > The SpaceX rockets launch routinely with automation at takeoff and landing. Sure. And there are bugs in their code, too. https://www.theverge.com/2019/9/3/20847243/spacex-starlink-satellite-european-space-agency-aeolus-conjunction-space-debris https://www.theverge.com/2019/9/3/20847243/spacex-starlink-s... "SpaceX acknowledges that it failed to communicate due to the bug and missed the emails about a higher probability of collision. Finally, on Monday morning in Europe, ESA made the call and used the thrusters on Aeolus to raise the satellite’s orbit by about 984 feet (300 meters) without waiting for SpaceX to take corrective action."
fwlr 4y ago
“the bug was missed because it only presented when many aircraft at the same time were using the system.” The system reports data on number of passengers, weight of cargo, plane balance, etc., to the pilots. The calculation is done by the plane’s flight computer. How can it be off by 20,000 pounds, but only under heavy server load? The explanation that comes to mind is that DynamicSource has a subservice for each source of weight and one of those subservices crashed under heavy usage. So the top-level aggregate-and-report service got an error from one subservice and said “well, guess it’s zero, lol”?
- jcutrell 4y ago
  Makes me think this could have been a race condition where these aircraft somehow received the load calculations from other aircraft being calculated at the same time window. Some kind of poorly managed event stream / consumer that assumed no parallel jobs would be run? Or maybe the jobs are deduped by departure / arrival instead of flight number? Seems possible
- AceyMan 4y ago
  >The calculation is done by the plane’s flight computer. Not exactly true. The dispatch release has a flight & performance plan based on the load manifest. It's calculated out of band (here, by the software vendor) and punched into the airplane configuration for departure. If the plan is wrong, the plane flies wrong. GIGO at its finest.
- newZWhoDis 4y ago
  This is why explicit error handling is so critical. Being forced to handle the timeout error would make such a bug obvious.
1101010010 4y ago
Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this time it is human lives that are commodified and have costs externalized unto.
- Game_Ender 4y ago
  In a competitive industry saving money can mean lower fares for customers. It can also mean fewer total planes or engines produced reducing environmental side effects of that activity.
  - terr-dav 4y ago
    It can mean those things, yes, but only as side-effects. The primary effect being to increase shareholder value. There’s no need to have market based competition to keep costs down. Look at USPS for example. That operation is run without a profit motive, serves literally every zip code in the US (by definition), and runs healthily year after year (except for the part where Congress imposed arbitrary pension funding requirements[1] and private capital interests have been chomping at the bit to buy up all the real-estate holdings). They also don’t have UPS’s problem of not putting AC in their delivery vehicles. [1] https://ips-dc.org/how-congress-manufactured-a-postal-crisis-and-how-to-fix-it/ https://ips-dc.org/how-congress-manufactured-a-postal-crisis...
- pixl97 4y ago
  I mean yes, would you rather flights are $10,000 or $100,000 each so the engines are taken apart and rebuilt each time after a flight? Reducing the maintenance interval requirements 'safely' is good for everyone.
  - 1101010010 4y ago
    Let's reduce the intervals even further so flights are only $10 then, right?
NotYourLawyer 4y ago
It’s software. It’s made of bugs.
- wkat4242 4y ago
  So is oil but it works fine :P
nop_slide 4y ago
This other article linked near the end of the source article is even sketchier! It basically boils down to “this flight started nosediving, almost hit the ocean, and we don’t know why”. https://www.nytimes.com/2023/02/13/us/united-maui-flight.html https://www.nytimes.com/2023/02/13/us/united-maui-flight.htm...
jcutrell 4y ago
Seems like something as critical as thrust in this phase of flight should have some kind of cross-check validation. Sounds like a bad single point of failure to me, but maybe there is a cross check I don’t know about that isn’t covered in the story. What I do know is that with something like this, a little could go a long way. I wonder what the inspection and repair for a tail strike is, and whether that cancels out the money saved by minimum viable thrust across the fleet. I’m sure someone is punching the calculator on this to determine that.
Merad 4y ago
I was under the impression that SOP for most aircraft was to take off at full power, and if you happen to be lightly loaded you just get into the air faster. Was I wrong, or is this a recent change? If so, it seems like a concerning way for airlines to cut costs. Having excess power available at take off means there's a greater margin for error to handle emergencies (engine failure, bird strike, etc.).
- salawat 4y ago
  Everyone else's (consumer's/engineer's/pilot's) "safety margin" is an executive's "cost to be optimized". Make no mistake. The Exec/finance class will cut into any margin that you aren't willing to make them directly legally culpable for.
  - lcuff 4y ago
    Yes. Chelsea Sullenberger ("Sully") made the same point in an interview. He said in the early days of the airlines, the CEO was always a pilot, and error margins were set much higher than legally required. Now that CEOs tend NOT to be retired pilots, new regulations are needed to raise the safety margins beyond current regulatory points to what a safety-conscious pilot would choose.
- gdmt 4y ago
  My understanding is that using less than full power prolongs the life span of engine components and makes fatigue-related engine failures less likely.
- sokoloff 4y ago
  Jets have been using reduced power takeoffs for many decades, with good operational results. There’s an AC about it from 35 years ago: https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC25-13.pdf https://www.faa.gov/documentLibrary/media/Advisory_Circular/... Note that these are manufacturer-sanctioned procedures, not applicable to small piston aircraft, which should be using full power/RPM if the AFM/POH calls for that. Note also that the full rated power (and sometimes more) is available in the case of an abnormal or emergency.
roamerz 4y ago
Seems like any modern jet would have a load sensing system built into the landing gear. Something so critical such as takeoff weight would be both estimated by a software system and compared to an onboard system for redundancy. Any discrepancy beyond a set limit would turn on a warning light.
Max_Horstmann 4y ago
> The Alaska captain said that, as for many things in aviation, pilots routinely use an acronym when they do the pre-takeoff “sanity check”: TLAR, which means “That Looks About Right.” Aviation's LGTM
015a 4y ago
> Bret Peyton, Alaska’s on-duty director of operations, immediately ordered no more planes were to take off across the airline’s network. All Alaska flights not already airborne were stopped nationwide. An absolute pro. There's a hundred variations of this story, to varying degrees of criticality and impact; seeing a pattern out of two data points, connecting the dots, making the tremendous call to immediately pull the plug, to stop the world and give engineering time, then diagnosing and triaging the problem in less than a half hour; that's world class reliability engineering.
- raldi 4y ago
  Yes. Even better if there are situations declared in advance to be automatic halts so no call needs to be made in the moment.
  - hef19898 4y ago
    No matter what complex system it is, you always need humans in the loop. Especially for networks and not just single nodes.
    - raldi 4y ago
      I think you misunderstand my comment. I'm saying, for instance, that during a calm period when a group of people can take their time and review each other's reasoning, a rule might be put in effect like, "Two tail strikes within X hours of each other means an automatic, immediate halt." Humans execute that procedure (or override it at the risk of their career) and then decide what to do next, having been in the loop the entire time. Such a rule likely saved an SR-71 once: https://theaviationgeekclub.com/the-story-of-belmont-86-the-sr-71-that-made-an-emergency-landing-in-norway-after-spying-over-murmansk/ https://theaviationgeekclub.com/the-story-of-belmont-86-the-... The natural tendency for military air crews is to complete the mission if humanly possible. To counter this inclination, the Wing Commander had designated certain emergencies sufficiently critical to require immediate landing. This was one of those emergencies.
sithadmin 4y ago
737-900s are notoriously tail-heavy. Some airlines require that they use tail stands when parked, and some airlines (like United) have taken measures to redistribute weight forward of the rear landing gear (by moving one of the economy lavatories so that it sits directly behind first class/in front of most economy seating).
- coin 4y ago
  Height of the landing gear is fixed, they've milked the 737 design by stretching the length making it vulnerable to a tail strike.
- wkat4242 4y ago
  Do 737s in the US still even have first class?. I'm surprised. I've never seen any in Europe in the last 15 years on short haul which is of course what a 737 is used for.
  - therealcamino 4y ago
    Yes, they do, but the domestic US version of first class. Also, the 737s in the article were departing on a 6+ hour flight to Hawaii, so they're not only used for short haul flights. Lots of coast to coast flights of 5ish hours are also on single aisle planes.
lowbloodsugar 4y ago
My fucking washing machine can sense how big a load is in it. Surely the plane could measure how fast it is actually accelerating given its thrust and thus derive its actual mass. If it doesn’t match what it’s been told it should alarm. Also shocking that the system gives an incorrect answer when it is under load. Lack of load testing is a problem, sure, but an architecture that allows the answer to just be wrong is fundamentally flawed.
drewg123 4y ago
I don't understand why an airplane being heavier than what was estimated by the software would cause the tail to drag. I would assume it would cause the plane to not have sufficient thrust to take off, or to use more runway. But having the tail drag makes it seem like it had too much thrust, which translated to a steeper take off than anticipated... Can somebody explain?
- cmurf 4y ago
  In order to achieve the required lift force for takeoff, a higher angle of attack was needed to compensate for the incorrect low rotation airspeed. In effect, pilots raised to a higher nose-high attitude to achieve lift off by the expected point, and this resulted in a tail strike.
- lcuff 4y ago
  Not an expert, but here's my best guess. When I learned to fly a small plane, you pulled back the yoke at a specific speed. This is called 'rotation' and lifts the nose of the plane off the ground, (wheels are still on the ground) and changes the attitude of the wings, creating substantially more lift for the speed. The plane then lifts quickly off the runway. Presumably different possible weights in a larger plane would cause you to choose different speeds at which to rotate. I also presume rotation is also a more gradual process in a larger plane (a few seconds long???), so the 'early' part of the rotation occurs, gets the plane off the ground, and allows the space for the angle to become steeper in the later part of the rotation. Would be happy for an expert to review and comment on this.
- pqdbr 4y ago
  Let's say your target rotation angle is 10 degrees, so at the specified speed (called VR) the pilots 'rotate' the aircraft nose up by 10 degrees. When they start the 'rotate' maneuver, the nose of the aircraft doesn't immediately go to (for instance) 10 degrees. It's a progression. As soon as the nose starts pitching up (by 1 degree for instance), the aircraft is already generating lift to pull the aircraft up. If the speed is set right (so you're fast), when you get to that 10 degrees angle of attack, your tail has already cleared the runway by a very safe margin, because you've been producing X lift from 0 to 10 degrees of rotation. However, if your speed is set too slow, you're not generating as much lift, so the aircraft will be pulled up slower (you have been producing Y% less lift from 0 to 10 degrees of rotation). This means that by the time you reach your designated 10 degrees of angle of attack, you haven't cleared the tail yet, and it will hit the runway.
signal_space 4y ago
calculation of total weight ends up being a pde right?
bogomipz 4y ago
The article states the following about the software: >"It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity. >In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be ready to lift off." Given that overhead bins are regularly maxed out with carry-on luggage now since airlines began charging for checked bags, how are they able to accurately account for the weight and balance? Airlines seem to almost never weigh customers carry-on at check in.
- ceejayoz 4y ago
  "Regularly maxed-out" translates to "fairly predictable". If the 20 people in first class all have 500 pounds of uranium each they've snuck onboard, you have a problem, but otherwise, there's plenty of wiggle room to account for minor variability of this kind. They'll have a decent idea of the average weight of the average passenger (https://airinsight.com/the-pending-new-faa-weight-balance-rules/ https://airinsight.com/the-pending-new-faa-weight-balance-ru...). If you're chartered by an anorexia treatment facility to move 300 patients, you may put a little extra attention to the exact values. > The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively. This includes an extra ten pounds for winter and five pounds for summer. This also includes 16 pounds for personal items, up from ten. Airlines must increase the average weight of female passengers and their carry-ons from 145 pounds to 179 pounds in summer, and from 150 pounds to 184 pounds in winter. The average weight for males with carry-ons is increased from 185 pounds in summer to 200 pounds, and from 190 pounds to 205 pounds in winter.
batch12 4y ago
This would be an effective side channel attack to use against one's enemies. Even if the true source was never known, the chaos and distrust in infrastructure after the crash would be nice second-order effects.
Animats 4y ago
"Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system." Now that's strange. Anyone have more details? Why should there be any connection between the calculations for different aircraft?
OliverJones 4y ago
A weight discrepancy of ten to fifteen TONS???? That's no small mistake. It's not a rounding error on the weight of the soda cans loaded into the galley for the flight. That's most of the passengers. It's most of the freight. It's heavy enough to be a spare engine in the hold. It's a goodsize fraction of the fuel needed to fly from Seattle to Honolulu. It's certainly enough to foul up the mandatory weight-and-balance computation the pilot in command is required to do. Somehow the input to this software package missed something big. It would be interesting to know exactly what was missed.
orasis 4y ago
Why is there so much aviation disaster porn on here recently? This is turning into the same sort of high drama news site that most of us are trying to get away from.
- atom058 4y ago
  HN readers notice more aviation stories, because they have been reading more aviation stories than before? I notice this behaviour pattern in myself now and again. In this instance, at least it was a software bug that was the cause. So quite suitable for HN.
dehrmann 4y ago
> Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. Interesting that baggage weight changes by destination. I'm sure passenger weight does too, but for different reasons.
mike_hock 4y ago
\> open website and start reading \> an ad (or whatever, didn't read it) popup appears \> close popup, try to start reading again \> another popup appears and blurs the article \> close website, I wasn't that interested anyway
janalsncm 4y ago
Dumb question. A heavier aircraft should be slower to accelerate. Given that these planes were at least 13% heavier than was thought and the force applied during takeoff is known, shouldn’t there have been some kind of alarm? “We think you should be going 120 mph but you’re actually going 100 mph”. And based on the inferred mass, can’t you directly calculate what the takeoff velocity should be?
ris 4y ago
It's a bit of a shame the way journalism works this story will do the rounds with the headlines associating "Alaska Airlines" with "tailstrikes" ("takeoff incidents", whatever term gets used), while actually it looks like Alaska Airlines were the heroes of this story.
konschubert 4y ago
> Still, the mishaps point to the need for more vigilance by pilots in checking automated data. No. You don’t get to build complex configurations that only work with supporting software and then BLAME THE USER when the software fails. “Ah, but you didn’t follow rule 137 in the 2000-page manual that nobody reads because computers made it obsolete” is a deflection of responsibility from the owners to the operators and it’s not okay.
ldarby 4y ago
> If the automatically loaded data strikes either pilot as not right, they can make a manual request for takeoff data from the airline operations center. “But 99.8% of the time, the data is accurate,” he said. > Alaska’s Peyton said “several crews noticed the error and notified dispatch.” > The pilot at American Airlines said “requesting manual data is not standard” and that if there’s a glitch, naturally some pilot somewhere is going to miss it. > “Not everyone gets eight hours sleep the night before. Someone is going through a divorce. Someone is not so sharp that morning,” he said. “The sanity check isn’t perfect every day of the week.” So they already knew (or should have known) about this glitch, and did nothing about it until it started causing damage to aircraft. I would have thought the pilots who reported it earlier would have also predicted the outcome and got it fixed then. From my lay perspective, it's obvious that if aircraft is heavier than the flight computer was told it is, then something bad is going to happen (e.g. could have been complete failure to take off and crash at the end of the runway). Well, maybe such incidents where pilots do make the prediction, and get it fixed before it causes the problem they predicted, never make it to the news...

242 points by dlgeek 4y ago

If weight calculation is so crucial for take off, it seems like redundancy of the weight calculation is needed - a different weight calculation from a different company using a different method to verify against the main one.

tzs 4y ago
Would it be possible to actually weigh the loaded plane to check the calculations, say with something like a bigger version of truck scales that the plane could taxi over?
- pqdbr 4y ago
  Why doesn't the plane weight itself? Couldn't there be sensors in the landing gear or something?

Aloha 4y ago

This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground stop * The problem was figured out quickly and a work-around was developed * Flights Resumed after successfully deploying the work around to the 'production process' * A patch was quickly developed and deployed once the underlying bug was uncovered. I think everyone (but perhaps the developers) comes out looking like a champ. Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable - particularly if I was held to account for undetected failures. The other part that I'd note here, is process is just as important as software could process alone have caught this failure without the tail strikes? possibly, and its something worth looking into further - but only if it doesn't add an unacceptable workload burden to pilots workload that could otherwise compromise safety further.

basch 4y ago
The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes. >Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. > Both planes headed down the runway with less power and at lower speed than they should have. And with the jets judged lighter than they actually were, the pilots rotated too early. >The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins. Both aircraft got airborne well within safety limits despite the lower thrust. It’s also probably lucky that two Hawai’i bound flights both didn’t notice the discrepancy, or two bumps wouldn’t have bee as suspicious. Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human. Like all complex failures at least 5 things had to happen. Software update that morning, some kind of hard to simulate resource issue in processing, overweight flights, pilot not pricing plane data appeared light, pilot turning early despite low speed, two in a row.
- pxmpxm 4y ago
  >Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human. Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.
  - hooverd 4y ago
    I thought a lot of safety thinking was built around acknowledging that humans don't do everything perfectly all the time.

fortran77 4y ago

[flagged]

JamesonNetworks 4y ago
The article says the reason the weights were wrong was due to high load on the DynamicSource software. Seems more like a concurrency issue
deleted 4y ago
[deleted]
loeg 4y ago
Both aircraft seat about 180 pax. To be 20,000 lbs off in your hypothesis, each passenger would have to be 110 lbs heavier than modeled, on average. For 30,000 lbs, 165 lbs.
- fortran77 4y ago
  See: https://airinsight.com/the-pending-new-faa-weight-balance-rules/ https://airinsight.com/the-pending-new-faa-weight-balance-ru... There were recent changes to the weight and balance rules because of increased weight. It's certainly possible this was a factor.
  - loeg 4y ago
    > The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively. This is a lot smaller in magnitude than +110-165 lb.

hodgesrm 4y ago

That was an impressively quick call based on ability to reason about unexpected behavior of the system. It should give pause to anyone who wants to take the human out of the loop in safety critical software.

guitarbill 4y ago
Agreed, although it takes an individual with a huge amount of integrity or a culture that values and promotes integrity to make a call like this: > “At that point, two in a row like that, that’s when I said, ‘No, we’re done,'” said Peyton. “That’s when I stopped things.” Hats off to Peyton, and possibly Alaska. I would also love to hear about times when something similar happened, and it was discovered that nothing was wrong.
- CPLX 4y ago
  Amazingly, this really is the approach that has characterized all aspects of aviation for decades. It’s incredibly admirable and worth emulating in many other situations. It’s also fragile, and the fact that it strikes many as an uncommon level of integrity for a business decision is precisely why aviation needs to be shielded from having the prevailing business culture sneak in, like we have seen happen at Boeing in a few highly notable instances.
  - hodgesrm 4y ago
    This type of judgment call by a single person has also helped avoid nuclear war on at least two documented occasions. [0,1] Again, food for thought. [0] https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alarm_incident https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar... [1] https://en.wikipedia.org/wiki/Able_Archer_83 https://en.wikipedia.org/wiki/Able_Archer_83

neonate 4y ago

https://archive.md/XOgpi https://archive.md/XOgpi

sudhirj 4y ago

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

cratermoon 4y ago
Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs
- djanogo 4y ago
  This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them. "the bug was missed because it only presented when many aircraft at the same time were using the system"
  - hn_throwaway_99 4y ago
    There is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/ https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare.... Obviously sessions should be independent and not sharing data, but that's why it was a bug.
  - usea 4y ago
    I believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.
- howinteresting 4y ago

Tempest1981 4y ago

Curious about the actual bug: > the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system > the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings. Multithreading/contention issue? But how would that alter the weights? > The software code was permanently repaired about five hours later That's surprisingly fast, isn't it?

sidlls 4y ago
Multithreading/contention issue? But how would that alter the weights? Off the top of my head: failure to sum values correctly, misreporting the weight to the target plane (reading values from another plane’s weight data).
hansvm 4y ago
Could be something as simple as the first concurrency 101 example: atomic addition. The += operator behaves poorly on concurrent access.
- Jtsummers 4y ago
  We had that error in an application at work. Essentially data was coming in on multiple threads, but there was a shared value between them that was being incremented and decremented. Without a lock and without using atomics, the tally could be incorrect. But under low load this didn't appear because the odds of interrupting +=/-=/++/-- was low. Under high load, however, the odds of a thread being interrupted mid-operation increased and so it appeared in production. At least in our case it was just an annoyance (no critical decisions were based on the value, it was merely informative). If there are multiple data streams feeding in weight data per aircraft, under a low aircraft load (the scenarios they apparently tested) the odds of interruption during an operation like += can be low enough to not see the issue. Under high load, though, the odds of interruption and incorrect tallying increases substantially.

deleted 4y ago

[deleted]

justinclift 4y ago

With Boeing's recent 737 family history (MAX fiasco, etc), it's very concerning to see "software errors" still happening in operation. :(

ceejayoz 4y ago
How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable? This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to account for the root cause.
- sopooneo 4y ago
  I'm with you that a couple layers of the "safety in depth" worked impressively well here. But a bug of this types points to egregiously bad process and standards with the software vendor. And the airlines or FAA should have audits to catch that well upstream of this incident.
- abudabi123 4y ago
  The SpaceX rockets launch routinely with automation at takeoff and landing. There is no need for human heroic drama in the loop when the software is AAA grade.
  - ceejayoz 4y ago
    > The SpaceX rockets launch routinely with automation at takeoff and landing. Sure. And there are bugs in their code, too. https://www.theverge.com/2019/9/3/20847243/spacex-starlink-satellite-european-space-agency-aeolus-conjunction-space-debris https://www.theverge.com/2019/9/3/20847243/spacex-starlink-s... "SpaceX acknowledges that it failed to communicate due to the bug and missed the emails about a higher probability of collision. Finally, on Monday morning in Europe, ESA made the call and used the thrusters on Aeolus to raise the satellite’s orbit by about 984 feet (300 meters) without waiting for SpaceX to take corrective action."

fwlr 4y ago

“the bug was missed because it only presented when many aircraft at the same time were using the system.” The system reports data on number of passengers, weight of cargo, plane balance, etc., to the pilots. The calculation is done by the plane’s flight computer. How can it be off by 20,000 pounds, but only under heavy server load? The explanation that comes to mind is that DynamicSource has a subservice for each source of weight and one of those subservices crashed under heavy usage. So the top-level aggregate-and-report service got an error from one subservice and said “well, guess it’s zero, lol”?

jcutrell 4y ago
Makes me think this could have been a race condition where these aircraft somehow received the load calculations from other aircraft being calculated at the same time window. Some kind of poorly managed event stream / consumer that assumed no parallel jobs would be run? Or maybe the jobs are deduped by departure / arrival instead of flight number? Seems possible
AceyMan 4y ago
>The calculation is done by the plane’s flight computer. Not exactly true. The dispatch release has a flight & performance plan based on the load manifest. It's calculated out of band (here, by the software vendor) and punched into the airplane configuration for departure. If the plan is wrong, the plane flies wrong. GIGO at its finest.
newZWhoDis 4y ago
This is why explicit error handling is so critical. Being forced to handle the timeout error would make such a bug obvious.

1101010010 4y ago

Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article: > the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. This is not an accident, but rather a feature of capitalism, this time it is human lives that are commodified and have costs externalized unto.

Game_Ender 4y ago
In a competitive industry saving money can mean lower fares for customers. It can also mean fewer total planes or engines produced reducing environmental side effects of that activity.
- terr-dav 4y ago
  It can mean those things, yes, but only as side-effects. The primary effect being to increase shareholder value. There’s no need to have market based competition to keep costs down. Look at USPS for example. That operation is run without a profit motive, serves literally every zip code in the US (by definition), and runs healthily year after year (except for the part where Congress imposed arbitrary pension funding requirements[1] and private capital interests have been chomping at the bit to buy up all the real-estate holdings). They also don’t have UPS’s problem of not putting AC in their delivery vehicles. [1] https://ips-dc.org/how-congress-manufactured-a-postal-crisis-and-how-to-fix-it/ https://ips-dc.org/how-congress-manufactured-a-postal-crisis...
pixl97 4y ago
I mean yes, would you rather flights are $10,000 or $100,000 each so the engines are taken apart and rebuilt each time after a flight? Reducing the maintenance interval requirements 'safely' is good for everyone.
- 1101010010 4y ago
  Let's reduce the intervals even further so flights are only $10 then, right?

NotYourLawyer 4y ago

It’s software. It’s made of bugs.

wkat4242 4y ago
So is oil but it works fine :P

nop_slide 4y ago

This other article linked near the end of the source article is even sketchier! It basically boils down to “this flight started nosediving, almost hit the ocean, and we don’t know why”. https://www.nytimes.com/2023/02/13/us/united-maui-flight.html https://www.nytimes.com/2023/02/13/us/united-maui-flight.htm...

jcutrell 4y ago

Seems like something as critical as thrust in this phase of flight should have some kind of cross-check validation. Sounds like a bad single point of failure to me, but maybe there is a cross check I don’t know about that isn’t covered in the story. What I do know is that with something like this, a little could go a long way. I wonder what the inspection and repair for a tail strike is, and whether that cancels out the money saved by minimum viable thrust across the fleet. I’m sure someone is punching the calculator on this to determine that.

Merad 4y ago

I was under the impression that SOP for most aircraft was to take off at full power, and if you happen to be lightly loaded you just get into the air faster. Was I wrong, or is this a recent change? If so, it seems like a concerning way for airlines to cut costs. Having excess power available at take off means there's a greater margin for error to handle emergencies (engine failure, bird strike, etc.).

salawat 4y ago
Everyone else's (consumer's/engineer's/pilot's) "safety margin" is an executive's "cost to be optimized". Make no mistake. The Exec/finance class will cut into any margin that you aren't willing to make them directly legally culpable for.
- lcuff 4y ago
  Yes. Chelsea Sullenberger ("Sully") made the same point in an interview. He said in the early days of the airlines, the CEO was always a pilot, and error margins were set much higher than legally required. Now that CEOs tend NOT to be retired pilots, new regulations are needed to raise the safety margins beyond current regulatory points to what a safety-conscious pilot would choose.
gdmt 4y ago
My understanding is that using less than full power prolongs the life span of engine components and makes fatigue-related engine failures less likely.
sokoloff 4y ago
Jets have been using reduced power takeoffs for many decades, with good operational results. There’s an AC about it from 35 years ago: https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC25-13.pdf https://www.faa.gov/documentLibrary/media/Advisory_Circular/... Note that these are manufacturer-sanctioned procedures, not applicable to small piston aircraft, which should be using full power/RPM if the AFM/POH calls for that. Note also that the full rated power (and sometimes more) is available in the case of an abnormal or emergency.

roamerz 4y ago

Seems like any modern jet would have a load sensing system built into the landing gear. Something so critical such as takeoff weight would be both estimated by a software system and compared to an onboard system for redundancy. Any discrepancy beyond a set limit would turn on a warning light.

Max_Horstmann 4y ago

> The Alaska captain said that, as for many things in aviation, pilots routinely use an acronym when they do the pre-takeoff “sanity check”: TLAR, which means “That Looks About Right.” Aviation's LGTM

015a 4y ago

> Bret Peyton, Alaska’s on-duty director of operations, immediately ordered no more planes were to take off across the airline’s network. All Alaska flights not already airborne were stopped nationwide. An absolute pro. There's a hundred variations of this story, to varying degrees of criticality and impact; seeing a pattern out of two data points, connecting the dots, making the tremendous call to immediately pull the plug, to stop the world and give engineering time, then diagnosing and triaging the problem in less than a half hour; that's world class reliability engineering.

raldi 4y ago
Yes. Even better if there are situations declared in advance to be automatic halts so no call needs to be made in the moment.
- hef19898 4y ago
  No matter what complex system it is, you always need humans in the loop. Especially for networks and not just single nodes.
  - raldi 4y ago
    I think you misunderstand my comment. I'm saying, for instance, that during a calm period when a group of people can take their time and review each other's reasoning, a rule might be put in effect like, "Two tail strikes within X hours of each other means an automatic, immediate halt." Humans execute that procedure (or override it at the risk of their career) and then decide what to do next, having been in the loop the entire time. Such a rule likely saved an SR-71 once: https://theaviationgeekclub.com/the-story-of-belmont-86-the-sr-71-that-made-an-emergency-landing-in-norway-after-spying-over-murmansk/ https://theaviationgeekclub.com/the-story-of-belmont-86-the-... The natural tendency for military air crews is to complete the mission if humanly possible. To counter this inclination, the Wing Commander had designated certain emergencies sufficiently critical to require immediate landing. This was one of those emergencies.

sithadmin 4y ago

737-900s are notoriously tail-heavy. Some airlines require that they use tail stands when parked, and some airlines (like United) have taken measures to redistribute weight forward of the rear landing gear (by moving one of the economy lavatories so that it sits directly behind first class/in front of most economy seating).

coin 4y ago
Height of the landing gear is fixed, they've milked the 737 design by stretching the length making it vulnerable to a tail strike.
wkat4242 4y ago
Do 737s in the US still even have first class?. I'm surprised. I've never seen any in Europe in the last 15 years on short haul which is of course what a 737 is used for.
- therealcamino 4y ago
  Yes, they do, but the domestic US version of first class. Also, the 737s in the article were departing on a 6+ hour flight to Hawaii, so they're not only used for short haul flights. Lots of coast to coast flights of 5ish hours are also on single aisle planes.

lowbloodsugar 4y ago

My fucking washing machine can sense how big a load is in it. Surely the plane could measure how fast it is actually accelerating given its thrust and thus derive its actual mass. If it doesn’t match what it’s been told it should alarm. Also shocking that the system gives an incorrect answer when it is under load. Lack of load testing is a problem, sure, but an architecture that allows the answer to just be wrong is fundamentally flawed.

drewg123 4y ago

I don't understand why an airplane being heavier than what was estimated by the software would cause the tail to drag. I would assume it would cause the plane to not have sufficient thrust to take off, or to use more runway. But having the tail drag makes it seem like it had too much thrust, which translated to a steeper take off than anticipated... Can somebody explain?

cmurf 4y ago
In order to achieve the required lift force for takeoff, a higher angle of attack was needed to compensate for the incorrect low rotation airspeed. In effect, pilots raised to a higher nose-high attitude to achieve lift off by the expected point, and this resulted in a tail strike.
lcuff 4y ago
Not an expert, but here's my best guess. When I learned to fly a small plane, you pulled back the yoke at a specific speed. This is called 'rotation' and lifts the nose of the plane off the ground, (wheels are still on the ground) and changes the attitude of the wings, creating substantially more lift for the speed. The plane then lifts quickly off the runway. Presumably different possible weights in a larger plane would cause you to choose different speeds at which to rotate. I also presume rotation is also a more gradual process in a larger plane (a few seconds long???), so the 'early' part of the rotation occurs, gets the plane off the ground, and allows the space for the angle to become steeper in the later part of the rotation. Would be happy for an expert to review and comment on this.
pqdbr 4y ago
Let's say your target rotation angle is 10 degrees, so at the specified speed (called VR) the pilots 'rotate' the aircraft nose up by 10 degrees. When they start the 'rotate' maneuver, the nose of the aircraft doesn't immediately go to (for instance) 10 degrees. It's a progression. As soon as the nose starts pitching up (by 1 degree for instance), the aircraft is already generating lift to pull the aircraft up. If the speed is set right (so you're fast), when you get to that 10 degrees angle of attack, your tail has already cleared the runway by a very safe margin, because you've been producing X lift from 0 to 10 degrees of rotation. However, if your speed is set too slow, you're not generating as much lift, so the aircraft will be pulled up slower (you have been producing Y% less lift from 0 to 10 degrees of rotation). This means that by the time you reach your designated 10 degrees of angle of attack, you haven't cleared the tail yet, and it will hit the runway.

signal_space 4y ago

calculation of total weight ends up being a pde right?

bogomipz 4y ago

The article states the following about the software: >"It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity. >In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be ready to lift off." Given that overhead bins are regularly maxed out with carry-on luggage now since airlines began charging for checked bags, how are they able to accurately account for the weight and balance? Airlines seem to almost never weigh customers carry-on at check in.

ceejayoz 4y ago
"Regularly maxed-out" translates to "fairly predictable". If the 20 people in first class all have 500 pounds of uranium each they've snuck onboard, you have a problem, but otherwise, there's plenty of wiggle room to account for minor variability of this kind. They'll have a decent idea of the average weight of the average passenger (https://airinsight.com/the-pending-new-faa-weight-balance-rules/ https://airinsight.com/the-pending-new-faa-weight-balance-ru...). If you're chartered by an anorexia treatment facility to move 300 patients, you may put a little extra attention to the exact values. > The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively. This includes an extra ten pounds for winter and five pounds for summer. This also includes 16 pounds for personal items, up from ten. Airlines must increase the average weight of female passengers and their carry-ons from 145 pounds to 179 pounds in summer, and from 150 pounds to 184 pounds in winter. The average weight for males with carry-ons is increased from 185 pounds in summer to 200 pounds, and from 190 pounds to 205 pounds in winter.

batch12 4y ago

This would be an effective side channel attack to use against one's enemies. Even if the true source was never known, the chaos and distrust in infrastructure after the crash would be nice second-order effects.

Animats 4y ago

"Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system." Now that's strange. Anyone have more details? Why should there be any connection between the calculations for different aircraft?

OliverJones 4y ago

A weight discrepancy of ten to fifteen TONS???? That's no small mistake. It's not a rounding error on the weight of the soda cans loaded into the galley for the flight. That's most of the passengers. It's most of the freight. It's heavy enough to be a spare engine in the hold. It's a goodsize fraction of the fuel needed to fly from Seattle to Honolulu. It's certainly enough to foul up the mandatory weight-and-balance computation the pilot in command is required to do. Somehow the input to this software package missed something big. It would be interesting to know exactly what was missed.

orasis 4y ago

Why is there so much aviation disaster porn on here recently? This is turning into the same sort of high drama news site that most of us are trying to get away from.

atom058 4y ago
HN readers notice more aviation stories, because they have been reading more aviation stories than before? I notice this behaviour pattern in myself now and again. In this instance, at least it was a software bug that was the cause. So quite suitable for HN.

dehrmann 4y ago

> Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. Interesting that baggage weight changes by destination. I'm sure passenger weight does too, but for different reasons.

mike_hock 4y ago

\> open website and start reading \> an ad (or whatever, didn't read it) popup appears \> close popup, try to start reading again \> another popup appears and blurs the article \> close website, I wasn't that interested anyway

janalsncm 4y ago

Dumb question. A heavier aircraft should be slower to accelerate. Given that these planes were at least 13% heavier than was thought and the force applied during takeoff is known, shouldn’t there have been some kind of alarm? “We think you should be going 120 mph but you’re actually going 100 mph”. And based on the inferred mass, can’t you directly calculate what the takeoff velocity should be?

ris 4y ago

It's a bit of a shame the way journalism works this story will do the rounds with the headlines associating "Alaska Airlines" with "tailstrikes" ("takeoff incidents", whatever term gets used), while actually it looks like Alaska Airlines were the heroes of this story.

konschubert 4y ago

> Still, the mishaps point to the need for more vigilance by pilots in checking automated data. No. You don’t get to build complex configurations that only work with supporting software and then BLAME THE USER when the software fails. “Ah, but you didn’t follow rule 137 in the 2000-page manual that nobody reads because computers made it obsolete” is a deflection of responsibility from the owners to the operators and it’s not okay.

ldarby 4y ago

> If the automatically loaded data strikes either pilot as not right, they can make a manual request for takeoff data from the airline operations center. “But 99.8% of the time, the data is accurate,” he said. > Alaska’s Peyton said “several crews noticed the error and notified dispatch.” > The pilot at American Airlines said “requesting manual data is not standard” and that if there’s a glitch, naturally some pilot somewhere is going to miss it. > “Not everyone gets eight hours sleep the night before. Someone is going through a divorce. Someone is not so sharp that morning,” he said. “The sanity check isn’t perfect every day of the week.” So they already knew (or should have known) about this glitch, and did nothing about it until it started causing damage to aircraft. I would have thought the pilots who reported it earlier would have also predicted the outcome and got it fixed then. From my lay perspective, it's obvious that if aircraft is heavier than the flight computer was told it is, then something bad is going to happen (e.g. could have been complete failure to take off and crash at the end of the runway). Well, maybe such incidents where pilots do make the prediction, and get it fixed before it causes the problem they predicted, never make it to the news...