4 ms·

This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground

by Aloha 4y ago

This seems like mostly how it ought to work - * There were two unusual events in very short order * Someone quickly noticed and gave the order to go to ground stop * The problem was figured out quickly and a work-around was developed * Flights Resumed after successfully deploying the work around to the 'production process' * A patch was quickly developed and deployed once the underlying bug was uncovered. I think everyone (but perhaps the developers) comes out looking like a champ. Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable - particularly if I was held to account for undetected failures. The other part that I'd note here, is process is just as important as software could process alone have caught this failure without the tail strikes? possibly, and its something worth looking into further - but only if it doesn't add an unacceptable workload burden to pilots workload that could otherwise compromise safety further.

basch 4y ago
The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes. >Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. > Both planes headed down the runway with less power and at lower speed than they should have. And with the jets judged lighter than they actually were, the pilots rotated too early. >The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins. Both aircraft got airborne well within safety limits despite the lower thrust. It’s also probably lucky that two Hawai’i bound flights both didn’t notice the discrepancy, or two bumps wouldn’t have bee as suspicious. Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human. Like all complex failures at least 5 things had to happen. Software update that morning, some kind of hard to simulate resource issue in processing, overweight flights, pilot not pricing plane data appeared light, pilot turning early despite low speed, two in a row.
- pxmpxm 4y ago
  >Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human. Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.
  - hooverd 4y ago
    I thought a lot of safety thinking was built around acknowledging that humans don't do everything perfectly all the time.
    - ddulaney 4y ago
      It does, and one class of mitigation is to improve automation, which airlines (and their regulators) do all the time. But another class of mitigations is to make the human less error-prone in the first place. This is especially important in preventing failures that were caused by improper management of the automation. Hence the emphasis on rest periods, training, and other mitigations on the human side. In the "swiss cheese" model, a disaster needs to slip through both the human's defenses and the automation's defenses to actually happen. It makes sense to improve both of those layers of defense against disaster, while acknowledging that humans will never be perfect. But at least they can be awake and knowledgable.
  - NovemberWhiskey 4y ago
    Right. Literally every pilot in the US has been taught the IMSAFE mnemonic to assess fitness to fly, even before their first solo in a single-engine piston: Illness, Medication, Stress, Alcohol, Fatigue and Emotion.
  - basch 4y ago
    I just don’t see how it fit the article. Unless they are surmising the pilots poorly sanityized the data going into the computer because they were having an off day.
sidlls 4y ago
This is a failure in the development of the software that good process in the purchaser’s handbook saved them from. That there were no tests for this system under high load indicates a poor engineering practice at DynamicSource. They don’t deserve any kudos for a post-hoc patch that a test in production discovered the bug for when the system is a safety critical system like this.
- Aloha 4y ago
  They may have thought their test coverage, covered it adequately - who knows, the success was on Alaska's part. Also process, procedure and more importantly people are always the last line of defense against failure.
  - lupire 4y ago
    Making excuses for the failures in the first line of defense is the first domino in catastrophic failure. Normalization of deviance.
    - sidlls 4y ago
      I don't think OP was making excuses. It is true that the software vendor's missing test case indicates a poor quality development environment and that humans are the last line of defense. No development process is perfectly resilient against errors, even good quality processes.
- admax88qqq 4y ago
  My completely uninformed opinion is the design is flawed if the calculations across planes can interact. Is this some single tenant cloud based system with poor multithreaded safety? Like why would the increased system load change the results. The calculations should be independent and designed in a way where they can't interact.
- wly_cdgr 4y ago
  What do you propose they should have done? Keep testing until they were certain they caught every bug? :)
cm2187 4y ago
The bug is in the weight of the payload of the plane. I don't believe pilots have any way to measure the actual weight of the payload of the plane, they rely on that sort of estimates to determine how much fuel they need, take-off speed, etc. It's not really a loss of authority. Bad weight estimates have resulted in crashes in the past.
- wkat4242 4y ago
  I wonder if it would be possible to put a weighing pad at airports for a double-check. Like on the access ramp to the runway, meaning you'd only need a handful per airport and planes pass them naturally.
  - wolrah 4y ago
    > I wonder if it would be possible to put a weighing pad at airports for a double-check. A few modern pickup trucks have a system for measuring how much weight is in the bed or being transferred through the tongue of a trailer because of how common it is for people to overload their trucks. AFAIK these systems just use the same sorts of suspension height sensors that have been used for auto-levelling for years and are factory calibrated to know that if the rear end squats by X distance then there must be Y load in it. Seems like something along those lines would be technically easy to retrofit to large aircraft, possibly even as a software update if they already have some relevant sensors.
t0mas88 4y ago
> Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable Pilot here :) the journalist got it a bit wrong, performance data isn't primarily about how much thrust to use. It is part of the calculations, but crews are free to deviate from reduced thrust calculations and will do so. This incident is much more about the calculated speeds to use and aircraft trim settings. That's what caused the tail strikes, trying to lift off at a speed that is too low for the weight. Those numbers are required by every airliner taking off. And getting them from software is standard everywhere. The alternative is doing the calculations manually using paper charts, every pilot is required to know how to do that, but nobody does it that way for commercial operations.
- Aloha 4y ago
  Thank you for the thoughtful comment! I thought I was missing something in the way the story read, I'm glad to know I was.
- konschubert 4y ago
  How do you feel about the pilots getting blamed for relying on the software? > “Still, the mishaps point to the need for more vigilance by pilots in checking automated data.”
  - t0mas88 4y ago
    In aviation incidents there isn't really a "blame", it's about finding all contributing factors and working together to eliminate those. The intention of investigations is never to assign blame, because that stands in the way of finding all factors and improving them together. So the conclusion "pilots should check the automated data" is a correct one. It's not the only cause, but it is one of the factors and thus it needs to be discussed. In commercial aviation there can never be a single point of failure that could lead to catastrophic outcome. The performance calculations are such a thing, so there must be validation steps. In many airlines for example if the performance calculations are done manually (or by an app based on manual input) they need to be done by both pilots independently and the results cross-checked. Otherwise a single typo could have a big impact.
_moof 4y ago
If it makes you feel any better, the people providing these calculations to the pilots are FAA-certified airline dispatchers. That means they've passed the same exam that airline pilots have to take, minus the flying. They may not have personal experience flying the airplane but they do have to know it inside and out. And if the dispatchers weren't running the numbers, the pilots would be running them using the same software and/or tables, so there really isn't much difference.

by Aloha 4y ago

basch 4y ago

The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. >Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy. > That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes. >Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. > Both planes headed down the runway with less power and at lower speed than they should have. And with the jets judged lighter than they actually were, the pilots rotated too early. >The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins. Both aircraft got airborne well within safety limits despite the lower thrust. It’s also probably lucky that two Hawai’i bound flights both didn’t notice the discrepancy, or two bumps wouldn’t have bee as suspicious. Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human. Like all complex failures at least 5 things had to happen. Software update that morning, some kind of hard to simulate resource issue in processing, overweight flights, pilot not pricing plane data appeared light, pilot turning early despite low speed, two in a row.

pxmpxm 4y ago
>Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human. Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.
- hooverd 4y ago
  I thought a lot of safety thinking was built around acknowledging that humans don't do everything perfectly all the time.
  - ddulaney 4y ago
    It does, and one class of mitigation is to improve automation, which airlines (and their regulators) do all the time. But another class of mitigations is to make the human less error-prone in the first place. This is especially important in preventing failures that were caused by improper management of the automation. Hence the emphasis on rest periods, training, and other mitigations on the human side. In the "swiss cheese" model, a disaster needs to slip through both the human's defenses and the automation's defenses to actually happen. It makes sense to improve both of those layers of defense against disaster, while acknowledging that humans will never be perfect. But at least they can be awake and knowledgable.
- NovemberWhiskey 4y ago
  Right. Literally every pilot in the US has been taught the IMSAFE mnemonic to assess fitness to fly, even before their first solo in a single-engine piston: Illness, Medication, Stress, Alcohol, Fatigue and Emotion.
- basch 4y ago
  I just don’t see how it fit the article. Unless they are surmising the pilots poorly sanityized the data going into the computer because they were having an off day.

sidlls 4y ago

This is a failure in the development of the software that good process in the purchaser’s handbook saved them from. That there were no tests for this system under high load indicates a poor engineering practice at DynamicSource. They don’t deserve any kudos for a post-hoc patch that a test in production discovered the bug for when the system is a safety critical system like this.

Aloha 4y ago
They may have thought their test coverage, covered it adequately - who knows, the success was on Alaska's part. Also process, procedure and more importantly people are always the last line of defense against failure.
- lupire 4y ago
  Making excuses for the failures in the first line of defense is the first domino in catastrophic failure. Normalization of deviance.
  - sidlls 4y ago
    I don't think OP was making excuses. It is true that the software vendor's missing test case indicates a poor quality development environment and that humans are the last line of defense. No development process is perfectly resilient against errors, even good quality processes.
admax88qqq 4y ago
My completely uninformed opinion is the design is flawed if the calculations across planes can interact. Is this some single tenant cloud based system with poor multithreaded safety? Like why would the increased system load change the results. The calculations should be independent and designed in a way where they can't interact.
wly_cdgr 4y ago
What do you propose they should have done? Keep testing until they were certain they caught every bug? :)

cm2187 4y ago

The bug is in the weight of the payload of the plane. I don't believe pilots have any way to measure the actual weight of the payload of the plane, they rely on that sort of estimates to determine how much fuel they need, take-off speed, etc. It's not really a loss of authority. Bad weight estimates have resulted in crashes in the past.

wkat4242 4y ago
I wonder if it would be possible to put a weighing pad at airports for a double-check. Like on the access ramp to the runway, meaning you'd only need a handful per airport and planes pass them naturally.
- wolrah 4y ago
  > I wonder if it would be possible to put a weighing pad at airports for a double-check. A few modern pickup trucks have a system for measuring how much weight is in the bed or being transferred through the tongue of a trailer because of how common it is for people to overload their trucks. AFAIK these systems just use the same sorts of suspension height sensors that have been used for auto-levelling for years and are factory calibrated to know that if the rear end squats by X distance then there must be Y load in it. Seems like something along those lines would be technically easy to retrofit to large aircraft, possibly even as a software update if they already have some relevant sensors.

t0mas88 4y ago

> Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable Pilot here :) the journalist got it a bit wrong, performance data isn't primarily about how much thrust to use. It is part of the calculations, but crews are free to deviate from reduced thrust calculations and will do so. This incident is much more about the calculated speeds to use and aircraft trim settings. That's what caused the tail strikes, trying to lift off at a speed that is too low for the weight. Those numbers are required by every airliner taking off. And getting them from software is standard everywhere. The alternative is doing the calculations manually using paper charts, every pilot is required to know how to do that, but nobody does it that way for commercial operations.

Aloha 4y ago
Thank you for the thoughtful comment! I thought I was missing something in the way the story read, I'm glad to know I was.
konschubert 4y ago
How do you feel about the pilots getting blamed for relying on the software? > “Still, the mishaps point to the need for more vigilance by pilots in checking automated data.”
- t0mas88 4y ago
  In aviation incidents there isn't really a "blame", it's about finding all contributing factors and working together to eliminate those. The intention of investigations is never to assign blame, because that stands in the way of finding all factors and improving them together. So the conclusion "pilots should check the automated data" is a correct one. It's not the only cause, but it is one of the factors and thus it needs to be discussed. In commercial aviation there can never be a single point of failure that could lead to catastrophic outcome. The performance calculations are such a thing, so there must be validation steps. In many airlines for example if the performance calculations are done manually (or by an app based on manual input) they need to be done by both pilots independently and the results cross-checked. Otherwise a single typo could have a big impact.

_moof 4y ago

If it makes you feel any better, the people providing these calculations to the pilots are FAA-certified airline dispatchers. That means they've passed the same exam that airline pilots have to take, minus the flying. They may not have personal experience flying the airplane but they do have to know it inside and out. And if the dispatchers weren't running the numbers, the pilots would be running them using the same software and/or tables, so there really isn't much difference.