14 ms·

Bug identified after Alaska Airlines planes bump runway while taking off

229 points by leetrout 4y ago

leetrout 4y ago
Seems like an opportunity for some sort of automated testing by an isolated system.
- tantalor 4y ago
  > the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system That only raises further questions!
  - weard_beard 4y ago
    “The bug was missed because it only appeared when the user recited the magic incantation: ‘Yes.’” 1) Why would a condition ever ONLY occur then? That is a huge red flag that your whole system is fucked. 2) Why didn’t you test that!?
  - bmitc 4y ago
    Seems scary that something like the weight of a single airplane depends upon the amount of planes using some external system. So it seems that they not only discovered this particular bug by "luck" but that they've discovered a whole new vector of bugs.
    - yetihehe 4y ago
      Quantum entangled planes, what a time to be alive...
loufe 4y ago
Comment thread from when this came up a couple days ago: https://news.ycombinator.com/item?id=34847550 https://news.ycombinator.com/item?id=34847550
- gnabgib 4y ago
  241pts, 182 comments. This is an exact dupe (different URL, but exact same syndicated content). Comments are starting to look the same too
FL410 4y ago
Oof. Good thing they found the bug, and good thing the only issue was two tail strikes. Inaccurate weight and balance data is no joke, with the potential to cause much more serious issues.
bunnyswipecom 4y ago
I wonder how many off by one exists in a plane
- andrelaszlo 4y ago
  Don't worry, they probably mostly cancel each other out on average...
bb123 4y ago
So the bug told pilots the aircraft was lighter than it was. I wonder if that would have also factored into the fuel load calculations? It seems like it could have the potential for the crew to take off with less fuel than they need.
- FourHand451 4y ago
  Or perhaps the opposite. If the plane is saying it's lighter than expected, they may think too little fuel has been loaded.
- jcutrell 4y ago
  Possibly, but aircraft carry a significant amount of extra fuel anyway. (That's how you can have a jet holding for 2 hours at a fix unexpectedly because of weather or whatever.)
  - jacquesm 4y ago
    Author of a fuel estimation program for 747 freighters here... Yes, they take a significant amount of extra fuel, but that significant amount is very carefully calculated because carrying more than legally required adds up to a loss quickly, especially if your competitors are sharper at calculating these figures. The are ~100 parameters that go into these calculations, ranging from almanacs with tables of various trade winds and densities to engine types, consumption at various altitudes and power settings and so on. And then the required hold time, first alternate, second alternate and legal reserve required on landing. Getting that code certified still makes me break out in sweat, but good thing that this kind of thing is reviewed very strictly. That also makes me wonder how wise it was to have the software this article is about as a SaaS rather than something more integrated and less easy to mess up using an update. Maybe doing it as a SaaS is a kind of loophole that avoids certification? (not current on this stuff).
    - jcutrell 4y ago
      That's super interesting - I didn't realize that certification would be required for this. I would suspect that certification would break down at some point; does a plain old calculator need to be certified? Certainly not, but I'd imagine the FAA or whoever if asked would prefer pilots use calculators than do everything by hand. I'm curious about EFBs as well - I run W&B for my little bugsmasher in Foreflight. Has that gone through certification? Certainly someone from the FAA wouldn't have to review every release for that if so... So when does certification come into play versus it just being another tool that pilots are responsible for validating?
photochemsyn 4y ago
This is worth emphasizing: > "The bug was identified quickly in part because some flight crews noticed the weights didn’t seem right and asked for manual validation of the figures....The Alaska captain said that, as for many things in aviation, pilots routinely use an acronym when they do the pre-takeoff “sanity check”: TLAR, which means “That Looks About Right.”" Keeping well-trained humans in the loop in all partially or highly automated systems is important for this reason - and although corporate bean-counters in the overly finacialized US economy, whose prime motivation is to maximize shareholder profits by cutting labor costs, may not like to hear this, that mentality is the central reason behind the Boeing MAX debacle.
- jmeister 4y ago
  [flagged]
  - kahrl 4y ago
    "the market" doesn't exist in this situation. You have powerless, disenfranchised individuals with no choice.
  - Swenrekcah 4y ago
    And that answer is demonstrably wrong.
  - nervousvarun 4y ago
    And yeah...we know how that works in America. The land of the free, the home of the public losses/privatized profits.
  - pixl97 4y ago
    Right, lets get rid of the NTSB, the companies will investigate themselves.... And find themselves guilty of no wrongdoing.
  - aflukasz 4y ago
    "The market" is and always will be underinformed, including the knowledge about any potential ongoing "economic punishment".
londons_explore 4y ago
We currently use a complex system of trying to weigh everything that goes onto the plane - baggage, cargo, fuel, and estimate the weight and distribution of the people. Sure, estimates are good... But why not just weigh the whole plane just before takeoff? Simple strain gauges in the suspension struts would give you all the info you need. And then you can still use the old system of weighing all the stuff you put on the plane as a backup/sanity check if you want to.
- SketchySeaBeast 4y ago
  Where would these struts go? I assume in the landing gear suspension, and if so does that provide enough feedback to adequately understand the load and distribution? That's usually only 3 points of contact across the whole of the aircraft, isn't it?
  - londons_explore 4y ago
    3 points is exactly what you need to know the total weight, forward-back weight distribution, and left-right weight distribution. You don't get to know the top-bottom weight distribution, but that is rarely an issue in aviation.
    - SketchySeaBeast 4y ago
      Fair enough, I assumed there might need to be more consideration in where the weight in in terms of where along the wing it is, stuff like that, but I guess not.
  - KMag 4y ago
    You can still work out the total weight and the location of the center of mass (in 2D) from the strain on the three landing gear struts. It's a linear system of 3 equations and 3 unknowns. As another commenter has pointed out, there may be an issue with wind, as the whole point of the airplane is that it efficiently converts a relative air velocity into an apparent change in weight.
- 4y ago
londons_explore 4y ago
There is another bug here... If the plane was told how much weight was onboard, and it knows what throttle it was at... then it should know what acceleration it was expecting. If it accelerated slower than expected, then there must be something wrong! So abort the takeoff... Planes should have alerts for this kind of thing. If, 1 second after throttle is applied, the plane hasn't achieved the predicted speed, then the takeoff should be aborted - by this stage, you'll be well under the maximum abort speed, so you can stay safely on the ground and figure out whats wrong. Being overweight is just one possible cause - it might instead be that an engine isn't producing sufficient thrust, and that would probably doom the plane.
- guepe 4y ago
  No that simple ; what is wind speed and direction for instance, which changes acceleration? Air density will also affect, although that's easier to account for.
  - mannykannot 4y ago
    These things all have to be taken into account in the original calculation for how much power to use. It is impossible to determine the answer to that question without having the information necessary to calculate an expected acceleration profile for the takeoff. If you are using approximations in that calculation (such as the average weight of a passenger), then that same approximation can be used in calculating the acceleration profile, and if you are falling significantly short, then it shows that the calculation of the power setting needed for a safe takeoff is unsound in some way. That alone is sufficient to justify an abort.
  - spoonfeeder006 4y ago
    Can't the plane measure wind speed and direction and air density and all, and account for those?
    - dghlsakjg 4y ago
      Not especially well, especially close to the ground.
raverbashing 4y ago
> even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. Errrr what? How do you even get this type of bug?! These scare me more than just a calculation error Edit: on second thought this looks more like of some race condition on inputting the data, or maybe some query that fails and gets replaced by a 0
- napsterbr 4y ago
  Don't worry -- if you go to https://www.dynamicsource.se/ https://www.dynamicsource.se/ you'll see they are hiring a QA manager!
  - jacquesm 4y ago
    And here I was thinking you were kidding, but you're not... https://www.dynamicsource.se/recruitment/quality-assurance-manager/ https://www.dynamicsource.se/recruitment/quality-assurance-m...
gnfargbl 4y ago
> the bug was missed because it only presented when many aircraft at the same time were using the system. How the heck do you end up with a concurrency bug in a system where the calculations appear to be logically independent?
- speedgoose 4y ago
  A concurrency bug that got fixed in production 5 hours later. It must be very unusual code or some people are excellent at debugging concurrency problems.
  - 83 4y ago
    Article says that bug was introduced in an update that morning. Somewhere there was a developer familiar with the code and exactly what changed. Wouldn't be surprised if he smacked himself on the forehead and knew immediately what the issue was when he heard about the bug.
    - ldarby 4y ago
      No it doesn't say that. > That morning, a software bug in an update to the DynamicSource tool caused ... This says when the bug happened, not when the update was made.
  - andix 4y ago
    Or they just rolled back an update. I don’t think you can do much more in 5 hours.
    - hungryforcodes 4y ago
      Right. Even if the developer knew exactly what he did -- any sane provider would put the updated update through a full QC cycle, which would take a while I'd figure.
  - jacquesm 4y ago
    This sort of thing - in my uninformed opinion - should not be a service, and should be validated strictly as though it is part of the avionics software. After all it sends data that directly affects the power settings the pilots will use just as if those power settings were directly affected by the output of the program. That makes it an integral part of the plane to me.
jakub_g 4y ago
The juice about the bug: > On the morning of Jan. 26, as two Alaska Airlines flights from Seattle to Hawaii lifted off six minutes apart, the pilots each felt a slight bump and the flight attendants at the back of the cabin heard a scraping noise. > ...a software bug was sending bad takeoff weight data to its crews... > ...To determine the thrust and speed settings for takeoff, Alaska’s pilots and others use a performance calculation tool supplied by a Swedish company called DynamicSource. It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity. In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be ready to lift off. > [!!] the bug [..] only presented when many aircraft at the same time were using the system.
- adamhp 4y ago
  > the bug [..] only presented when many aircraft at the same time were using the system. This implies that the calculations are done externally to the plane? That seems... interesting.
  - yunohn 4y ago
    And somehow this algorithm is not isolated by plane!? That’s quite literally insane - why would it not be a pure function…
    - jaywalk 4y ago
      Having no knowledge about how the system works, I could imagine the bug being that one of the calculations timed out under load and wasn't caught, leading to the output not being added to the final calculation.
    - cratermoon 4y ago
      What does that even mean?
    - is_true 4y ago
bilekas 4y ago
> the bug was missed because it only presented when many aircraft at the same time were using the system. You've got to be kidding me.. > Subsequently, a test of the software under high demand was developed. This is wild that a basic kind of load test would not be done given the delicate nature of this system
geocrasher 4y ago
The article downplayed the seriousness of a tail strike, but they can be quite serious. A tail strike can damage not just the skin of the aircraft, but also the aft pressure bulkheads of some aircraft. A damaged pressure bulkhead could cause the cabin to suddenly lose pressurization at altitude, with potentially major consequences, especially on a flight across open ocean. https://en.wikipedia.org/wiki/Aft_pressure_bulkhead https://en.wikipedia.org/wiki/Aft_pressure_bulkhead
- greggyb 4y ago
  As a layperson, "they landed immediately and required inspection before being allowed to fly again" (not a direct quote, but it represents what the article said), seems pretty severe to me.
  - jcutrell 4y ago
    Ultimately this would be true for most anomalies. It's one of the reasons commercial air travel is as safe as it is. In a car, you hit a bump, you keep going. In a plane, something unexpected happens and that plane is grounded. The utility function is just too skewed to take even a slight risk in this regard.
    - mikepurvis 4y ago
      It's a matter of regulation and policy. I was on a regional bus recently in the rain (Ontario's GO Transit) and a small car next to us hydroplaned and bumped the side of the bus. The driver immediately - Pulled over onto the shoulder, - Passed around a pad to collect everyone's contact information, - Summoned a replacement bus, and - Called the provincial police to escort us along the shoulder to the replacement bus. In the moment, all of this felt a little over the top, and the total delay of about 20mins total was pretty frustrating given that I ended up missing a connection. But at the end of the day, it builds my trust in the safety of that system that the fitness of any particular piece of equipment is not up to the individual drivers to judge— part of paying for a ticket on the state-run transit system is expecting a higher level of safety than, say, what you might get on a Chinatown bus line (https://en.wikipedia.org/wiki/Chinatown_bus_lines#Accidents https://en.wikipedia.org/wiki/Chinatown_bus_lines#Accidents). And I think I was also glad that it hadn't happened to me in my personal vehicle, where I would have to be the one to assess fitness and then deal with the insurance and so-on in the aftermath.
nabbe23 4y ago
I do wonder if in all of the cost cutting some of these massive companies have focused on in previous years, we are going to see more and more dangerous bugs like this come out of the webwork as a result of refusal to pay their best engineers above market rate. Anecdotal, and probably just a result of software being more widespread in general, but it feels like incidents like these are becoming more frequent.
skwheel 4y ago
"The software code was permanently repaired about five hours later." I always try to permanently repair the software when I find bugs.
- gnu8 4y ago
  I wonder if that is aviation language instead of software language. Permanently repaired like replacing damaged aircraft skin so that no further action is required, as opposed to grinding off the jagged edges and patching the opening, but still requiring a real repair after flying back to a maintenance depot.
andix 4y ago
It’s really interesting, that airlines use a lot of externally computed information, just to copy it over to the planes systems manually. I’ve heard about issues with that before, because some airlines used unsuitable systems for calculations. It’s really interesting, that there are no standardized systems, that interface with the planes directly.
belter 4y ago
Sounds like this one and it's coded in Fortran. https://www.dynamicsource.se/service/ds-weight-balance/ https://www.dynamicsource.se/service/ds-weight-balance/ https://aviation.stackexchange.com/questions/32888/is-the-fortran-language-still-being-used-in-aviation#:~:text=They%20are%20written%20in%20Fortran,managed%20to%20make%20this%20possible https://aviation.stackexchange.com/questions/32888/is-the-fo.... Looks like lots of people were very very luck here. Plus finding a bug and correction to production in 5 hours? That is at the same time good and bad.. "Analysis of aircraft weight and balance related safety occurrences" - https://reports.nlr.nl/server/api/core/bitstreams/9dc15cb9-a90e-486b-b29f-4800d3eea645/content https://reports.nlr.nl/server/api/core/bitstreams/9dc15cb9-a...
AndyMcConachie 4y ago
22 minutes! They figured this out and issued a patch in 22 minutes! Is anyone else just absolutely astounded by how short that response time was? >That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes. The real issue question here is how often do they patch this software? Airline control software should not be optimized for feature velocity. It should be optimized for quality and change really slowly. Ideally, we would have as few updates to this software as possible. We're the pilots informed that the software had updated that morning? I worry that our increasing comfort with updating constantly broken software bleeds into systems like this that matter for people's lives. Apple has me updating almost weekly because they can't seem to write exploit free software, but my iPhone is not going to kill me. That attitude is not Ok for airplanes.
- jacquesm 4y ago
  I'm astounded that it was a problem to begin with. The only reason they managed to fix it so quickly is because they messed it up in the first place and shipped it to production without a stress test.
- medler 4y ago
  They didn’t issue a software patch in 22 minutes. It sounds like 22 minutes was the time it took the airline to figure out it was a software bug and tell pilots not to trust the software. The article says the software bug was fixed in 5 hours, but it’s not clear to me if that was a bug fix or a rollback.
- prmph 4y ago
  > but my iPhone is not going to kill me. That attitude is not Ok for airplanes. This is the key point. I imagine the stakeholders here do not think the weight software is that critical, when in fact it could be. This is a common issue with complex systems. Even though the software does not directly interface with the plane, its outputs are used to make key decisions on takeoff. Normally, there would be a large margin of error allowed, but in this case, due to the desire for max efficiency, the behavior of the aircraft is sensitive to the outputs. The solutions are to (1) go back to having a large error margin, which of course should be cross-checked for sanity by the pilots, or (2) consider that the software is safety-critical, and should meet the same quality standards as the in-built flight systems.
jack_riminton 4y ago
Have to say that's a well written article, logical flow, accurate, lack of hyperbole. In this age of content mills such things are refreshing
WaitWaitWha 4y ago
I read the title and for a minute I thought it must have been a type of beetle, maybe some wasp? Then, trying to figure out how the pumping along the runway is relevant. Maybe it fell out of the luggage compartment? And, where does Alaska Airlines fly that they could pick up suck insects? Finally I read the article. That was a bucket of cold water...
- kentrf 4y ago
  Same here. I also thought it was a new kind of bug somewhere in Alaska. Presented with the words "bug" and "airplane", "bug" is mostly associated with the insect kind, not the software kind.
game_the0ry 4y ago
>A pilot at American Airlines, which uses the same DynamicSource performance data tool, and who also spoke anonymously because he didn’t have authorization, explained that the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. Cost cutting can do more harm than good, and occasionally puts lives in danger - they need to teach that in MBA classes.
- anaganisk 4y ago
  Your advice is good, but the spirit is wrong. They do that to save fuel not just to cut costs but also to reduce consumption, which is net positive if you factor in environmental conerns. Some locations require it for noise abatement. Also, reducing the wear on the engine means more cycles and less maintenance/chance of engine damage. These parameters are well tested and are very well understood. Planes take off at mere 150kts, compared to what their cruise at, which is more than 250 knots. If you can take off at lesser thrust, it's not bad to do it. It's not like Airlines are making huge profits out of it too.
  - game_the0ry 4y ago
    Upvoted - I see what you are saying and I am mostly convinced, except: > It's not like Airlines are making huge profits out of it too. True, but airline margins are also very thin, so there is some incentive to "pinch pennies." And I do not put it past management to cost cut as much a possible for those sweet, sweet quarterly stock increases. Given the recent record of lately in terms of safety, timely service, cancellations, forced retirements for pilots, etc, the trends are not looking good for the airline consumer.
cld8483 4y ago
I must need more coffee, my first thought was that the bug must have been enormous for anybody on the plane to feel it being run over.
- dry_soup 4y ago
  I too was hoping that this was a bizarre crossover between aviation and entomology
twawaaay 4y ago
(I am sorry, I posted on a wrong article)
- jaywalk 4y ago
  How does Apple's anti-tracking for AirTags make them unusable?
harshreality 4y ago
> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. There are only two hard problems in computer science: - cache invalidation - naming things - off-by-one errors - concurrency/race conditions - concurrency/race conditions
- cratermoon 4y ago
  Here's a bit of detail. The software was supposed to calculate the speed based on the weight of the aircraft and determine if the available runways were adequate. The nature of the bug was that if it couldn't find a usable runway, it would lower the weight until it did fit the runway it could find. That's right, the software would say, "oh, the aircraft can't take off on the runway with this weight, let me just lower the weight so it can". Apparently the runway data was missing or wrong, but instead of saying, "no, you can't take off from there" the software said "here's a solution allowing you to take off, but at this lower weight". I'm not sure how/if that was caused by a concurrency bug, but that's what I know.
  - cptcobalt 4y ago
    what! that's incredible — do you have a link to the source (of this information), by chance?
    - cratermoon 4y ago
      It's something I learned from someone who works there. I can vouch for its reliability. Make what you will of that.
  - dghlsakjg 4y ago
    That makes sense. If you are flying out of Denver (high altitude) on a summer day, you need a TON of runway to get off the ground. There are some aircraft that simply can't take off fully loaded from high altitude with the runways that exist. The solution is to not take off fully loaded (lower the weight) So you either don't fully fuel up, or you remove cargo/passengers. It's very common, and by design. The software exists so that you know not to fully load the cargo bays if the conditions don't allow it (cargo makes airlines a lot of money).
kaapipo 4y ago
https://news.ycombinator.com/item?id=34847550 https://news.ycombinator.com/item?id=34847550
foolinaround 4y ago
maybe a secondary system that could raise alarms since in this case, the weight calculated must have been outside of the standard deviance of the weights recorded over time?
- hungryforcodes 4y ago
  Right like running a second instance and if both disagree there is an issue.
thimkerbell 4y ago
I wonder how common it is for a bug to be finally detected by the manufacturer months or years after users were impacted by it and thought it was fishy. I wonder what design practices extend that intervening time, and which ones shorten it.
lisper 4y ago
Private pilot here. This story buries the lede. There were tail strikes, which can be serious, but usually aren't, and in this case weren't. But the reason there were tail strikes is because software told the pilots to take off with throttle settings that were too low, and that could very easily have ended catastrophically. If takeoff power settings are too low and the pilots don't notice in time, you can end up in an unrecoverable state where the aircraft cannot take off but there is not enough runway left to abort. The passengers and crew on these flights got very, very lucky that it was "just" a tail strike.
- skeeter2020 4y ago
  >> The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins,” said the union’s Alaska unit chair, Will McQuillen, in a statement.
  - lisper 4y ago
    Sure, but you're missing the point. This particular bug produced an erroneous output that resulted in a tail strike. A different bug, with all the other circumstances unchanged, could have produced an erroneous output that resulted in a catastrophe. One should not shrug this off as "oh, it was just a tail strike, therefore it was no big deal." It was very nearly a very big deal.
    - Someone1234 4y ago
      If the data was too far off it is likely the pilots would have noticed it. This was a situation where it was off, but still "sane," so went unnoticed. The airline completely grounded their entire fleet while they investigated this issue, so I don't really understand why you're trying to act like nobody took this seriously. The article makes it quite clear this was handled as a serious problem and steps taken. The Alaska Airlines' pilots union is both qualified to have an opinion AND looking out primarily for their members (i.e., pilots), and they had this to say: > The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins,” said the union’s Alaska unit chair, Will McQuillen, in a statement. If the union was unhappy with the remediation steps taken, they'd say so. That is quite literally their primary function.
IG_Semmelweiss 4y ago
Id love to have a simple FAA rule that says the following: Every time there is a plane inspection, software update, at least 1 random person of the mechanical or software team for each party that directly touched either the plane or ground systems must be a passenger on the maiden operating flight for that plane/airport. This would apply to all parties involved. I'm sure the FAA can compensate and the FAA already has rules that are far costlier than this

229 points by leetrout 4y ago

leetrout 4y ago

Seems like an opportunity for some sort of automated testing by an isolated system.

tantalor 4y ago
> the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system That only raises further questions!
- weard_beard 4y ago
  “The bug was missed because it only appeared when the user recited the magic incantation: ‘Yes.’” 1) Why would a condition ever ONLY occur then? That is a huge red flag that your whole system is fucked. 2) Why didn’t you test that!?
- bmitc 4y ago
  Seems scary that something like the weight of a single airplane depends upon the amount of planes using some external system. So it seems that they not only discovered this particular bug by "luck" but that they've discovered a whole new vector of bugs.
  - yetihehe 4y ago
    Quantum entangled planes, what a time to be alive...

loufe 4y ago

Comment thread from when this came up a couple days ago: https://news.ycombinator.com/item?id=34847550 https://news.ycombinator.com/item?id=34847550

gnabgib 4y ago
241pts, 182 comments. This is an exact dupe (different URL, but exact same syndicated content). Comments are starting to look the same too

FL410 4y ago

Oof. Good thing they found the bug, and good thing the only issue was two tail strikes. Inaccurate weight and balance data is no joke, with the potential to cause much more serious issues.

bunnyswipecom 4y ago

I wonder how many off by one exists in a plane

andrelaszlo 4y ago
Don't worry, they probably mostly cancel each other out on average...

bb123 4y ago

So the bug told pilots the aircraft was lighter than it was. I wonder if that would have also factored into the fuel load calculations? It seems like it could have the potential for the crew to take off with less fuel than they need.

FourHand451 4y ago
Or perhaps the opposite. If the plane is saying it's lighter than expected, they may think too little fuel has been loaded.
jcutrell 4y ago
Possibly, but aircraft carry a significant amount of extra fuel anyway. (That's how you can have a jet holding for 2 hours at a fix unexpectedly because of weather or whatever.)
- jacquesm 4y ago
  Author of a fuel estimation program for 747 freighters here... Yes, they take a significant amount of extra fuel, but that significant amount is very carefully calculated because carrying more than legally required adds up to a loss quickly, especially if your competitors are sharper at calculating these figures. The are ~100 parameters that go into these calculations, ranging from almanacs with tables of various trade winds and densities to engine types, consumption at various altitudes and power settings and so on. And then the required hold time, first alternate, second alternate and legal reserve required on landing. Getting that code certified still makes me break out in sweat, but good thing that this kind of thing is reviewed very strictly. That also makes me wonder how wise it was to have the software this article is about as a SaaS rather than something more integrated and less easy to mess up using an update. Maybe doing it as a SaaS is a kind of loophole that avoids certification? (not current on this stuff).
  - jcutrell 4y ago
    That's super interesting - I didn't realize that certification would be required for this. I would suspect that certification would break down at some point; does a plain old calculator need to be certified? Certainly not, but I'd imagine the FAA or whoever if asked would prefer pilots use calculators than do everything by hand. I'm curious about EFBs as well - I run W&B for my little bugsmasher in Foreflight. Has that gone through certification? Certainly someone from the FAA wouldn't have to review every release for that if so... So when does certification come into play versus it just being another tool that pilots are responsible for validating?

photochemsyn 4y ago

This is worth emphasizing: > "The bug was identified quickly in part because some flight crews noticed the weights didn’t seem right and asked for manual validation of the figures....The Alaska captain said that, as for many things in aviation, pilots routinely use an acronym when they do the pre-takeoff “sanity check”: TLAR, which means “That Looks About Right.”" Keeping well-trained humans in the loop in all partially or highly automated systems is important for this reason - and although corporate bean-counters in the overly finacialized US economy, whose prime motivation is to maximize shareholder profits by cutting labor costs, may not like to hear this, that mentality is the central reason behind the Boeing MAX debacle.

jmeister 4y ago
[flagged]
- kahrl 4y ago
  "the market" doesn't exist in this situation. You have powerless, disenfranchised individuals with no choice.
- Swenrekcah 4y ago
  And that answer is demonstrably wrong.
- nervousvarun 4y ago
  And yeah...we know how that works in America. The land of the free, the home of the public losses/privatized profits.
- pixl97 4y ago
  Right, lets get rid of the NTSB, the companies will investigate themselves.... And find themselves guilty of no wrongdoing.
- aflukasz 4y ago
  "The market" is and always will be underinformed, including the knowledge about any potential ongoing "economic punishment".

londons_explore 4y ago

We currently use a complex system of trying to weigh everything that goes onto the plane - baggage, cargo, fuel, and estimate the weight and distribution of the people. Sure, estimates are good... But why not just weigh the whole plane just before takeoff? Simple strain gauges in the suspension struts would give you all the info you need. And then you can still use the old system of weighing all the stuff you put on the plane as a backup/sanity check if you want to.

SketchySeaBeast 4y ago
Where would these struts go? I assume in the landing gear suspension, and if so does that provide enough feedback to adequately understand the load and distribution? That's usually only 3 points of contact across the whole of the aircraft, isn't it?
- londons_explore 4y ago
  3 points is exactly what you need to know the total weight, forward-back weight distribution, and left-right weight distribution. You don't get to know the top-bottom weight distribution, but that is rarely an issue in aviation.
  - SketchySeaBeast 4y ago
    Fair enough, I assumed there might need to be more consideration in where the weight in in terms of where along the wing it is, stuff like that, but I guess not.
- KMag 4y ago
  You can still work out the total weight and the location of the center of mass (in 2D) from the strain on the three landing gear struts. It's a linear system of 3 equations and 3 unknowns. As another commenter has pointed out, there may be an issue with wind, as the whole point of the airplane is that it efficiently converts a relative air velocity into an apparent change in weight.
4y ago

londons_explore 4y ago

There is another bug here... If the plane was told how much weight was onboard, and it knows what throttle it was at... then it should know what acceleration it was expecting. If it accelerated slower than expected, then there must be something wrong! So abort the takeoff... Planes should have alerts for this kind of thing. If, 1 second after throttle is applied, the plane hasn't achieved the predicted speed, then the takeoff should be aborted - by this stage, you'll be well under the maximum abort speed, so you can stay safely on the ground and figure out whats wrong. Being overweight is just one possible cause - it might instead be that an engine isn't producing sufficient thrust, and that would probably doom the plane.

guepe 4y ago
No that simple ; what is wind speed and direction for instance, which changes acceleration? Air density will also affect, although that's easier to account for.
- mannykannot 4y ago
  These things all have to be taken into account in the original calculation for how much power to use. It is impossible to determine the answer to that question without having the information necessary to calculate an expected acceleration profile for the takeoff. If you are using approximations in that calculation (such as the average weight of a passenger), then that same approximation can be used in calculating the acceleration profile, and if you are falling significantly short, then it shows that the calculation of the power setting needed for a safe takeoff is unsound in some way. That alone is sufficient to justify an abort.
- spoonfeeder006 4y ago
  Can't the plane measure wind speed and direction and air density and all, and account for those?
  - dghlsakjg 4y ago
    Not especially well, especially close to the ground.

raverbashing 4y ago

> even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. Errrr what? How do you even get this type of bug?! These scare me more than just a calculation error Edit: on second thought this looks more like of some race condition on inputting the data, or maybe some query that fails and gets replaced by a 0

napsterbr 4y ago
Don't worry -- if you go to https://www.dynamicsource.se/ https://www.dynamicsource.se/ you'll see they are hiring a QA manager!
- jacquesm 4y ago
  And here I was thinking you were kidding, but you're not... https://www.dynamicsource.se/recruitment/quality-assurance-manager/ https://www.dynamicsource.se/recruitment/quality-assurance-m...

gnfargbl 4y ago

> the bug was missed because it only presented when many aircraft at the same time were using the system. How the heck do you end up with a concurrency bug in a system where the calculations appear to be logically independent?

speedgoose 4y ago
A concurrency bug that got fixed in production 5 hours later. It must be very unusual code or some people are excellent at debugging concurrency problems.
- 83 4y ago
  Article says that bug was introduced in an update that morning. Somewhere there was a developer familiar with the code and exactly what changed. Wouldn't be surprised if he smacked himself on the forehead and knew immediately what the issue was when he heard about the bug.
  - ldarby 4y ago
    No it doesn't say that. > That morning, a software bug in an update to the DynamicSource tool caused ... This says when the bug happened, not when the update was made.
- andix 4y ago
  Or they just rolled back an update. I don’t think you can do much more in 5 hours.
  - hungryforcodes 4y ago
    Right. Even if the developer knew exactly what he did -- any sane provider would put the updated update through a full QC cycle, which would take a while I'd figure.
- jacquesm 4y ago
  This sort of thing - in my uninformed opinion - should not be a service, and should be validated strictly as though it is part of the avionics software. After all it sends data that directly affects the power settings the pilots will use just as if those power settings were directly affected by the output of the program. That makes it an integral part of the plane to me.

jakub_g 4y ago

The juice about the bug: > On the morning of Jan. 26, as two Alaska Airlines flights from Seattle to Hawaii lifted off six minutes apart, the pilots each felt a slight bump and the flight attendants at the back of the cabin heard a scraping noise. > ...a software bug was sending bad takeoff weight data to its crews... > ...To determine the thrust and speed settings for takeoff, Alaska’s pilots and others use a performance calculation tool supplied by a Swedish company called DynamicSource. It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity. In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be ready to lift off. > [!!] the bug [..] only presented when many aircraft at the same time were using the system.

adamhp 4y ago
> the bug [..] only presented when many aircraft at the same time were using the system. This implies that the calculations are done externally to the plane? That seems... interesting.
- yunohn 4y ago
  And somehow this algorithm is not isolated by plane!? That’s quite literally insane - why would it not be a pure function…
  - jaywalk 4y ago
    Having no knowledge about how the system works, I could imagine the bug being that one of the calculations timed out under load and wasn't caught, leading to the output not being added to the final calculation.
  - cratermoon 4y ago
    What does that even mean?
  - is_true 4y ago

bilekas 4y ago

> the bug was missed because it only presented when many aircraft at the same time were using the system. You've got to be kidding me.. > Subsequently, a test of the software under high demand was developed. This is wild that a basic kind of load test would not be done given the delicate nature of this system

geocrasher 4y ago

The article downplayed the seriousness of a tail strike, but they can be quite serious. A tail strike can damage not just the skin of the aircraft, but also the aft pressure bulkheads of some aircraft. A damaged pressure bulkhead could cause the cabin to suddenly lose pressurization at altitude, with potentially major consequences, especially on a flight across open ocean. https://en.wikipedia.org/wiki/Aft_pressure_bulkhead https://en.wikipedia.org/wiki/Aft_pressure_bulkhead

greggyb 4y ago
As a layperson, "they landed immediately and required inspection before being allowed to fly again" (not a direct quote, but it represents what the article said), seems pretty severe to me.
- jcutrell 4y ago
  Ultimately this would be true for most anomalies. It's one of the reasons commercial air travel is as safe as it is. In a car, you hit a bump, you keep going. In a plane, something unexpected happens and that plane is grounded. The utility function is just too skewed to take even a slight risk in this regard.
  - mikepurvis 4y ago
    It's a matter of regulation and policy. I was on a regional bus recently in the rain (Ontario's GO Transit) and a small car next to us hydroplaned and bumped the side of the bus. The driver immediately - Pulled over onto the shoulder, - Passed around a pad to collect everyone's contact information, - Summoned a replacement bus, and - Called the provincial police to escort us along the shoulder to the replacement bus. In the moment, all of this felt a little over the top, and the total delay of about 20mins total was pretty frustrating given that I ended up missing a connection. But at the end of the day, it builds my trust in the safety of that system that the fitness of any particular piece of equipment is not up to the individual drivers to judge— part of paying for a ticket on the state-run transit system is expecting a higher level of safety than, say, what you might get on a Chinatown bus line (https://en.wikipedia.org/wiki/Chinatown_bus_lines#Accidents https://en.wikipedia.org/wiki/Chinatown_bus_lines#Accidents). And I think I was also glad that it hadn't happened to me in my personal vehicle, where I would have to be the one to assess fitness and then deal with the insurance and so-on in the aftermath.

nabbe23 4y ago

I do wonder if in all of the cost cutting some of these massive companies have focused on in previous years, we are going to see more and more dangerous bugs like this come out of the webwork as a result of refusal to pay their best engineers above market rate. Anecdotal, and probably just a result of software being more widespread in general, but it feels like incidents like these are becoming more frequent.

skwheel 4y ago

"The software code was permanently repaired about five hours later." I always try to permanently repair the software when I find bugs.

gnu8 4y ago
I wonder if that is aviation language instead of software language. Permanently repaired like replacing damaged aircraft skin so that no further action is required, as opposed to grinding off the jagged edges and patching the opening, but still requiring a real repair after flying back to a maintenance depot.

andix 4y ago

It’s really interesting, that airlines use a lot of externally computed information, just to copy it over to the planes systems manually. I’ve heard about issues with that before, because some airlines used unsuitable systems for calculations. It’s really interesting, that there are no standardized systems, that interface with the planes directly.

belter 4y ago

Sounds like this one and it's coded in Fortran. https://www.dynamicsource.se/service/ds-weight-balance/ https://www.dynamicsource.se/service/ds-weight-balance/ https://aviation.stackexchange.com/questions/32888/is-the-fortran-language-still-being-used-in-aviation#:~:text=They%20are%20written%20in%20Fortran,managed%20to%20make%20this%20possible https://aviation.stackexchange.com/questions/32888/is-the-fo.... Looks like lots of people were very very luck here. Plus finding a bug and correction to production in 5 hours? That is at the same time good and bad.. "Analysis of aircraft weight and balance related safety occurrences" - https://reports.nlr.nl/server/api/core/bitstreams/9dc15cb9-a90e-486b-b29f-4800d3eea645/content https://reports.nlr.nl/server/api/core/bitstreams/9dc15cb9-a...

AndyMcConachie 4y ago

22 minutes! They figured this out and issued a patch in 22 minutes! Is anyone else just absolutely astounded by how short that response time was? >That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes. The real issue question here is how often do they patch this software? Airline control software should not be optimized for feature velocity. It should be optimized for quality and change really slowly. Ideally, we would have as few updates to this software as possible. We're the pilots informed that the software had updated that morning? I worry that our increasing comfort with updating constantly broken software bleeds into systems like this that matter for people's lives. Apple has me updating almost weekly because they can't seem to write exploit free software, but my iPhone is not going to kill me. That attitude is not Ok for airplanes.

jacquesm 4y ago
I'm astounded that it was a problem to begin with. The only reason they managed to fix it so quickly is because they messed it up in the first place and shipped it to production without a stress test.
medler 4y ago
They didn’t issue a software patch in 22 minutes. It sounds like 22 minutes was the time it took the airline to figure out it was a software bug and tell pilots not to trust the software. The article says the software bug was fixed in 5 hours, but it’s not clear to me if that was a bug fix or a rollback.
prmph 4y ago
> but my iPhone is not going to kill me. That attitude is not Ok for airplanes. This is the key point. I imagine the stakeholders here do not think the weight software is that critical, when in fact it could be. This is a common issue with complex systems. Even though the software does not directly interface with the plane, its outputs are used to make key decisions on takeoff. Normally, there would be a large margin of error allowed, but in this case, due to the desire for max efficiency, the behavior of the aircraft is sensitive to the outputs. The solutions are to (1) go back to having a large error margin, which of course should be cross-checked for sanity by the pilots, or (2) consider that the software is safety-critical, and should meet the same quality standards as the in-built flight systems.

jack_riminton 4y ago

Have to say that's a well written article, logical flow, accurate, lack of hyperbole. In this age of content mills such things are refreshing

WaitWaitWha 4y ago

I read the title and for a minute I thought it must have been a type of beetle, maybe some wasp? Then, trying to figure out how the pumping along the runway is relevant. Maybe it fell out of the luggage compartment? And, where does Alaska Airlines fly that they could pick up suck insects? Finally I read the article. That was a bucket of cold water...

kentrf 4y ago
Same here. I also thought it was a new kind of bug somewhere in Alaska. Presented with the words "bug" and "airplane", "bug" is mostly associated with the insect kind, not the software kind.

game_the0ry 4y ago

>A pilot at American Airlines, which uses the same DynamicSource performance data tool, and who also spoke anonymously because he didn’t have authorization, explained that the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. >“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance. Cost cutting can do more harm than good, and occasionally puts lives in danger - they need to teach that in MBA classes.

anaganisk 4y ago
Your advice is good, but the spirit is wrong. They do that to save fuel not just to cut costs but also to reduce consumption, which is net positive if you factor in environmental conerns. Some locations require it for noise abatement. Also, reducing the wear on the engine means more cycles and less maintenance/chance of engine damage. These parameters are well tested and are very well understood. Planes take off at mere 150kts, compared to what their cruise at, which is more than 250 knots. If you can take off at lesser thrust, it's not bad to do it. It's not like Airlines are making huge profits out of it too.
- game_the0ry 4y ago
  Upvoted - I see what you are saying and I am mostly convinced, except: > It's not like Airlines are making huge profits out of it too. True, but airline margins are also very thin, so there is some incentive to "pinch pennies." And I do not put it past management to cost cut as much a possible for those sweet, sweet quarterly stock increases. Given the recent record of lately in terms of safety, timely service, cancellations, forced retirements for pilots, etc, the trends are not looking good for the airline consumer.

cld8483 4y ago

I must need more coffee, my first thought was that the bug must have been enormous for anybody on the plane to feel it being run over.

dry_soup 4y ago
I too was hoping that this was a bizarre crossover between aviation and entomology

twawaaay 4y ago

(I am sorry, I posted on a wrong article)

jaywalk 4y ago
How does Apple's anti-tracking for AirTags make them unusable?

harshreality 4y ago

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. There are only two hard problems in computer science: - cache invalidation - naming things - off-by-one errors - concurrency/race conditions - concurrency/race conditions

cratermoon 4y ago
Here's a bit of detail. The software was supposed to calculate the speed based on the weight of the aircraft and determine if the available runways were adequate. The nature of the bug was that if it couldn't find a usable runway, it would lower the weight until it did fit the runway it could find. That's right, the software would say, "oh, the aircraft can't take off on the runway with this weight, let me just lower the weight so it can". Apparently the runway data was missing or wrong, but instead of saying, "no, you can't take off from there" the software said "here's a solution allowing you to take off, but at this lower weight". I'm not sure how/if that was caused by a concurrency bug, but that's what I know.
- cptcobalt 4y ago
  what! that's incredible — do you have a link to the source (of this information), by chance?
  - cratermoon 4y ago
    It's something I learned from someone who works there. I can vouch for its reliability. Make what you will of that.
- dghlsakjg 4y ago
  That makes sense. If you are flying out of Denver (high altitude) on a summer day, you need a TON of runway to get off the ground. There are some aircraft that simply can't take off fully loaded from high altitude with the runways that exist. The solution is to not take off fully loaded (lower the weight) So you either don't fully fuel up, or you remove cargo/passengers. It's very common, and by design. The software exists so that you know not to fully load the cargo bays if the conditions don't allow it (cargo makes airlines a lot of money).

kaapipo 4y ago

https://news.ycombinator.com/item?id=34847550 https://news.ycombinator.com/item?id=34847550

foolinaround 4y ago

maybe a secondary system that could raise alarms since in this case, the weight calculated must have been outside of the standard deviance of the weights recorded over time?

hungryforcodes 4y ago
Right like running a second instance and if both disagree there is an issue.

thimkerbell 4y ago

I wonder how common it is for a bug to be finally detected by the manufacturer months or years after users were impacted by it and thought it was fishy. I wonder what design practices extend that intervening time, and which ones shorten it.

lisper 4y ago

Private pilot here. This story buries the lede. There were tail strikes, which can be serious, but usually aren't, and in this case weren't. But the reason there were tail strikes is because software told the pilots to take off with throttle settings that were too low, and that could very easily have ended catastrophically. If takeoff power settings are too low and the pilots don't notice in time, you can end up in an unrecoverable state where the aircraft cannot take off but there is not enough runway left to abort. The passengers and crew on these flights got very, very lucky that it was "just" a tail strike.

skeeter2020 4y ago
>> The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins,” said the union’s Alaska unit chair, Will McQuillen, in a statement.
- lisper 4y ago
  Sure, but you're missing the point. This particular bug produced an erroneous output that resulted in a tail strike. A different bug, with all the other circumstances unchanged, could have produced an erroneous output that resulted in a catastrophe. One should not shrug this off as "oh, it was just a tail strike, therefore it was no big deal." It was very nearly a very big deal.
  - Someone1234 4y ago
    If the data was too far off it is likely the pilots would have noticed it. This was a situation where it was off, but still "sane," so went unnoticed. The airline completely grounded their entire fleet while they investigated this issue, so I don't really understand why you're trying to act like nobody took this seriously. The article makes it quite clear this was handled as a serious problem and steps taken. The Alaska Airlines' pilots union is both qualified to have an opinion AND looking out primarily for their members (i.e., pilots), and they had this to say: > The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins,” said the union’s Alaska unit chair, Will McQuillen, in a statement. If the union was unhappy with the remediation steps taken, they'd say so. That is quite literally their primary function.

IG_Semmelweiss 4y ago

Id love to have a simple FAA rule that says the following: Every time there is a plane inspection, software update, at least 1 random person of the mechanical or software team for each party that directly touched either the plane or ground systems must be a passenger on the maiden operating flight for that plane/airport. This would apply to all parties involved. I'm sure the FAA can compensate and the FAA already has rules that are far costlier than this