5 ms·

Curious about the actual bug: > the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented

by Tempest1981 4y ago

Curious about the actual bug: > the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system > the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings. Multithreading/contention issue? But how would that alter the weights? > The software code was permanently repaired about five hours later That's surprisingly fast, isn't it?

sidlls 4y ago
Multithreading/contention issue? But how would that alter the weights? Off the top of my head: failure to sum values correctly, misreporting the weight to the target plane (reading values from another plane’s weight data).
hansvm 4y ago
Could be something as simple as the first concurrency 101 example: atomic addition. The += operator behaves poorly on concurrent access.
- Jtsummers 4y ago
  We had that error in an application at work. Essentially data was coming in on multiple threads, but there was a shared value between them that was being incremented and decremented. Without a lock and without using atomics, the tally could be incorrect. But under low load this didn't appear because the odds of interrupting +=/-=/++/-- was low. Under high load, however, the odds of a thread being interrupted mid-operation increased and so it appeared in production. At least in our case it was just an annoyance (no critical decisions were based on the value, it was merely informative). If there are multiple data streams feeding in weight data per aircraft, under a low aircraft load (the scenarios they apparently tested) the odds of interruption during an operation like += can be low enough to not see the issue. Under high load, though, the odds of interruption and incorrect tallying increases substantially.
  - KennyBlanken 4y ago
    In what scenario would you anticipate data for the calculation of rotation speed and engine thrust for one particular plane to be based on a value that is not only shared and writable, but changing each time a plane's performance data is calculated? "multiple data streams feeding in weight data per aircraft"? What does that word salad even mean? Even if what any of what you said was true: these developers are writing critical safety software. If they can't manage to write code without generating race conditions and testing their system for such conditions, they are grossly unqualified to be writing this sort of software. As others have said: this is a huge fuck-up.
    - Jtsummers 4y ago
      It is a huge fuckup, I didn't say otherwise. And that's not word salad, though maybe not as clear as I could have made it. But to clarify for this hypothetical (if this is the issue): Each luggage weigh station submits data to the application or database which then has to tally the weight. Those are multiple data streams, not a hard concept. An airport has many luggage weigh stations all potentially submitting data at the same time. If those are being processed concurrently and the database or however the data is stored is not properly locked then the tally can be incorrect for the same reason non-atomic increments/decrements can become incorrect in a multithreaded application. But just like the problem with non-atomic (unlocked) increments/decrements the problem may not manifest without high enough load on the system. Low-load testing (what the article seems to describe) means that updates can happen fast enough (server or application is under loaded, less likely to interrupt any update) or submitted slow enough that the issue never appears. So you need to test with a higher load to detect the problem (if you're going to be able detect it in testing at all).
    - richwater 4y ago
      I fucking cracked up at: "multiple data streams feeding in weight data per aircraft" it's literally some integers (floats at the worst?).
      - Dylan16807 4y ago
johntb86 4y ago
> Multithreading/contention issue? But how would that alter the weights? Maybe they stored something in a global variable, and with two planes querying at the same time values for the wrong plane could be retrieved.
renewiltord 4y ago
Concurrency is one source, yes, but integration is another possible source. If the vendor software interfaced with Alaska's software then the interfacing layer could imperfectly handle pagination, assume ordering where there is none, and other such data source problems. A trivial error could be "list all active planes", "list current luggage weight for all these plane IDs", and have the latter give responses in a different order than the input list. Or perhaps "fuel load" returns in a different order. Or something. Concurrency seems like the obvious one, but it's not the only source I can imagine here. And ultimately, the max is under 300 planes. You can do that serially unless the integration layer is slow.
mmcconnell1618 4y ago
I'm surprised there wasn't any human or general guardrail around the calculation that would have flagged such a significant weight difference. No one looked at the passenger count and the weight calculation and thought "whoa, that doesn't look right?"
- hodgesrm 4y ago
  > The bug was identified quickly in part because some flight crews noticed the weights didn’t seem right and asked for manual validation of the figures. Actually there were such guardrails, so the system had multiple manual checks built in.
ww520 4y ago
The fix was probably just a rollback to the previous version.

sidlls 4y ago

Multithreading/contention issue? But how would that alter the weights? Off the top of my head: failure to sum values correctly, misreporting the weight to the target plane (reading values from another plane’s weight data).

hansvm 4y ago

Could be something as simple as the first concurrency 101 example: atomic addition. The += operator behaves poorly on concurrent access.

Jtsummers 4y ago
We had that error in an application at work. Essentially data was coming in on multiple threads, but there was a shared value between them that was being incremented and decremented. Without a lock and without using atomics, the tally could be incorrect. But under low load this didn't appear because the odds of interrupting +=/-=/++/-- was low. Under high load, however, the odds of a thread being interrupted mid-operation increased and so it appeared in production. At least in our case it was just an annoyance (no critical decisions were based on the value, it was merely informative). If there are multiple data streams feeding in weight data per aircraft, under a low aircraft load (the scenarios they apparently tested) the odds of interruption during an operation like += can be low enough to not see the issue. Under high load, though, the odds of interruption and incorrect tallying increases substantially.
- KennyBlanken 4y ago
  In what scenario would you anticipate data for the calculation of rotation speed and engine thrust for one particular plane to be based on a value that is not only shared and writable, but changing each time a plane's performance data is calculated? "multiple data streams feeding in weight data per aircraft"? What does that word salad even mean? Even if what any of what you said was true: these developers are writing critical safety software. If they can't manage to write code without generating race conditions and testing their system for such conditions, they are grossly unqualified to be writing this sort of software. As others have said: this is a huge fuck-up.
  - Jtsummers 4y ago
    It is a huge fuckup, I didn't say otherwise. And that's not word salad, though maybe not as clear as I could have made it. But to clarify for this hypothetical (if this is the issue): Each luggage weigh station submits data to the application or database which then has to tally the weight. Those are multiple data streams, not a hard concept. An airport has many luggage weigh stations all potentially submitting data at the same time. If those are being processed concurrently and the database or however the data is stored is not properly locked then the tally can be incorrect for the same reason non-atomic increments/decrements can become incorrect in a multithreaded application. But just like the problem with non-atomic (unlocked) increments/decrements the problem may not manifest without high enough load on the system. Low-load testing (what the article seems to describe) means that updates can happen fast enough (server or application is under loaded, less likely to interrupt any update) or submitted slow enough that the issue never appears. So you need to test with a higher load to detect the problem (if you're going to be able detect it in testing at all).
  - richwater 4y ago
    I fucking cracked up at: "multiple data streams feeding in weight data per aircraft" it's literally some integers (floats at the worst?).
    - Dylan16807 4y ago

johntb86 4y ago

> Multithreading/contention issue? But how would that alter the weights? Maybe they stored something in a global variable, and with two planes querying at the same time values for the wrong plane could be retrieved.

renewiltord 4y ago

Concurrency is one source, yes, but integration is another possible source. If the vendor software interfaced with Alaska's software then the interfacing layer could imperfectly handle pagination, assume ordering where there is none, and other such data source problems. A trivial error could be "list all active planes", "list current luggage weight for all these plane IDs", and have the latter give responses in a different order than the input list. Or perhaps "fuel load" returns in a different order. Or something. Concurrency seems like the obvious one, but it's not the only source I can imagine here. And ultimately, the max is under 300 planes. You can do that serially unless the integration layer is slow.

mmcconnell1618 4y ago

I'm surprised there wasn't any human or general guardrail around the calculation that would have flagged such a significant weight difference. No one looked at the passenger count and the weight calculation and thought "whoa, that doesn't look right?"

hodgesrm 4y ago
> The bug was identified quickly in part because some flight crews noticed the weights didn’t seem right and asked for manual validation of the figures. Actually there were such guardrails, so the system had multiple manual checks built in.

ww520 4y ago

The fix was probably just a rollback to the previous version.