7 ms·

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented w

> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system. That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.

cratermoon 4y ago
Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs
- djanogo 4y ago
  This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them. "the bug was missed because it only presented when many aircraft at the same time were using the system"
  - hn_throwaway_99 4y ago
    There is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/ https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare.... Obviously sessions should be independent and not sharing data, but that's why it was a bug.
  - usea 4y ago
    I believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.
- howinteresting 4y ago
  Should have used Rust.
  - klyrs 4y ago
    A proper transactional database would suffice. You could do this safely and performantly on '00s hardware in PHP3 for crying out loud, serving thousands of planes per second.
    - AdrianB1 4y ago
      Bad coding happen even in transactional databases, see my example in the other comment.
      - klyrs 4y ago
        Note that I didn't say you couldn't write bad PHP3.
  - howinteresting 4y ago
    (This was a joke!)
- speed_spread 4y ago
  I refuse to believe that summing up weights for each of 300 planes is so hard that it can't be made correct by a team of moderately competent developers operating under sane management. There might be some complexity, but it's 2023 and we have enough knowledge and tools to solve that kind of stuff routinely and reliably. Somebody fucked up big time and should be fired never to work in software again.
  - lcuff 4y ago
    I agree that it's hard to see from the outside how calculating weight figures for an airplane could be software load-dependent. But this all smacks of the point that Brooks makes in 'The Mythical Man Month' all those years ago. Two guys in a garage can make a program that does 'x', but to make a programming system product that does 'x' is more than an order of magnitude more work. The complexities introduced don't relate to the 'x', but the intricacies of large systems. Deciding that 'someone has fucked up and should never work again'? Sorry, competent and conscientious people regularly make such mistakes. Which is why we need tests, software review, engineering processes et al to catch those mistakes.
  - justinclift 4y ago
    Boeing itself (unsure about this company though) was widely reported to have offshored their senior engineers a few years ago. eg: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... (paywalled) https://archive.is/vdg9S https://archive.is/vdg9S (paywall workaround) Note that there were several articles about it around the time, the above one is just the first reasonable seeming one from a quick search.
  - AdrianB1 4y ago
    I can give you a simple example on how this can happen, from a real production system in a different regulated environment: all the SQL SELECT statements are WITH (NOLOCK) on MS SQL server, resulting in non-committed (dirty) reads. Under very light load it works fine, on higher loads it reads some dirty data. If you calculate the weight of the plane by adding the weight of the checked in luggage plus passengers read from a shared database, some differences will appear and they will increase with the load as more transactions will be in flight and possibly read or ignored.
basch 4y ago
Maybe a timeout. Some part of a value returned 0 if it didn’t return in time.
- coin 4y ago
  Sounds like poor error handing. 0 is valid value is not a substitute for no data or error.
- sudhirj 4y ago
  Not according to the article, the values returned were within the realm of possibility, but wrong. Giving out zeroes might have actually been fine, it’s easy enough to realise there’s a problem if you see your plane weighs 0 pounds.
  - basch 4y ago
    Hence “some part of a value.” Such as weight=plane+fuel+people+luggage. If one of the variables became 0 you’d still have the other 3. It was a low enough value to have made multiple pilots question its value that day, like fuel or passenger load was missing completely.
  - advisedwang 4y ago
    I mean it might be just one part. E.g. luggage gets counted as zero but passenger weight, fuel weight and aircraft weight correct - that might be pluausible but low.
- dehrmann 4y ago
  Classic pokemon exception handling.
hn_throwaway_99 4y ago
> but being wrong under load sounds like a horrible internal architecture problem. Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to GitHub: https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/ https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare...
- lupire 4y ago
  "huge" is not the same as "horrible". One racy boi can take out an airplane.
  - hn_throwaway_99 4y ago
    But calling something an "internal architectural problem" implies things are fundamentally designed wrong, and the code needs a rewrite. My point was the architecture can definitely be fine, but you can still have bugs (even catastrophic ones) because someone left out a `synchronized` keyword somewhere.
    - thrashh 4y ago
      It could still be For example, one way to use SQL is to escape your parameters and then concat them straight into your SQL string. However, if someone leaves out an ‘escape’ somewhere, you have a major security incident The other way is to pass parameters separately as data to your SQL driver. It completely negates the problem If you’ve chosen the first way in your project already, you’ve committed to a major internal architectural problem. In the same vein, maybe if your code requires sprinkling ‘synchronized’ everywhere, you did it wrong But that said, we don’t know what actually is the problem since it’s just a marketing statement saying that high load was the cause and that doesn’t tell us anything
- Ozzie_osman 4y ago
  True, but software for domains where people can die is very different than Cloudflare. The scale is much higher in the latter, and it needs to be less "bulletproof". If you're writing software for giant metal flying cylinder carrying hundreds of people, we can't just brush mistakes off the same way we might with web-based software (and yes, privacy is important, but it's not life or death).
exabrial 4y ago
I was just about to post the _Exact_ same thing. Each execution of the program should be completely independent from other calculations. Something is _Horribly_ wrong with their architecture.
- lostlogin 4y ago
  I wonder how it works. Obviously you are correct, but the number of moving parts must be huge. The number of passengers, amount of cargo and the fuel load must be changing the whole time.
- tbrownaw 4y ago
  ... Call to the baggage-tracking service returned 429 and an empty body, and that was treated as "there are no bags" because some dweeb [dev, but I'm leaving in what autocorrect said] had been reading to much design guidance from Netflix/Facebook/etc that assumes missing features on a fraction of page loads are NBD?
ericmcer 4y ago
If you view software as a bunch of contracts between different services/libraries/etc. it is hard to ensure that each piece upholds its contract, so we trust them to a degree. I can see how a failure in one place due to load would be hard to catch or gracefully fail from. That said they have a pretty explicit cap on maximum activity (number of planes) so it is weird they didn't test around this. It isn't like 400,000 aircraft suddenly DDOS their system.
jg42 4y ago
The key here seems to be the word "many." Alaska Airlines has 289 airplanes (per wikipedia). All the other arguments seem to assume a large consumer type of load - tens of thousands of users, etc... I just can't see an undue strain being placed on a well designed system from < 300 data points. And I haven't even accounted for the distribution of needing to compute takeoff data over the course of a day nor how many planes are NOT taking off at the same time, etc... Also, to somewhat change the topic, didn't Alaska Airlines disband their QA org a few years ago as part of cost cutting? IIRC, they did this to model the software company models (that ship bugs regularly to consumers) and seem to be getting some data that they need to bring back that org...
- Aloha 4y ago
  They also mentioned American was using the same system too. So it may have been load from more than one company.
  - hn_go_brrrrr 4y ago
    Good observation. I had assumed each company had their own instance, but that's not necessarily true.
dilyevsky 4y ago
From the description it sounds like they tried to give a non-technical explanation of a concurrency bug
stateofinquiry 4y ago
So much I can't understand here. Why would the load matter- is this a web app? Why would that possibly be a good idea? This seems like software that should run locally, for security, assurance, and auditability reasons. Follow up thought: When software results are this critical, I wonder if a totally separate program should be used as well and result compared. An independent implementation from another vendor.
deleted 4y ago
[deleted]

cratermoon 4y ago

Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs

djanogo 4y ago
This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them. "the bug was missed because it only presented when many aircraft at the same time were using the system"
- hn_throwaway_99 4y ago
  There is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/ https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare.... Obviously sessions should be independent and not sharing data, but that's why it was a bug.
- usea 4y ago
  I believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.
howinteresting 4y ago
Should have used Rust.
- klyrs 4y ago
  A proper transactional database would suffice. You could do this safely and performantly on '00s hardware in PHP3 for crying out loud, serving thousands of planes per second.
  - AdrianB1 4y ago
    Bad coding happen even in transactional databases, see my example in the other comment.
    - klyrs 4y ago
      Note that I didn't say you couldn't write bad PHP3.
- howinteresting 4y ago
  (This was a joke!)
speed_spread 4y ago
I refuse to believe that summing up weights for each of 300 planes is so hard that it can't be made correct by a team of moderately competent developers operating under sane management. There might be some complexity, but it's 2023 and we have enough knowledge and tools to solve that kind of stuff routinely and reliably. Somebody fucked up big time and should be fired never to work in software again.
- lcuff 4y ago
  I agree that it's hard to see from the outside how calculating weight figures for an airplane could be software load-dependent. But this all smacks of the point that Brooks makes in 'The Mythical Man Month' all those years ago. Two guys in a garage can make a program that does 'x', but to make a programming system product that does 'x' is more than an order of magnitude more work. The complexities introduced don't relate to the 'x', but the intricacies of large systems. Deciding that 'someone has fucked up and should never work again'? Sorry, competent and conscientious people regularly make such mistakes. Which is why we need tests, software review, engineering processes et al to catch those mistakes.
- justinclift 4y ago
  Boeing itself (unsure about this company though) was widely reported to have offshored their senior engineers a few years ago. eg: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... (paywalled) https://archive.is/vdg9S https://archive.is/vdg9S (paywall workaround) Note that there were several articles about it around the time, the above one is just the first reasonable seeming one from a quick search.
- AdrianB1 4y ago
  I can give you a simple example on how this can happen, from a real production system in a different regulated environment: all the SQL SELECT statements are WITH (NOLOCK) on MS SQL server, resulting in non-committed (dirty) reads. Under very light load it works fine, on higher loads it reads some dirty data. If you calculate the weight of the plane by adding the weight of the checked in luggage plus passengers read from a shared database, some differences will appear and they will increase with the load as more transactions will be in flight and possibly read or ignored.

basch 4y ago

Maybe a timeout. Some part of a value returned 0 if it didn’t return in time.

coin 4y ago
Sounds like poor error handing. 0 is valid value is not a substitute for no data or error.
sudhirj 4y ago
Not according to the article, the values returned were within the realm of possibility, but wrong. Giving out zeroes might have actually been fine, it’s easy enough to realise there’s a problem if you see your plane weighs 0 pounds.
- basch 4y ago
  Hence “some part of a value.” Such as weight=plane+fuel+people+luggage. If one of the variables became 0 you’d still have the other 3. It was a low enough value to have made multiple pilots question its value that day, like fuel or passenger load was missing completely.
- advisedwang 4y ago
  I mean it might be just one part. E.g. luggage gets counted as zero but passenger weight, fuel weight and aircraft weight correct - that might be pluausible but low.
dehrmann 4y ago
Classic pokemon exception handling.

hn_throwaway_99 4y ago

> but being wrong under load sounds like a horrible internal architecture problem. Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to GitHub: https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/ https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare...

lupire 4y ago
"huge" is not the same as "horrible". One racy boi can take out an airplane.
- hn_throwaway_99 4y ago
  But calling something an "internal architectural problem" implies things are fundamentally designed wrong, and the code needs a rewrite. My point was the architecture can definitely be fine, but you can still have bugs (even catastrophic ones) because someone left out a `synchronized` keyword somewhere.
  - thrashh 4y ago
    It could still be For example, one way to use SQL is to escape your parameters and then concat them straight into your SQL string. However, if someone leaves out an ‘escape’ somewhere, you have a major security incident The other way is to pass parameters separately as data to your SQL driver. It completely negates the problem If you’ve chosen the first way in your project already, you’ve committed to a major internal architectural problem. In the same vein, maybe if your code requires sprinkling ‘synchronized’ everywhere, you did it wrong But that said, we don’t know what actually is the problem since it’s just a marketing statement saying that high load was the cause and that doesn’t tell us anything
Ozzie_osman 4y ago
True, but software for domains where people can die is very different than Cloudflare. The scale is much higher in the latter, and it needs to be less "bulletproof". If you're writing software for giant metal flying cylinder carrying hundreds of people, we can't just brush mistakes off the same way we might with web-based software (and yes, privacy is important, but it's not life or death).

exabrial 4y ago

I was just about to post the _Exact_ same thing. Each execution of the program should be completely independent from other calculations. Something is _Horribly_ wrong with their architecture.

lostlogin 4y ago
I wonder how it works. Obviously you are correct, but the number of moving parts must be huge. The number of passengers, amount of cargo and the fuel load must be changing the whole time.
tbrownaw 4y ago
... Call to the baggage-tracking service returned 429 and an empty body, and that was treated as "there are no bags" because some dweeb [dev, but I'm leaving in what autocorrect said] had been reading to much design guidance from Netflix/Facebook/etc that assumes missing features on a fraction of page loads are NBD?

ericmcer 4y ago

If you view software as a bunch of contracts between different services/libraries/etc. it is hard to ensure that each piece upholds its contract, so we trust them to a degree. I can see how a failure in one place due to load would be hard to catch or gracefully fail from. That said they have a pretty explicit cap on maximum activity (number of planes) so it is weird they didn't test around this. It isn't like 400,000 aircraft suddenly DDOS their system.

jg42 4y ago

The key here seems to be the word "many." Alaska Airlines has 289 airplanes (per wikipedia). All the other arguments seem to assume a large consumer type of load - tens of thousands of users, etc... I just can't see an undue strain being placed on a well designed system from < 300 data points. And I haven't even accounted for the distribution of needing to compute takeoff data over the course of a day nor how many planes are NOT taking off at the same time, etc... Also, to somewhat change the topic, didn't Alaska Airlines disband their QA org a few years ago as part of cost cutting? IIRC, they did this to model the software company models (that ship bugs regularly to consumers) and seem to be getting some data that they need to bring back that org...

Aloha 4y ago
They also mentioned American was using the same system too. So it may have been load from more than one company.
- hn_go_brrrrr 4y ago
  Good observation. I had assumed each company had their own instance, but that's not necessarily true.

dilyevsky 4y ago

From the description it sounds like they tried to give a non-technical explanation of a concurrency bug

stateofinquiry 4y ago

So much I can't understand here. Why would the load matter- is this a web app? Why would that possibly be a good idea? This seems like software that should run locally, for security, assurance, and auditability reasons. Follow up thought: When software results are this critical, I wonder if a totally separate program should be used as well and result compared. An independent implementation from another vendor.

deleted 4y ago

[deleted]