4 ms·

> the bug was missed because it only presented when many aircraft at the same time were using the system. How the heck do you end up with a concurrency bug in

> the bug was missed because it only presented when many aircraft at the same time were using the system. How the heck do you end up with a concurrency bug in a system where the calculations appear to be logically independent?

speedgoose 4y ago
A concurrency bug that got fixed in production 5 hours later. It must be very unusual code or some people are excellent at debugging concurrency problems.
- 83 4y ago
  Article says that bug was introduced in an update that morning. Somewhere there was a developer familiar with the code and exactly what changed. Wouldn't be surprised if he smacked himself on the forehead and knew immediately what the issue was when he heard about the bug.
  - ldarby 4y ago
    No it doesn't say that. > That morning, a software bug in an update to the DynamicSource tool caused ... This says when the bug happened, not when the update was made.
- andix 4y ago
  Or they just rolled back an update. I don’t think you can do much more in 5 hours.
  - hungryforcodes 4y ago
    Right. Even if the developer knew exactly what he did -- any sane provider would put the updated update through a full QC cycle, which would take a while I'd figure.
- jacquesm 4y ago
  This sort of thing - in my uninformed opinion - should not be a service, and should be validated strictly as though it is part of the avionics software. After all it sends data that directly affects the power settings the pilots will use just as if those power settings were directly affected by the output of the program. That makes it an integral part of the plane to me.
  - notahacker 4y ago
    I believe the tool takes data from multiple sources including airline reservation systems, baggage handling systems, refuelling systems and cabin crew confirming people are correctly seated on handheld devices and validates it for consistency before performing a standardised set of calcs. In which case the safety argument for having that airgapped from the flight computer, which receives only a few final calculated values (manually input by a pilot who also can potentially spot outliers) is probably stronger than the safety argument for integrating everything or manually inputting a large number of values into the flight computer (creating potential remote attack vectors and increased likelihood of human error respectively). Would have thought lower than ideal safety margins on takeoff speed (the calculations were apparently <20% off the true value) were a more concerning problem
    - cratermoon 4y ago
      > the tool takes data from multiple sources And 90% of those sources are ancient balls of mud that spit out stuff that looks like !ORD 06/001 ORD RWY 04L/22R CLSD 2106231700-2106232300 or KAUS 092135Z 26018G25KT 8SM -TSRA BR SCT045CB BKN060 OVC080 30/21 A2992 RMK FQT LTGICCCCG OHD-W MOVG E RAB25 TSB32 CB ALQDS SLP132 P0035 T03020210 =
      - ArchOversight 4y ago
        Excuse me sir... that language is simply inappropriate here! The fact that a lot of this output is almost complete inscrutable means you have to build a tool or something to translate it from that back to something a human can quickly read/grasp... and translation mistakes can happen there too :/
basch 4y ago
Resource contention causing part of sum to fail to be incorporated.
- ballenf 4y ago
  That seems plausible. E.g., if the package cargo API timed out it could be treated as a 0.
  - jcutrell 4y ago
    It does seem plausible, but also almost heinous design if true.
- jcutrell 4y ago
  Yeah the more I think about it the more this makes sense to me. Both were lighter than expected (so if it was a swap with another aircraft we wouldn't necessarily know - possible that 4 aircraft got the wrong data and we don't know the other two). But it feels more likely that the sums just failed and they incorrectly caught the failures instead of letting it fall out. catch(e){} is a bug factory for sure.
- pixl97 4y ago
  That was my guess as I've seen people use non fatal timeout conditions and keep processing when a value isn't returned from an external data source.
fabian2k 4y ago
Maybe it sent the result of a different aircraft? So each calculation was still correct, but they got mixed up when responding.
- jcutrell 4y ago
  I've thought about this - if this is the case, they are extremely lucky because different planes could have significantly different payloads. Of course, then the TLAR would signal an issue. Also this would mean some other aircraft had a particularly powerful departure, since both aircraft had lower-than-true weight calcs.
dezgeg 4y ago
Someone forgot their 'if err != nil { return err }' check from their microservice call (whenever it happens to be overloaded) and just plowed through with some intermediate weight component being zero?
- Filligree 4y ago
  "On error resume next" is a perfectly fine default error handling strategy.
  - iforgotpassword 4y ago
    Especially for mission critical software that lives depend on.
the-rc 4y ago
It could be something as pedestrian as the software writing and later reading back from /tmp/somerandomfile. Or retrying with similar, predictable names.
pmarreck 4y ago
It's almost like we should try to stick to languages where creating concurrency bugs is nearly impossible, thereby pre-empting an entire class of bug that might have resulted in deaths
- hungryforcodes 4y ago
  What language would be a good example of that?
  - pmarreck 4y ago
    any functional one with immutable data would suffice. my current favorite is Elixir, but others fit this bill. There is still the matter of dealing with mutable stores like databases, but well-written code would make that impossible as well, plus permit tests to run concurrently

speedgoose 4y ago

A concurrency bug that got fixed in production 5 hours later. It must be very unusual code or some people are excellent at debugging concurrency problems.

83 4y ago
Article says that bug was introduced in an update that morning. Somewhere there was a developer familiar with the code and exactly what changed. Wouldn't be surprised if he smacked himself on the forehead and knew immediately what the issue was when he heard about the bug.
- ldarby 4y ago
  No it doesn't say that. > That morning, a software bug in an update to the DynamicSource tool caused ... This says when the bug happened, not when the update was made.
andix 4y ago
Or they just rolled back an update. I don’t think you can do much more in 5 hours.
- hungryforcodes 4y ago
  Right. Even if the developer knew exactly what he did -- any sane provider would put the updated update through a full QC cycle, which would take a while I'd figure.
jacquesm 4y ago
This sort of thing - in my uninformed opinion - should not be a service, and should be validated strictly as though it is part of the avionics software. After all it sends data that directly affects the power settings the pilots will use just as if those power settings were directly affected by the output of the program. That makes it an integral part of the plane to me.
- notahacker 4y ago
  I believe the tool takes data from multiple sources including airline reservation systems, baggage handling systems, refuelling systems and cabin crew confirming people are correctly seated on handheld devices and validates it for consistency before performing a standardised set of calcs. In which case the safety argument for having that airgapped from the flight computer, which receives only a few final calculated values (manually input by a pilot who also can potentially spot outliers) is probably stronger than the safety argument for integrating everything or manually inputting a large number of values into the flight computer (creating potential remote attack vectors and increased likelihood of human error respectively). Would have thought lower than ideal safety margins on takeoff speed (the calculations were apparently <20% off the true value) were a more concerning problem
  - cratermoon 4y ago
    > the tool takes data from multiple sources And 90% of those sources are ancient balls of mud that spit out stuff that looks like !ORD 06/001 ORD RWY 04L/22R CLSD 2106231700-2106232300 or KAUS 092135Z 26018G25KT 8SM -TSRA BR SCT045CB BKN060 OVC080 30/21 A2992 RMK FQT LTGICCCCG OHD-W MOVG E RAB25 TSB32 CB ALQDS SLP132 P0035 T03020210 =
    - ArchOversight 4y ago
      Excuse me sir... that language is simply inappropriate here! The fact that a lot of this output is almost complete inscrutable means you have to build a tool or something to translate it from that back to something a human can quickly read/grasp... and translation mistakes can happen there too :/

basch 4y ago

Resource contention causing part of sum to fail to be incorporated.

ballenf 4y ago
That seems plausible. E.g., if the package cargo API timed out it could be treated as a 0.
- jcutrell 4y ago
  It does seem plausible, but also almost heinous design if true.
jcutrell 4y ago
Yeah the more I think about it the more this makes sense to me. Both were lighter than expected (so if it was a swap with another aircraft we wouldn't necessarily know - possible that 4 aircraft got the wrong data and we don't know the other two). But it feels more likely that the sums just failed and they incorrectly caught the failures instead of letting it fall out. catch(e){} is a bug factory for sure.
pixl97 4y ago
That was my guess as I've seen people use non fatal timeout conditions and keep processing when a value isn't returned from an external data source.

fabian2k 4y ago

Maybe it sent the result of a different aircraft? So each calculation was still correct, but they got mixed up when responding.

jcutrell 4y ago
I've thought about this - if this is the case, they are extremely lucky because different planes could have significantly different payloads. Of course, then the TLAR would signal an issue. Also this would mean some other aircraft had a particularly powerful departure, since both aircraft had lower-than-true weight calcs.

dezgeg 4y ago

Someone forgot their 'if err != nil { return err }' check from their microservice call (whenever it happens to be overloaded) and just plowed through with some intermediate weight component being zero?

Filligree 4y ago
"On error resume next" is a perfectly fine default error handling strategy.
- iforgotpassword 4y ago
  Especially for mission critical software that lives depend on.

the-rc 4y ago

It could be something as pedestrian as the software writing and later reading back from /tmp/somerandomfile. Or retrying with similar, predictable names.

pmarreck 4y ago

It's almost like we should try to stick to languages where creating concurrency bugs is nearly impossible, thereby pre-empting an entire class of bug that might have resulted in deaths

hungryforcodes 4y ago
What language would be a good example of that?
- pmarreck 4y ago
  any functional one with immutable data would suffice. my current favorite is Elixir, but others fit this bill. There is still the matter of dealing with mutable stores like databases, but well-written code would make that impossible as well, plus permit tests to run concurrently