3 ms·
Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs
by cratermoon 4y ago
Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs
- djanogo 4y agoThis wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them. "the bug was missed because it only presented when many aircraft at the same time were using the system"
- hn_throwaway_99 4y agoThere is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/ https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare.... Obviously sessions should be independent and not sharing data, but that's why it was a bug.
- usea 4y agoI believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.
- howinteresting 4y agoShould have used Rust.
- klyrs 4y agoA proper transactional database would suffice. You could do this safely and performantly on '00s hardware in PHP3 for crying out loud, serving thousands of planes per second.
- howinteresting 4y ago(This was a joke!)
- speed_spread 4y agoI refuse to believe that summing up weights for each of 300 planes is so hard that it can't be made correct by a team of moderately competent developers operating under sane management. There might be some complexity, but it's 2023 and we have enough knowledge and tools to solve that kind of stuff routinely and reliably. Somebody fucked up big time and should be fired never to work in software again.
- lcuff 4y agoI agree that it's hard to see from the outside how calculating weight figures for an airplane could be software load-dependent. But this all smacks of the point that Brooks makes in 'The Mythical Man Month' all those years ago. Two guys in a garage can make a program that does 'x', but to make a programming system product that does 'x' is more than an order of magnitude more work. The complexities introduced don't relate to the 'x', but the intricacies of large systems. Deciding that 'someone has fucked up and should never work again'? Sorry, competent and conscientious people regularly make such mistakes. Which is why we need tests, software review, engineering processes et al to catch those mistakes.
- justinclift 4y agoBoeing itself (unsure about this company though) was widely reported to have offshored their senior engineers a few years ago. eg: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... (paywalled) https://archive.is/vdg9S https://archive.is/vdg9S (paywall workaround) Note that there were several articles about it around the time, the above one is just the first reasonable seeming one from a quick search.
- AdrianB1 4y agoI can give you a simple example on how this can happen, from a real production system in a different regulated environment: all the SQL SELECT statements are WITH (NOLOCK) on MS SQL server, resulting in non-committed (dirty) reads. Under very light load it works fine, on higher loads it reads some dirty data. If you calculate the weight of the plane by adding the weight of the checked in luggage plus passengers read from a shared database, some differences will appear and they will increase with the load as more transactions will be in flight and possibly read or ignored.