3 ms·

I remember reading --- and vehemently disagreeing --- with the report on the incidents, which danced around the matter but didn't point directly at the underlyi

I remember reading --- and vehemently disagreeing --- with the report on the incidents, which danced around the matter but didn't point directly at the underlying cause: excessive complexity in the software, which easily created bugs and hid them. For example, they used multiple threads and a whole OS, when a simple loop would've been sufficient, perhaps in a misguided attempt at trying to keep the UI "responsive"; there would not be any race conditions if everything ran in the same thread. As Tony Hoare says: "There are two ways to develop software: Make it so simple that there are obviously no bugs, or so complex that there are no obvious bugs."

AlotOfReading 2y ago
Having written a lot of safety critical code, I don't think you could ever convince me that multithreading was the most important safety issue. It was a safety critical system written by a single anonymous person in assembly without adequate testing. The process was fundamentally broken. Changing one technical decision would have simply exposed another failure somewhere else.
- MarkusWandel 2y ago
  It's been a while since I read the detailed analysis, but wasn't it basically this? 1. A bug in the software that was prevented from doing harm (not necessarily masked but reduced to a minor irritant) by a hardware safety interlock. 2. The hardware safety interlock being removed in a cost-reduction. I know that "field mileage" is a powerful indicator of quality. Even if the code is awful and full of workarounds - if you have a lot of field experience you'd be loath to mess with it. So by the time the cost reduction came along the software would be fully trusted.
  - AlotOfReading 2y ago
    Yes, that's essentially the story. It misses a lot of safeguards that should have existed (from a hindsight is 20/20 perspective). There's no "fully trusted" in safety critical code, so the software should have gone through a complete review when the safety model changed from hardware based to software based, including a HARA analysis and single point of failure analysis. The software should have had a specification and testing that included things like counter rollover and variable race conditions. The single developer should have had someone else to review their work and catch mistakes. This is standard practice these days, in large part specifically to prevent disasters like therac-25. While it's a little unfair to judge them by standards that didn't exist yet, it was still a broken process.
  - reginald78 2y ago
    IIRC some of the underlying bugs came about because they used a much less dangerous device (X-Ray only) as the base and added the more dangerous system on top. The reused software components had different units of measure than the new mode which wasn't exposed until things were in "unexpected" error states. Then there was the usual arrogant gas lighting of anyone who reported the machine injuring and killing people. So the standard mix of computers are infallible and everyone in the change covering their asses by covering their ears and eyes so that the machine could kill again and again.
- userbinator 2y ago
  I fully believe this software could've been written correctly by a single person; just one who knows the simplicity of the requirements and isn't trying to show off with complexity. If you've written safety-critical code then you've probably worked in an industry that tries its hardest to dilute and reduce responsibility, and while imposing onerous processes may help, it doesn't fix the underlying problem.
jwr 2y ago
I think about software reliability a lot and try to generalize failure causes (I have about 30 years of experience designing and building systems). While I do have a number of top bug-causing patterns, the above comment is spot-on: unnecessary complexity, while not being the direct cause of problems, almost always comes up when analyzing failures. You have a much better chance of getting your system right if you avoid complexity. If you want practical real-life examples: many engineers when building a small embedded device will use a Raspberry Pi or something similar that runs Linux. It's easier for them: they get a full OS that they are used to, there is Python, all familiar. And yet, quite often all this is overkill for the task at hand, which is monitoring temperature, or controlling a bunch of GPIOs. You pull in a humongously complex stack of software, with thousands of quirks, bugs, and quite a bit of emergent behavior that no one understands, even though the task could have been performed by a bare-metal program running on a microcontroller, or (if we want convenience) running under Zephyr.
- preisschild 2y ago
  An even worse example would be using a full-blown windows & small PC just to show a kiosk interface, for example to show ride information in public transit. It's broken so often due to various issues (booting issues, windows main screen without programs, other random windows issues and so on)
michaelt 2y ago
> For example, they used multiple threads and a whole OS, when a simple loop would've been sufficient, perhaps in a misguided attempt at trying to keep the UI "responsive"; there would not be any race conditions if everything ran in the same thread. Of course, there are also high-profile computer errors caused by UI unresponsiveness (in combination with inept programming). Here in the UK a bunch of post office employees were convicted of fraud when they'd done nothing wrong, due to a computer system with hundreds of bugs. "One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for." https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...
- userbinator 2y ago
  That looks like a different type of multithreading bug, again caused by "decoupling" the UI.

AlotOfReading 2y ago

Having written a lot of safety critical code, I don't think you could ever convince me that multithreading was the most important safety issue. It was a safety critical system written by a single anonymous person in assembly without adequate testing. The process was fundamentally broken. Changing one technical decision would have simply exposed another failure somewhere else.

MarkusWandel 2y ago
It's been a while since I read the detailed analysis, but wasn't it basically this? 1. A bug in the software that was prevented from doing harm (not necessarily masked but reduced to a minor irritant) by a hardware safety interlock. 2. The hardware safety interlock being removed in a cost-reduction. I know that "field mileage" is a powerful indicator of quality. Even if the code is awful and full of workarounds - if you have a lot of field experience you'd be loath to mess with it. So by the time the cost reduction came along the software would be fully trusted.
- AlotOfReading 2y ago
  Yes, that's essentially the story. It misses a lot of safeguards that should have existed (from a hindsight is 20/20 perspective). There's no "fully trusted" in safety critical code, so the software should have gone through a complete review when the safety model changed from hardware based to software based, including a HARA analysis and single point of failure analysis. The software should have had a specification and testing that included things like counter rollover and variable race conditions. The single developer should have had someone else to review their work and catch mistakes. This is standard practice these days, in large part specifically to prevent disasters like therac-25. While it's a little unfair to judge them by standards that didn't exist yet, it was still a broken process.
- reginald78 2y ago
  IIRC some of the underlying bugs came about because they used a much less dangerous device (X-Ray only) as the base and added the more dangerous system on top. The reused software components had different units of measure than the new mode which wasn't exposed until things were in "unexpected" error states. Then there was the usual arrogant gas lighting of anyone who reported the machine injuring and killing people. So the standard mix of computers are infallible and everyone in the change covering their asses by covering their ears and eyes so that the machine could kill again and again.
userbinator 2y ago
I fully believe this software could've been written correctly by a single person; just one who knows the simplicity of the requirements and isn't trying to show off with complexity. If you've written safety-critical code then you've probably worked in an industry that tries its hardest to dilute and reduce responsibility, and while imposing onerous processes may help, it doesn't fix the underlying problem.

jwr 2y ago

I think about software reliability a lot and try to generalize failure causes (I have about 30 years of experience designing and building systems). While I do have a number of top bug-causing patterns, the above comment is spot-on: unnecessary complexity, while not being the direct cause of problems, almost always comes up when analyzing failures. You have a much better chance of getting your system right if you avoid complexity. If you want practical real-life examples: many engineers when building a small embedded device will use a Raspberry Pi or something similar that runs Linux. It's easier for them: they get a full OS that they are used to, there is Python, all familiar. And yet, quite often all this is overkill for the task at hand, which is monitoring temperature, or controlling a bunch of GPIOs. You pull in a humongously complex stack of software, with thousands of quirks, bugs, and quite a bit of emergent behavior that no one understands, even though the task could have been performed by a bare-metal program running on a microcontroller, or (if we want convenience) running under Zephyr.

preisschild 2y ago
An even worse example would be using a full-blown windows & small PC just to show a kiosk interface, for example to show ride information in public transit. It's broken so often due to various issues (booting issues, windows main screen without programs, other random windows issues and so on)

michaelt 2y ago

> For example, they used multiple threads and a whole OS, when a simple loop would've been sufficient, perhaps in a misguided attempt at trying to keep the UI "responsive"; there would not be any race conditions if everything ran in the same thread. Of course, there are also high-profile computer errors caused by UI unresponsiveness (in combination with inept programming). Here in the UK a bunch of post office employees were convicted of fraud when they'd done nothing wrong, due to a computer system with hundreds of bugs. "One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for." https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...

userbinator 2y ago
That looks like a different type of multithreading bug, again caused by "decoupling" the UI.