8 ms·

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development pr

> software quality doesn't appear because you have good developers. It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think type-systems, unit-testing and defensive-coding can solve all software problems. They definitely can help a lot, but the real failure in the story of the Therac-25 from my understanding, is that it took far too long for incidents to be reported, investigated and fixed. There was a great Cautionary Tales podcast about the device recently[0], one thing mentioned was that, even aside from the catasrophic accidents, Therac-25 machines were routinely seen by users to show unexplained errors, but these issues never made it to the desk of someone who might fix it. [0] https://timharford.com/2025/07/cautionary-tales-captain-kirk-forgot-to-put-the-machine-on-stun-2/ https://timharford.com/2025/07/cautionary-tales-captain-kirk...

ChrisMarshallNY 1y ago
I worked for a company that manufactured some of the highest-Quality photographic and scientific equipment that you can buy. It was expensive as hell, but our customers seemed to think it was worth it. > It's the end result of a process In my experience, it's even more than that. It's a culture.
- franktankbank 1y ago
  A culture of high-quality engineering, no doubt. Made up of: high quality engineers!
  - ChrisMarshallNY 1y ago
    Yes, but some of them were the most stubborn bastards I've ever worked with.
    - franktankbank 1y ago
      That's high praise I'm sure.
      - ChrisMarshallNY 1y ago
        The results spoke for themselves, but it could be maddening.
    - snerbles 1y ago
      They have to be.
  - anonymars 1y ago
    Isn't that exactly the opposite of the point being made? > software quality doesn't appear because you have good developers
    - ChrisMarshallNY 1y ago
      Not really. Good developers are a necessary ingredient of a much larger recipe. People think that a good process means you can toss in crap developers, or that great developers mean that you can have a bad process. In my experience, I worked for a 100-year-old Japanese engineering company that had a decades-long culture of Quality. People stayed at that company for their entire career, and most of them were top-shelf people. They had entire business units, dedicated to process improvement and QA. It was a combination of good talent, good process, and good culture. If any one of them sucks, so does the product.
      - ChrisMarshallNY 1y ago
        It really is funny how any discussion of improving software quality becomes unpopular, here.
        ponector 1y ago
        Not only here. Everyone wants to use a quality products, but no almost one is committed to deliver quality products/services. No one is working on quality, everyone works on new features. There is usually no incentive to increase quality, to improve speed, performance, etc.
  - herval 1y ago
    you don't need "high quality engineers" to have high-quality outputs. And vice-versa - lots of places with very high quality engineers produce terribly low-quality software
    - ChrisMarshallNY 1y ago
      I guess we see things differently. They don't need to be especially talented engineers, but, in my experience (and I actually have quite a bit of it, in this area), they need to be dedicated to a culture of Quality. And it is entirely possible for very talented engineers to produce shite. I've seen exactly that.
      - herval 1y ago
        A culture of quality doesn’t require particularly skilled individuals to function. That’s in fact the thesis for the entire Deming management philosophy, and in line with what I’m saying (you can produce high quality with a good process or a good culture, you don’t necessarily need high caliber individuals)
        ChrisMarshallNY 1y ago
        Japan used a lot of Deming’s theories, to significant success. I worked for a Japanese company. In my case, the company produced absolutely top-shelf stuff, but even relatively mediocre companies did well, using Deming’s techniques. It required that everyone be on board, wrt the culture, though. But I have found that a “good” engineer is one that takes their vocation seriously. They may not be that accomplished or skilled, but they have self-discipline, humility, and structure. I’ve met quite a few highly-skilled “not-good” engineers, in my day. I’m embarrassed to say that I’ve hired some of them.
    - fesenjoonior 1y ago
      > you don't need "high quality engineers" to have high-quality outputs. [citation needed]
  - kulahan 1y ago
    Unfortunately, software developers are the absolute most offensive use of the word "engineer", because 99.9% of the stuff this field makes is a competition to take the most unique approach to a solution, then getting it bandaged together with gum and paperclips. If this industry wants to be respected, it should start trying to be actual engineers. There should be tons and tons of standards which are enforced legally, but this is not often the case. Imagine if there were no real legal guardrails in, say, bridge building! edit: and imagine if any time you brought up this issue, bridge builders cockily responded with "well stuff seems to work fine so..."
    - I_dream_of_Gen1 1y ago
      [dead]
    - ChrisMarshallNY 1y ago
      Yeah, this is a fairly classic challenge. When it comes to safety stuff (like bridge building), there are (and should be) strict licensing requirements. I would have no problem requiring such for work on things like medical equipment, We already require security clearance for things like defense information (unless you're a DOGE bro, I guess). That's a bit different from engineering creds, but it's an example of imposed structure. But I think that it would be ridiculous to require it for someone that writes a fart app (unless it's a weaponized fart app). What is in those requirements then becomes a hot potato. There are folks that would insist that any "engineer" be required to know how to use a slide rule, and would ignore modern constructs like LLMs and ML. I'm not kidding. I know people exactly like that. If they get authority, watch out. They'll only "approve" stuff that they are good at. On the other hand, if the requirements are LeetCode, then it's useless. A lot of very unqualified people would easily pass, and wreak havoc. From what I can see, the IEEE seems to have a fairly good grasp on mixing classic structure and current tech. There's some very good people, there, and they are used to working in a structured manner. But software has developed a YOLO culture. People are used to having almost no structure, and they flit between organizations so rapidly, that it's almost impossible to keep track of who is working on what. The entire engineering culture needs to be changed. I don't see that being something that will come easily. I'm big on Structure and Discipline. A lot of it has to do with almost 27 years at a corporation with so much structure that a lot of folks here, would be whimpering under their standing desks. That structure was required, in order to develop equipment of the Quality they are famous for, but would be total overkill for a lot of stuff. I do think that we need to impose structure on software supply chains, though. That's not something that will be a popular stance. Structure is also not cheap. Someone needs to pay for it, and that's when you become a real skunk at the picnic.
- f1shy 1y ago
  It is a culture. No doubt. And certainly not only processes. I work in a company where there are literally processes for everything. For every engineer doing actual work (requirement engineering, architecture, coding, testing) there are at least 3!doing processes. The SW we make is the ultimative piece of shit, late, expensive, and to the brim full of bugs. So process is importannt, but good engineering, and culture also.
chairmansteve 1y ago
I would say the real lesson is that the Therac machine should have had hardware interlocks (mentioned but not emphasised in the article).
WalterBright 1y ago
I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the Therac-25, that means a detector of the amount of radiation being generated, which will cut it off if it exceeds a safe value. I'd also add that the radiation generator be physically incapable of generating excessive radiation.
- vjvjvjvjghv 1y ago
  In general I agree but there is bit more complexity. I work in medical devices and there are plenty of situations where a certain output is ok in some circumstance but deadly in another. That makes a stopgap a little more tricky. I agree with the previous poster that the feedback from the field is lacking a lot. A lot of doctors don’t report problems back because they are used to bad interfaces. And then the feedback gets filtered through several layers of sales reps and product management. So a lot of info gets lost and fixes that could be simple won’t get done. In general when you work in medical you are so overwhelmed by documentation and regulation that there isn’t much time left to do proper engineering. The FDA mostly looks at documentation done right and less at product done right.
  - WalterBright 1y ago
    At Boeing there's a required "failure analysis" document listing all the failure modes and why they won't cause a crash by themselves.
    - fsulkner 1y ago
      [dead]
    - Aloha 1y ago
      Agreed - This is essentially the corner stone of systems failure analysis - something I wish architects thought about more in the software space. I'm a product manager for an old (and if I'm being honest somewhat crusty) system of software, the software is buggy, all of it is, but its also self healing and resilient, so while yes, it fails with somewhat alarming regularity, with very lots and lots concerning looking error messages in the logs, but it never causes an outage because it self heals. Good systems design isn't making bug free software or a bug free system, but rather a system where a total outage requires N+1 (maybe even N+N) things to fail before the end user notices. Failures should be driven by at most, edge cases - basically where the system is being operated outside of its design parameters, and those parameters need to reflect the real world and be known by most stakeholders in the system. My gripe with software engineers sometimes, they're often too divorced from real users and real use cases, and too devoted to the written spec over what their users actually need to do with the software - I've seen some very elegant (and on paper, well designed) systems fall apart because if simple things like intermittent packet jitter, or latency swings (say between 10ms and 70ms) - these are real world conditions, often encountered by real world systems, but these spec driven systems fall apart once confronted with reality.
credit_guy 1y ago
I'm not sure. Most software (by orders of magnitude) is not critical software like the software running that X-ray machine. In general, if your software fails, a page loads too slow, or a report comes with lots of NaN's, or some batch job does not run at the right time, and someone needs to start it manually, etc. The cases where someone dies because of a software quality issue are very rare, and the developers working on that type of software know who they are and what their duties are (I hope).
rowanG077 1y ago
I think the opposite. The only reason software quality emerges is because of good developers. It's a prerequisites. Process helps good developers deliver quality. But there is no process that allows a team of bad developers deliver quality. you can't squeeze blood from a stone.
- benrutter 1y ago
  That's true, but even so - great developers still make mistakes, and if they don't hear about production errors because of a breakdown in the customer communications from sales etc, then those mistakes will never be fixed. It's not that great developers aren't necessary for software quality, more that they aren't sufficient.
  - rowanG077 1y ago
    I think great developers are necessary for software quality. But indeed often not sufficient.
msy 1y ago
I couldn't disagree more. Outside of exotic scenarios where things like formal proofing are possible and economically viable I've never seen a process that prevents bugs, only culture. Good engineering cultures are also often ones with well defined and tested processes and good testing practices but it's the culture and people giving a shit that makes the difference, not the other way around. Good product cultures are ones where natural communication between the field and engineering would mean issues get reported back up and make their way to the right people. No process will compensate for people not giving a shit.
- benrutter 1y ago
  > It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. I think the bit I quoted, especially if you read in the context of the article, is talking about culture. I.e. it's talking about a process that informs software development, management and sales. Things like formal proofing and type systems are the exact kind of processes that aren't what it's talking about. I kind of agree with you though about the process/culture distinction - ultimately, if you don't have a culture where people actively care about improving reliability, any process is just gonna become a tick-box exercise to appease management.
AdamN 1y ago
This is true but there also needs to be good developers as well. It can't just be great process and low quality developer practices. There needs to be: 1/ high quality individual processes (development being one of them), 2/ high quality delivery mechanisms, 3/ feedback loops to improve that quality, 4/ out of band mechanisms to inspect and improve the quality.
- Fr3dd1 1y ago
  I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.
  - franktankbank 1y ago
    The process that makes this work would be so onerous to create. Would you think you could do this to make a low quality machinist be able to build a high quality technical part? What would this look like? Quite a lot like machine code which doesn't really reduce the requirements does it? It actually just shifted the onerous requirement somewhere else.
  - vjvjvjvjghv 1y ago
    “ This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process” That’s a horrible take. There is no amount of reviews, guidelines and documentation that can compensate for low quality devs. You can’t throw garbage into the pipeline and then somehow process it to gold.
  - Cthulhu_ 1y ago
    > (this includes almost all of us at some point in time) I'd say this includes all of us all the time; a good developer never trusts their own work blindly, and spends more time gathering requirements and verifying their and others' work than writing code.
  - quietbritishjim 1y ago
deleted 1y ago
[deleted]
vorgol 1y ago
I was going to recommend that exact podcast episode but you beat me to it. Totally worth listening, especially if you're interested in software bugs. Another interesting fact mentioned in the podcast is that the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. Excellent demonstration of the Swiss Cheese Model: https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
- ipython 1y ago
  Don’t worry we are poised to re learn all these lessons once again with our fancy new agentic generative ai systems. The mechanical interlock essentially functioned as a limit outside of the control system. So you should build an ai system the same way- enforcing restrictions on the security agency from outside the control of the ai itself. Of course that doesn’t happen and devs naively trust that the ai can make its own security decisions. Another lesson from that era we are re learning- in-band signaling. Our 2025 version of the “blue box” is in full swing. Prompt injection is just a side effect of the fact that there is no out of band instruction mechanism for llms. Good news is - it’s not hard to learn the new technology when it’s just a matter of rediscovering the same security issues with a new name!
- bell-cot 1y ago
  >> the real failure in the story of the Therac-25 from my understanding, is that it took far too long for incidents to be reported, investigated and fixed. > the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. #1 virtue of electromechanical failsafes is that their conception, design, implementation, and failure modes tend to be orthogonal to those of the software. One of the biggest shortcomings of Swiss Cheese safety thinking is that you too-often end up using "neighbor slices from the same wheel of cheese". #2 virtue of electromechanical failsafes is that running into them (the fuse blew, or whatever) is usually more difficult for humans to ignore. Or at least it's easier to create processes and do training that actually gets the errors reported up the chain. (Compared to software - where the worker bees all know you gotta "ignore, click 'OK', retry, reboot" all the time, if you actually want to get anything done): But, sadly, electromechanical failsafes are far more expensive then "we'll just add some code to check that" optimism. And PHB's all know that picking up nickles in front of the steamroller is how you get to the C-suite.
sonicggg 1y ago
Not sure why the article is focusing so much on software development. That was just a piece of the problem. The entire product had design flaws. When the FDA for involved, the company wasn't just told to make software updates.
- speed_spread 1y ago
  Yet It doesn't take much to swamp a team of good developers. A poorly defined project, mismatched requirements, sent to production too early and then put in support mode with no time planned to plug the holes... There's only so much smart technicians can do when the organization is broken.
aaron695 1y ago
[dead]
0xDEAFBEAD 1y ago
Honestly I wish instead of the Therac-25, we were discussing a system which made use of unit testing and defensive coding, yet still failed. That would be more educational. It's too easy to look at the Therac-25 and think "I would never write a mess like that".
- roeles 1y ago
  One instance that crosses my mind often is the airbus a320 incident at Hamburg in 2008. Everything was done right there, but the requirements were wrong. Despite all the procedures and tests, the software still managed to endanger the lives of the passengers.
  - 0xDEAFBEAD 1y ago
    Interesting, do you happen to have a case study?
    - deleted 1y ago
      [deleted]
    - shmeeed 1y ago
      https://skybrary.aero/sites/default/files/bookshelf/1258.pdf https://skybrary.aero/sites/default/files/bookshelf/1258.pdf
  - I_dream_of_Geni 1y ago
    Speaking of Airbus, They 'lost' 3-4 different aircraft (from 1988 to 2015) which crashed during development, or, spectacularly during their first airshow. Never slowed down their customers at ALL, and to this day, Boeing has never lost one new commercial airliner in those same circumstances. Yet, Boeing gets all the hate. smh
  - Izkata 1y ago
    The Boeing 737 MAX had an additional safety feature that was causing crashes due to bad input from the sensors, that pilots didn't know about so they couldn't override. This was 2018 and 2019. After the first crash, the manuals and training were updated to explain what was going on and how to override it. https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Augmentation_System https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Au...
pjmlp 1y ago
The worst part is that many devlopers think that by not working with high integrity systems, such quality levels don't apply to them. Wrong, any software failure can have huge consequences in someone's life, or company, by preventing some critical flow to take place, corrupting data related to someone's life, professional or medical record, preventing a payment on some specific goods that had to be acquired on that moment or never,....
- ozim 1y ago
  Hey don’t blame developers. It is business who requests features ASAP to cut costs and and then there are customers who don’t want to pay for „ideal software” but rather have every software for free. Most devs and QA workers I know want to deliver best quality software and usually are gold plating stuff anyway.
  - pjmlp 1y ago
    Being a real Software Engineer, those that actually have the proper title, eventually with the final examination, means being able to deliver the best product within the set of given constraints. Also, speaking out when the train is visibly going against a wall.
    - bobmcnamara 1y ago
      In my country those are a myth. We had it as a professional engineering classification for a while but I'm not sure if anyone ever completed it. They cancelled it several years ago.
      - pjmlp 1y ago
        And in US anyone can call themselves whatever they feel like. If you want professional quality, we're the first line of actually making it happen, blaming others won't change anything.
        1y ago

ChrisMarshallNY 1y ago

I worked for a company that manufactured some of the highest-Quality photographic and scientific equipment that you can buy. It was expensive as hell, but our customers seemed to think it was worth it. > It's the end result of a process In my experience, it's even more than that. It's a culture.

franktankbank 1y ago
A culture of high-quality engineering, no doubt. Made up of: high quality engineers!
- ChrisMarshallNY 1y ago
  Yes, but some of them were the most stubborn bastards I've ever worked with.
  - franktankbank 1y ago
    That's high praise I'm sure.
    - ChrisMarshallNY 1y ago
      The results spoke for themselves, but it could be maddening.
  - snerbles 1y ago
    They have to be.
- anonymars 1y ago
  Isn't that exactly the opposite of the point being made? > software quality doesn't appear because you have good developers
  - ChrisMarshallNY 1y ago
    Not really. Good developers are a necessary ingredient of a much larger recipe. People think that a good process means you can toss in crap developers, or that great developers mean that you can have a bad process. In my experience, I worked for a 100-year-old Japanese engineering company that had a decades-long culture of Quality. People stayed at that company for their entire career, and most of them were top-shelf people. They had entire business units, dedicated to process improvement and QA. It was a combination of good talent, good process, and good culture. If any one of them sucks, so does the product.
    - ChrisMarshallNY 1y ago
      It really is funny how any discussion of improving software quality becomes unpopular, here.
      - ponector 1y ago
        Not only here. Everyone wants to use a quality products, but no almost one is committed to deliver quality products/services. No one is working on quality, everyone works on new features. There is usually no incentive to increase quality, to improve speed, performance, etc.
- herval 1y ago
  you don't need "high quality engineers" to have high-quality outputs. And vice-versa - lots of places with very high quality engineers produce terribly low-quality software
  - ChrisMarshallNY 1y ago
    I guess we see things differently. They don't need to be especially talented engineers, but, in my experience (and I actually have quite a bit of it, in this area), they need to be dedicated to a culture of Quality. And it is entirely possible for very talented engineers to produce shite. I've seen exactly that.
    - herval 1y ago
      A culture of quality doesn’t require particularly skilled individuals to function. That’s in fact the thesis for the entire Deming management philosophy, and in line with what I’m saying (you can produce high quality with a good process or a good culture, you don’t necessarily need high caliber individuals)
      - ChrisMarshallNY 1y ago
        Japan used a lot of Deming’s theories, to significant success. I worked for a Japanese company. In my case, the company produced absolutely top-shelf stuff, but even relatively mediocre companies did well, using Deming’s techniques. It required that everyone be on board, wrt the culture, though. But I have found that a “good” engineer is one that takes their vocation seriously. They may not be that accomplished or skilled, but they have self-discipline, humility, and structure. I’ve met quite a few highly-skilled “not-good” engineers, in my day. I’m embarrassed to say that I’ve hired some of them.
  - fesenjoonior 1y ago
    > you don't need "high quality engineers" to have high-quality outputs. [citation needed]
- kulahan 1y ago
  Unfortunately, software developers are the absolute most offensive use of the word "engineer", because 99.9% of the stuff this field makes is a competition to take the most unique approach to a solution, then getting it bandaged together with gum and paperclips. If this industry wants to be respected, it should start trying to be actual engineers. There should be tons and tons of standards which are enforced legally, but this is not often the case. Imagine if there were no real legal guardrails in, say, bridge building! edit: and imagine if any time you brought up this issue, bridge builders cockily responded with "well stuff seems to work fine so..."
  - I_dream_of_Gen1 1y ago
    [dead]
  - ChrisMarshallNY 1y ago
    Yeah, this is a fairly classic challenge. When it comes to safety stuff (like bridge building), there are (and should be) strict licensing requirements. I would have no problem requiring such for work on things like medical equipment, We already require security clearance for things like defense information (unless you're a DOGE bro, I guess). That's a bit different from engineering creds, but it's an example of imposed structure. But I think that it would be ridiculous to require it for someone that writes a fart app (unless it's a weaponized fart app). What is in those requirements then becomes a hot potato. There are folks that would insist that any "engineer" be required to know how to use a slide rule, and would ignore modern constructs like LLMs and ML. I'm not kidding. I know people exactly like that. If they get authority, watch out. They'll only "approve" stuff that they are good at. On the other hand, if the requirements are LeetCode, then it's useless. A lot of very unqualified people would easily pass, and wreak havoc. From what I can see, the IEEE seems to have a fairly good grasp on mixing classic structure and current tech. There's some very good people, there, and they are used to working in a structured manner. But software has developed a YOLO culture. People are used to having almost no structure, and they flit between organizations so rapidly, that it's almost impossible to keep track of who is working on what. The entire engineering culture needs to be changed. I don't see that being something that will come easily. I'm big on Structure and Discipline. A lot of it has to do with almost 27 years at a corporation with so much structure that a lot of folks here, would be whimpering under their standing desks. That structure was required, in order to develop equipment of the Quality they are famous for, but would be total overkill for a lot of stuff. I do think that we need to impose structure on software supply chains, though. That's not something that will be a popular stance. Structure is also not cheap. Someone needs to pay for it, and that's when you become a real skunk at the picnic.
f1shy 1y ago
It is a culture. No doubt. And certainly not only processes. I work in a company where there are literally processes for everything. For every engineer doing actual work (requirement engineering, architecture, coding, testing) there are at least 3!doing processes. The SW we make is the ultimative piece of shit, late, expensive, and to the brim full of bugs. So process is importannt, but good engineering, and culture also.

chairmansteve 1y ago

I would say the real lesson is that the Therac machine should have had hardware interlocks (mentioned but not emphasised in the article).

WalterBright 1y ago

I'm going to disagree. I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident. The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case." For the Therac-25, that means a detector of the amount of radiation being generated, which will cut it off if it exceeds a safe value. I'd also add that the radiation generator be physically incapable of generating excessive radiation.

vjvjvjvjghv 1y ago
In general I agree but there is bit more complexity. I work in medical devices and there are plenty of situations where a certain output is ok in some circumstance but deadly in another. That makes a stopgap a little more tricky. I agree with the previous poster that the feedback from the field is lacking a lot. A lot of doctors don’t report problems back because they are used to bad interfaces. And then the feedback gets filtered through several layers of sales reps and product management. So a lot of info gets lost and fixes that could be simple won’t get done. In general when you work in medical you are so overwhelmed by documentation and regulation that there isn’t much time left to do proper engineering. The FDA mostly looks at documentation done right and less at product done right.
- WalterBright 1y ago
  At Boeing there's a required "failure analysis" document listing all the failure modes and why they won't cause a crash by themselves.
  - fsulkner 1y ago
    [dead]
  - Aloha 1y ago
    Agreed - This is essentially the corner stone of systems failure analysis - something I wish architects thought about more in the software space. I'm a product manager for an old (and if I'm being honest somewhat crusty) system of software, the software is buggy, all of it is, but its also self healing and resilient, so while yes, it fails with somewhat alarming regularity, with very lots and lots concerning looking error messages in the logs, but it never causes an outage because it self heals. Good systems design isn't making bug free software or a bug free system, but rather a system where a total outage requires N+1 (maybe even N+N) things to fail before the end user notices. Failures should be driven by at most, edge cases - basically where the system is being operated outside of its design parameters, and those parameters need to reflect the real world and be known by most stakeholders in the system. My gripe with software engineers sometimes, they're often too divorced from real users and real use cases, and too devoted to the written spec over what their users actually need to do with the software - I've seen some very elegant (and on paper, well designed) systems fall apart because if simple things like intermittent packet jitter, or latency swings (say between 10ms and 70ms) - these are real world conditions, often encountered by real world systems, but these spec driven systems fall apart once confronted with reality.

credit_guy 1y ago

I'm not sure. Most software (by orders of magnitude) is not critical software like the software running that X-ray machine. In general, if your software fails, a page loads too slow, or a report comes with lots of NaN's, or some batch job does not run at the right time, and someone needs to start it manually, etc. The cases where someone dies because of a software quality issue are very rare, and the developers working on that type of software know who they are and what their duties are (I hope).

rowanG077 1y ago

I think the opposite. The only reason software quality emerges is because of good developers. It's a prerequisites. Process helps good developers deliver quality. But there is no process that allows a team of bad developers deliver quality. you can't squeeze blood from a stone.

benrutter 1y ago
That's true, but even so - great developers still make mistakes, and if they don't hear about production errors because of a breakdown in the customer communications from sales etc, then those mistakes will never be fixed. It's not that great developers aren't necessary for software quality, more that they aren't sufficient.
- rowanG077 1y ago
  I think great developers are necessary for software quality. But indeed often not sufficient.

msy 1y ago

I couldn't disagree more. Outside of exotic scenarios where things like formal proofing are possible and economically viable I've never seen a process that prevents bugs, only culture. Good engineering cultures are also often ones with well defined and tested processes and good testing practices but it's the culture and people giving a shit that makes the difference, not the other way around. Good product cultures are ones where natural communication between the field and engineering would mean issues get reported back up and make their way to the right people. No process will compensate for people not giving a shit.

benrutter 1y ago
> It's the end result of a process, and that process informs both your software development practices, but also your testing. Your management. Even your sales and servicing. I think the bit I quoted, especially if you read in the context of the article, is talking about culture. I.e. it's talking about a process that informs software development, management and sales. Things like formal proofing and type systems are the exact kind of processes that aren't what it's talking about. I kind of agree with you though about the process/culture distinction - ultimately, if you don't have a culture where people actively care about improving reliability, any process is just gonna become a tick-box exercise to appease management.

AdamN 1y ago

This is true but there also needs to be good developers as well. It can't just be great process and low quality developer practices. There needs to be: 1/ high quality individual processes (development being one of them), 2/ high quality delivery mechanisms, 3/ feedback loops to improve that quality, 4/ out of band mechanisms to inspect and improve the quality.

Fr3dd1 1y ago
I would argue that a good process always has a good self correction mechanism built in. This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process.
- franktankbank 1y ago
  The process that makes this work would be so onerous to create. Would you think you could do this to make a low quality machinist be able to build a high quality technical part? What would this look like? Quite a lot like machine code which doesn't really reduce the requirements does it? It actually just shifted the onerous requirement somewhere else.
- vjvjvjvjghv 1y ago
  “ This way, the work done by a "low quality" software developer (this includes almost all of us at some point in time), is always taken into account by the process” That’s a horrible take. There is no amount of reviews, guidelines and documentation that can compensate for low quality devs. You can’t throw garbage into the pipeline and then somehow process it to gold.
- Cthulhu_ 1y ago
  > (this includes almost all of us at some point in time) I'd say this includes all of us all the time; a good developer never trusts their own work blindly, and spends more time gathering requirements and verifying their and others' work than writing code.
- quietbritishjim 1y ago

deleted 1y ago

[deleted]

vorgol 1y ago

I was going to recommend that exact podcast episode but you beat me to it. Totally worth listening, especially if you're interested in software bugs. Another interesting fact mentioned in the podcast is that the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. Excellent demonstration of the Swiss Cheese Model: https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model

ipython 1y ago
Don’t worry we are poised to re learn all these lessons once again with our fancy new agentic generative ai systems. The mechanical interlock essentially functioned as a limit outside of the control system. So you should build an ai system the same way- enforcing restrictions on the security agency from outside the control of the ai itself. Of course that doesn’t happen and devs naively trust that the ai can make its own security decisions. Another lesson from that era we are re learning- in-band signaling. Our 2025 version of the “blue box” is in full swing. Prompt injection is just a side effect of the fact that there is no out of band instruction mechanism for llms. Good news is - it’s not hard to learn the new technology when it’s just a matter of rediscovering the same security issues with a new name!
bell-cot 1y ago
>> the real failure in the story of the Therac-25 from my understanding, is that it took far too long for incidents to be reported, investigated and fixed. > the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. #1 virtue of electromechanical failsafes is that their conception, design, implementation, and failure modes tend to be orthogonal to those of the software. One of the biggest shortcomings of Swiss Cheese safety thinking is that you too-often end up using "neighbor slices from the same wheel of cheese". #2 virtue of electromechanical failsafes is that running into them (the fuse blew, or whatever) is usually more difficult for humans to ignore. Or at least it's easier to create processes and do training that actually gets the errors reported up the chain. (Compared to software - where the worker bees all know you gotta "ignore, click 'OK', retry, reboot" all the time, if you actually want to get anything done): But, sadly, electromechanical failsafes are far more expensive then "we'll just add some code to check that" optimism. And PHB's all know that picking up nickles in front of the steamroller is how you get to the C-suite.

sonicggg 1y ago

Not sure why the article is focusing so much on software development. That was just a piece of the problem. The entire product had design flaws. When the FDA for involved, the company wasn't just told to make software updates.

speed_spread 1y ago
Yet It doesn't take much to swamp a team of good developers. A poorly defined project, mismatched requirements, sent to production too early and then put in support mode with no time planned to plug the holes... There's only so much smart technicians can do when the organization is broken.

aaron695 1y ago

[dead]

0xDEAFBEAD 1y ago

Honestly I wish instead of the Therac-25, we were discussing a system which made use of unit testing and defensive coding, yet still failed. That would be more educational. It's too easy to look at the Therac-25 and think "I would never write a mess like that".

roeles 1y ago
One instance that crosses my mind often is the airbus a320 incident at Hamburg in 2008. Everything was done right there, but the requirements were wrong. Despite all the procedures and tests, the software still managed to endanger the lives of the passengers.
- 0xDEAFBEAD 1y ago
  Interesting, do you happen to have a case study?
  - deleted 1y ago
    [deleted]
  - shmeeed 1y ago
    https://skybrary.aero/sites/default/files/bookshelf/1258.pdf https://skybrary.aero/sites/default/files/bookshelf/1258.pdf
- I_dream_of_Geni 1y ago
  Speaking of Airbus, They 'lost' 3-4 different aircraft (from 1988 to 2015) which crashed during development, or, spectacularly during their first airshow. Never slowed down their customers at ALL, and to this day, Boeing has never lost one new commercial airliner in those same circumstances. Yet, Boeing gets all the hate. smh
- Izkata 1y ago
  The Boeing 737 MAX had an additional safety feature that was causing crashes due to bad input from the sensors, that pilots didn't know about so they couldn't override. This was 2018 and 2019. After the first crash, the manuals and training were updated to explain what was going on and how to override it. https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Augmentation_System https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Au...

pjmlp 1y ago

The worst part is that many devlopers think that by not working with high integrity systems, such quality levels don't apply to them. Wrong, any software failure can have huge consequences in someone's life, or company, by preventing some critical flow to take place, corrupting data related to someone's life, professional or medical record, preventing a payment on some specific goods that had to be acquired on that moment or never,....

ozim 1y ago
Hey don’t blame developers. It is business who requests features ASAP to cut costs and and then there are customers who don’t want to pay for „ideal software” but rather have every software for free. Most devs and QA workers I know want to deliver best quality software and usually are gold plating stuff anyway.
- pjmlp 1y ago
  Being a real Software Engineer, those that actually have the proper title, eventually with the final examination, means being able to deliver the best product within the set of given constraints. Also, speaking out when the train is visibly going against a wall.
  - bobmcnamara 1y ago
    In my country those are a myth. We had it as a professional engineering classification for a while but I'm not sure if anyone ever completed it. They cancelled it several years ago.
    - pjmlp 1y ago
      And in US anyone can call themselves whatever they feel like. If you want professional quality, we're the first line of actually making it happen, blaming others won't change anything.
      - 1y ago