3 ms·

I'm surprised no one has mentioned Therac 25 or Normal Accidents yet. For reference, the Therac 25 was a computer-controlled radiation therapy machine involved

by csours 7y ago

I'm surprised no one has mentioned Therac 25 or Normal Accidents yet. For reference, the Therac 25 was a computer-controlled radiation therapy machine involved in several over-exposures due to replacement of physical controls with computer based ones without complete understanding of the interactions of the controls. The Max feels very much like that. No one can really keep a whole aircraft in their head, much less a whole aircraft development project. We use computers for that, as well as mental heuristics. But if those computers and brains are not fed all the proper data and connections, they will not find the all the problems. Additionally, there seems to be a lot of the tail wagging the dog. If this system is expected to perform according to X specifications, then by golly it will, and we will show that it does. https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 Edit: Please don't take the above as absolution of Boeing. Someone (a lot of someones) really should have known better.

mehrdadn 7y ago
I don't see these as equivalent, at least not based on what I've learned about the cases (feel free to correct). As I understand, Therac-25 was due to software bug and a genuine design process inadequacy that allowed it to cause a problem, that could happen with people acting entirely in good faith, simply because they didn't know better. That's why they created standards to address the design process. With 737MAX... pretty much literally everyone could tell you the decisions were bad, and so many seem to have been made in bad faith, specifically to e.g. avoid recertification and increase revenues in a pretty reckless manner.
- csours 7y ago
  It's not the equivalent, but it is the consequence of a chain of incremental changes, each of which is not sufficient to subvert safety margins, but together they change the paradigm. As to bad faith, yes, I'm sure there was some of that, but generally decisions like these don't look like bad faith to the people making them. It's easy to get swamped by technical details.
  - mehrdadn 7y ago
    > consequence of a chain of incremental changes > As to bad faith, yes, I'm sure there was some of that You're downplaying this. This is not downplayable, and this is not similar to Therac. "A consequence of incremental changes" is a rather gross way to paint this, as if it's hard for a single guy to see how e.g. non-redundant sensors is an extremely bad idea on its own, let alone everything else. There have been multiple huge missteps, not made by accident, each of which is individually worth a huge red flags obvious to people in different areas. That 1 single mistake wasn't enough to bring the plane down doesn't mean the mistakes must've been small or somehow downplayable. And no, this isn't some kind of gray area with people getting genuinely swamped by technical details. It's abundantly clear here's been a ton of bad faith here, that there is still ongoing bad faith even after the fact, and that they're still unwilling to address the problem properly.
    - csours 7y ago
      I'm sorry that it seemed like I was downplaying it. What I was actually thinking is that this kind of thing is likely to crop up in complex systems, and if we work on complex systems we should be wary for it.

by csours 7y ago

mehrdadn 7y ago

I don't see these as equivalent, at least not based on what I've learned about the cases (feel free to correct). As I understand, Therac-25 was due to software bug and a genuine design process inadequacy that allowed it to cause a problem, that could happen with people acting entirely in good faith, simply because they didn't know better. That's why they created standards to address the design process. With 737MAX... pretty much literally everyone could tell you the decisions were bad, and so many seem to have been made in bad faith, specifically to e.g. avoid recertification and increase revenues in a pretty reckless manner.

csours 7y ago
It's not the equivalent, but it is the consequence of a chain of incremental changes, each of which is not sufficient to subvert safety margins, but together they change the paradigm. As to bad faith, yes, I'm sure there was some of that, but generally decisions like these don't look like bad faith to the people making them. It's easy to get swamped by technical details.
- mehrdadn 7y ago
  > consequence of a chain of incremental changes > As to bad faith, yes, I'm sure there was some of that You're downplaying this. This is not downplayable, and this is not similar to Therac. "A consequence of incremental changes" is a rather gross way to paint this, as if it's hard for a single guy to see how e.g. non-redundant sensors is an extremely bad idea on its own, let alone everything else. There have been multiple huge missteps, not made by accident, each of which is individually worth a huge red flags obvious to people in different areas. That 1 single mistake wasn't enough to bring the plane down doesn't mean the mistakes must've been small or somehow downplayable. And no, this isn't some kind of gray area with people getting genuinely swamped by technical details. It's abundantly clear here's been a ton of bad faith here, that there is still ongoing bad faith even after the fact, and that they're still unwilling to address the problem properly.
  - csours 7y ago
    I'm sorry that it seemed like I was downplaying it. What I was actually thinking is that this kind of thing is likely to crop up in complex systems, and if we work on complex systems we should be wary for it.