23 ms·

Why is this a software problem? Even if there is a software “fix,” doesn’t it seem that this tells us more about regulatory capture than using software to patch

Why is this a software problem? Even if there is a software “fix,” doesn’t it seem that this tells us more about regulatory capture than using software to patch over bad hardware decisions?

carlmr 8y ago
It's a bad mechanical design, Boeing didn't want to spend money on retraining pilots, they asked software to fix it, software fixed it so that it remained the same, software had a bug, training didn't incorporate new system, because that was the point of the software, to not require a new type rating. I think this is process failure from start to finish. Any sufficiently complex system will have bugs in it, whether it's software or not. I think the other failures are much worse. You have to account for your system maybe failing and your pilots have to be trained to deal with this failure. And yes, software kills people, but no software kills even more people. Just look at how airplanes, cars etc. have improved safety through assistance systems.
- tluyben2 8y ago
  It is bad mechanical design and the planes need to be renamed and rerated imho. That is now it should be fixed. It will cost Boeing 100s of millions but they deserve that by now.
DanielBMarkham 8y ago
I really didn't want to flame the room. As a commercially-rated small plane pilot and coder, I have a passion here and it both hurts me and makes me really angry. I said this was a programmers problem, not a software problem, and I identified one of the key causes as programmers being isolated from user outcomes. Probably not a big deal if your user can't get to the website for 10 minutes. Different thing entirely here. So this is a complexity * project organization problem. Yep, I think there's regulatory capture, and it may or may not have added to the problem. But you could repeat the factors I've listed in any country or industry you like and get the same results. So this isn't a "Boeing has corrupted the FAA" thing. Sure, it might have, and that's a good thing to talk about. But that's not it. In fact, if you look at it that way, you're just continuing to silo off responsibility. This is another way of saying "Let's create yet another component that has oversight of all these other components that interact in ways we can't predict so that it can be the arbiter of what's safe or not" That's not a bad idea per se. It becomes bad because "the arbiter of what's safe or not" can mean all kinds of freaking things, many of which may be inconsistent or missing. So really it just makes it worse. Let's say you make changes A, B, and C. Then it happens again with other systems. We come back in five years with another crash and sure enough, we've got the same clusterfuck as everybody points at one another saying that they've done their job but the other guy screwed up. Is it possible to black box test extremely complex systems? A lot of engineers have pretended that for a long time that you could. We are reaching the limit of how politically plausible that is. I don't have an easy answer to the question "What do we do right now as programmers?" but I know that programmers, as a group of people, are in a unique spot to understand this problem and try to get attention for it. A lot of us have worked on subsystems that eventually got used in ways we might find horrible and immoral. In that case you make a judgment call on whether the overall good beats the damage you may cause. In this case, however, these components and systems were built and integrated for no other purpose than this particular airplane. And bad things are happening. Joe Programmer can't be responsible for fixing Boeing, but he sure as hell can be responsible for having Boeing and other entities that build complex systems understand the horrible results of what they're doing and why. ADD: Look at the comments here. It's a software thing. It's a hardware thing. Yesterday it was a training thing. A few days ago it was a distracted crew. Now we're talking about regulatory capture. Systemic problems are problems such that each piece can work in good faith and do a good job -- yet the overall results of the system gets worse and worse. A good sign of a systemic problem is when intelligent, well-meaning people end up pointing fingers all over the place. This is not a programming problem because it's in the program. It's a programming problem because we're in a spot to understand what got us here -- and I believe we have a moral responsibility to help others understand. A second sign is when new components are suggested as answers to problems the existing components are manifesting.
- carbocation 8y ago
  I agree with much of what you say here. I don’t agree that programmers are at fault or are particularly well suited or intervene, since the core is not a software problem. This is the nature of systems problems. Nevertheless, I think that we have a core agreement.
  - DanielBMarkham 8y ago
    I wrote this two years ago -- the first chapter of a book (which I'm trying really hard _not_ to pump here) https://wiki.info-ops.org/index.php/Main_Page#Info-Ops https://wiki.info-ops.org/index.php/Main_Page#Info-Ops There's a link to the first ten pages. I find it difficult to write an intro like that and then see this happen again.

Why is this a software problem? Even if there is a software “fix,” doesn’t it seem that this tells us more about regulatory capture than using software to patch over bad hardware decisions?

carlmr 8y ago

It's a bad mechanical design, Boeing didn't want to spend money on retraining pilots, they asked software to fix it, software fixed it so that it remained the same, software had a bug, training didn't incorporate new system, because that was the point of the software, to not require a new type rating. I think this is process failure from start to finish. Any sufficiently complex system will have bugs in it, whether it's software or not. I think the other failures are much worse. You have to account for your system maybe failing and your pilots have to be trained to deal with this failure. And yes, software kills people, but no software kills even more people. Just look at how airplanes, cars etc. have improved safety through assistance systems.

tluyben2 8y ago
It is bad mechanical design and the planes need to be renamed and rerated imho. That is now it should be fixed. It will cost Boeing 100s of millions but they deserve that by now.

DanielBMarkham 8y ago

I really didn't want to flame the room. As a commercially-rated small plane pilot and coder, I have a passion here and it both hurts me and makes me really angry. I said this was a programmers problem, not a software problem, and I identified one of the key causes as programmers being isolated from user outcomes. Probably not a big deal if your user can't get to the website for 10 minutes. Different thing entirely here. So this is a complexity * project organization problem. Yep, I think there's regulatory capture, and it may or may not have added to the problem. But you could repeat the factors I've listed in any country or industry you like and get the same results. So this isn't a "Boeing has corrupted the FAA" thing. Sure, it might have, and that's a good thing to talk about. But that's not it. In fact, if you look at it that way, you're just continuing to silo off responsibility. This is another way of saying "Let's create yet another component that has oversight of all these other components that interact in ways we can't predict so that it can be the arbiter of what's safe or not" That's not a bad idea per se. It becomes bad because "the arbiter of what's safe or not" can mean all kinds of freaking things, many of which may be inconsistent or missing. So really it just makes it worse. Let's say you make changes A, B, and C. Then it happens again with other systems. We come back in five years with another crash and sure enough, we've got the same clusterfuck as everybody points at one another saying that they've done their job but the other guy screwed up. Is it possible to black box test extremely complex systems? A lot of engineers have pretended that for a long time that you could. We are reaching the limit of how politically plausible that is. I don't have an easy answer to the question "What do we do right now as programmers?" but I know that programmers, as a group of people, are in a unique spot to understand this problem and try to get attention for it. A lot of us have worked on subsystems that eventually got used in ways we might find horrible and immoral. In that case you make a judgment call on whether the overall good beats the damage you may cause. In this case, however, these components and systems were built and integrated for no other purpose than this particular airplane. And bad things are happening. Joe Programmer can't be responsible for fixing Boeing, but he sure as hell can be responsible for having Boeing and other entities that build complex systems understand the horrible results of what they're doing and why. ADD: Look at the comments here. It's a software thing. It's a hardware thing. Yesterday it was a training thing. A few days ago it was a distracted crew. Now we're talking about regulatory capture. Systemic problems are problems such that each piece can work in good faith and do a good job -- yet the overall results of the system gets worse and worse. A good sign of a systemic problem is when intelligent, well-meaning people end up pointing fingers all over the place. This is not a programming problem because it's in the program. It's a programming problem because we're in a spot to understand what got us here -- and I believe we have a moral responsibility to help others understand. A second sign is when new components are suggested as answers to problems the existing components are manifesting.

carbocation 8y ago
I agree with much of what you say here. I don’t agree that programmers are at fault or are particularly well suited or intervene, since the core is not a software problem. This is the nature of systems problems. Nevertheless, I think that we have a core agreement.
- DanielBMarkham 8y ago
  I wrote this two years ago -- the first chapter of a book (which I'm trying really hard _not_ to pump here) https://wiki.info-ops.org/index.php/Main_Page#Info-Ops https://wiki.info-ops.org/index.php/Main_Page#Info-Ops There's a link to the first ten pages. I find it difficult to write an intro like that and then see this happen again.