4 ms·

There you have it, folks, programmers killing people. No, it wasn't on-purpose, but that doesn't mean it didn't happen. This, of course, has been happening for

There you have it, folks, programmers killing people. No, it wasn't on-purpose, but that doesn't mean it didn't happen. This, of course, has been happening for a long time, it's just getting more and more obvious now. I feel strongly that AF 447 was an example. It's important to understand that this is the natural result of two trends. One, software/hardware systems continue to become more and more complex, many times leading to unforeseen interactions. Two, programmers and programming management teams love to bunch up the work by things not related to the end-users: hardware component, new modules, architectural tier, underlying technology involved. It doesn't take very much of that at all before you have a whole bunch of people all doing work of various quality in small enough isolated silos such that everybody can do a wonderful job -- and the system still kill people. There's a lot more to add here. It's nothing new, and there are a ton of ways we've tried to fix this. My belief is that we've failed because we don't understand exactly what goes into making successful software at scale. I am quite concerned about it and I have been writing on this topic for several years. If you're interested in more, see my profile.

carbocation 8y ago
Why is this a software problem? Even if there is a software “fix,” doesn’t it seem that this tells us more about regulatory capture than using software to patch over bad hardware decisions?
- carlmr 8y ago
  It's a bad mechanical design, Boeing didn't want to spend money on retraining pilots, they asked software to fix it, software fixed it so that it remained the same, software had a bug, training didn't incorporate new system, because that was the point of the software, to not require a new type rating. I think this is process failure from start to finish. Any sufficiently complex system will have bugs in it, whether it's software or not. I think the other failures are much worse. You have to account for your system maybe failing and your pilots have to be trained to deal with this failure. And yes, software kills people, but no software kills even more people. Just look at how airplanes, cars etc. have improved safety through assistance systems.
  - tluyben2 8y ago
    It is bad mechanical design and the planes need to be renamed and rerated imho. That is now it should be fixed. It will cost Boeing 100s of millions but they deserve that by now.
- DanielBMarkham 8y ago
  I really didn't want to flame the room. As a commercially-rated small plane pilot and coder, I have a passion here and it both hurts me and makes me really angry. I said this was a programmers problem, not a software problem, and I identified one of the key causes as programmers being isolated from user outcomes. Probably not a big deal if your user can't get to the website for 10 minutes. Different thing entirely here. So this is a complexity * project organization problem. Yep, I think there's regulatory capture, and it may or may not have added to the problem. But you could repeat the factors I've listed in any country or industry you like and get the same results. So this isn't a "Boeing has corrupted the FAA" thing. Sure, it might have, and that's a good thing to talk about. But that's not it. In fact, if you look at it that way, you're just continuing to silo off responsibility. This is another way of saying "Let's create yet another component that has oversight of all these other components that interact in ways we can't predict so that it can be the arbiter of what's safe or not" That's not a bad idea per se. It becomes bad because "the arbiter of what's safe or not" can mean all kinds of freaking things, many of which may be inconsistent or missing. So really it just makes it worse. Let's say you make changes A, B, and C. Then it happens again with other systems. We come back in five years with another crash and sure enough, we've got the same clusterfuck as everybody points at one another saying that they've done their job but the other guy screwed up. Is it possible to black box test extremely complex systems? A lot of engineers have pretended that for a long time that you could. We are reaching the limit of how politically plausible that is. I don't have an easy answer to the question "What do we do right now as programmers?" but I know that programmers, as a group of people, are in a unique spot to understand this problem and try to get attention for it. A lot of us have worked on subsystems that eventually got used in ways we might find horrible and immoral. In that case you make a judgment call on whether the overall good beats the damage you may cause. In this case, however, these components and systems were built and integrated for no other purpose than this particular airplane. And bad things are happening. Joe Programmer can't be responsible for fixing Boeing, but he sure as hell can be responsible for having Boeing and other entities that build complex systems understand the horrible results of what they're doing and why. ADD: Look at the comments here. It's a software thing. It's a hardware thing. Yesterday it was a training thing. A few days ago it was a distracted crew. Now we're talking about regulatory capture. Systemic problems are problems such that each piece can work in good faith and do a good job -- yet the overall results of the system gets worse and worse. A good sign of a systemic problem is when intelligent, well-meaning people end up pointing fingers all over the place. This is not a programming problem because it's in the program. It's a programming problem because we're in a spot to understand what got us here -- and I believe we have a moral responsibility to help others understand. A second sign is when new components are suggested as answers to problems the existing components are manifesting.
boudin 8y ago
It's to early to conclude it's only a software bug. And blaming programmers is the absolutely not the way to avoid issues. Humans makes errors, always. Programmers or not. Some errors are unfortunately dramatic (talk about this to surgeons for example). Now to remove the human factor as much as possible, there is processes and engineering techniques. If the failure was indeed caused by the software, and not by other defects in the plane design, the real question is how could it pass any safety check. For me, in any scenario, the first really wrong thing is how the plane passed all safety checks without the defect being detected, internally at Boeing as well as externally. As more information unveil, it seems that the prime suspect is the pressure to release a plane too quickly.
tluyben2 8y ago
It is bad hardware design with a software bandaid rushed in to maximize profits. I agree with you though; I have said for many years that the way we write software (move fast, break things and shittier languages and infra like js and node) will eventually kill more people than most other man made disasters. Simply because the push for cheaper juniors in places where they do not belong. Less bad but still potentially life altering I see every day in healthcare, insurance and payments: software is so horribly patched together that money just gets lost or misdirected. I am biased as I was raised by a pupil of Dijkstra and studied under 2 of them as well, but for me there are solutions; tla+, Coq and languages with static typing. Not because these are silver bullets in their own right, but more because you cannot ‘move fast, break things’ in them; you have to think and plan a lot more as refactoring is very hard. Companies like Amazon uses tla+ a lot so I wonder why it is not standard practice in any place where lives are affected (aviation but also money and healthcare).
Pyxl101 8y ago
How can we be sure that this is programmers killing people, and not merely a busted angle of attack sensor, or an inadequate specification for how the plane should be designed (number of sensors or sensor robustness) or how the software should behave? This conclusion feels premature.
Slartie 8y ago
I am pretty sure the software here did exactly what it was supposed to do. There was no buffer overrun, no NullPointerException, not any other kind of bug. There was just software that reacted exactly as specified. The problem is: those specifications were bad. It's a bad idea to just rely on one single sensor when you actually have two, it's a bad idea to write software that disengages on manual input counteracting its decisions, then resume overriding pilot decisions out of the blue with a delay of a few seconds again. And it's an absurdly bad idea to do all of this while not telling the humans that will have to operate the machinery in the end ANYTHING about how all of this works. I totally get why Boeing thought all of those obviously bad ideas were "good" - because they helped their bottom line. It wouldn't be possible to completely hide the existence of the MCAS if there was a potential for a sensor disagreement that has to be solved manually by pilots, and if it hadn't been possible to completely hide the existence of MCAS, it wouldn't have been possible to get the new plane out on the market without requiring expensive re-training. And that would have made a very convenient marketing claim of Boeing impossible to make: "this new plane can be flown by all those pilots familiar with the old plane, no re-training necessary, just read these few pages on an iPad and you're good!". But this clearly makes this disaster not a "programmers killing people" incident. If any "xxx killing people" classification is to be made here, it's "managers killing people".
- kharms 8y ago
  Bad programming was not the sole or root cause but it was a cause, and people died. You can't punt on that unless you want software engineer to actually mean code monkey. The same way a building architect has to sign off on the design demands of a a client, a software engineer on safety-critical systems must evaluate the spec and consider the effects of edge cases.
  - peteradio 8y ago
    What do you suppose the spec was here and how did the Engineer neglect to fulfill it? These specs are very compartmentalized, you think the Engineer is responsible for the big picture understanding and gets to take it up with the VP? No.
CloudNetworking 8y ago
> There you have it, folks, programmers killing people In fact it is more like "unscrupulous capitalists killing people", though. The more we know about the incident the clearer it becomes.
ajuc 8y ago
> There you have it, folks, programmers killing people. Programmers were killing people since before we had electronic computers. IBM did punch-card system for mechanical computers for managing people in nazi death camps.
deleted 8y ago
[deleted]

carbocation 8y ago

Why is this a software problem? Even if there is a software “fix,” doesn’t it seem that this tells us more about regulatory capture than using software to patch over bad hardware decisions?

carlmr 8y ago
It's a bad mechanical design, Boeing didn't want to spend money on retraining pilots, they asked software to fix it, software fixed it so that it remained the same, software had a bug, training didn't incorporate new system, because that was the point of the software, to not require a new type rating. I think this is process failure from start to finish. Any sufficiently complex system will have bugs in it, whether it's software or not. I think the other failures are much worse. You have to account for your system maybe failing and your pilots have to be trained to deal with this failure. And yes, software kills people, but no software kills even more people. Just look at how airplanes, cars etc. have improved safety through assistance systems.
- tluyben2 8y ago
  It is bad mechanical design and the planes need to be renamed and rerated imho. That is now it should be fixed. It will cost Boeing 100s of millions but they deserve that by now.
DanielBMarkham 8y ago
I really didn't want to flame the room. As a commercially-rated small plane pilot and coder, I have a passion here and it both hurts me and makes me really angry. I said this was a programmers problem, not a software problem, and I identified one of the key causes as programmers being isolated from user outcomes. Probably not a big deal if your user can't get to the website for 10 minutes. Different thing entirely here. So this is a complexity * project organization problem. Yep, I think there's regulatory capture, and it may or may not have added to the problem. But you could repeat the factors I've listed in any country or industry you like and get the same results. So this isn't a "Boeing has corrupted the FAA" thing. Sure, it might have, and that's a good thing to talk about. But that's not it. In fact, if you look at it that way, you're just continuing to silo off responsibility. This is another way of saying "Let's create yet another component that has oversight of all these other components that interact in ways we can't predict so that it can be the arbiter of what's safe or not" That's not a bad idea per se. It becomes bad because "the arbiter of what's safe or not" can mean all kinds of freaking things, many of which may be inconsistent or missing. So really it just makes it worse. Let's say you make changes A, B, and C. Then it happens again with other systems. We come back in five years with another crash and sure enough, we've got the same clusterfuck as everybody points at one another saying that they've done their job but the other guy screwed up. Is it possible to black box test extremely complex systems? A lot of engineers have pretended that for a long time that you could. We are reaching the limit of how politically plausible that is. I don't have an easy answer to the question "What do we do right now as programmers?" but I know that programmers, as a group of people, are in a unique spot to understand this problem and try to get attention for it. A lot of us have worked on subsystems that eventually got used in ways we might find horrible and immoral. In that case you make a judgment call on whether the overall good beats the damage you may cause. In this case, however, these components and systems were built and integrated for no other purpose than this particular airplane. And bad things are happening. Joe Programmer can't be responsible for fixing Boeing, but he sure as hell can be responsible for having Boeing and other entities that build complex systems understand the horrible results of what they're doing and why. ADD: Look at the comments here. It's a software thing. It's a hardware thing. Yesterday it was a training thing. A few days ago it was a distracted crew. Now we're talking about regulatory capture. Systemic problems are problems such that each piece can work in good faith and do a good job -- yet the overall results of the system gets worse and worse. A good sign of a systemic problem is when intelligent, well-meaning people end up pointing fingers all over the place. This is not a programming problem because it's in the program. It's a programming problem because we're in a spot to understand what got us here -- and I believe we have a moral responsibility to help others understand. A second sign is when new components are suggested as answers to problems the existing components are manifesting.

boudin 8y ago

It's to early to conclude it's only a software bug. And blaming programmers is the absolutely not the way to avoid issues. Humans makes errors, always. Programmers or not. Some errors are unfortunately dramatic (talk about this to surgeons for example). Now to remove the human factor as much as possible, there is processes and engineering techniques. If the failure was indeed caused by the software, and not by other defects in the plane design, the real question is how could it pass any safety check. For me, in any scenario, the first really wrong thing is how the plane passed all safety checks without the defect being detected, internally at Boeing as well as externally. As more information unveil, it seems that the prime suspect is the pressure to release a plane too quickly.

tluyben2 8y ago

It is bad hardware design with a software bandaid rushed in to maximize profits. I agree with you though; I have said for many years that the way we write software (move fast, break things and shittier languages and infra like js and node) will eventually kill more people than most other man made disasters. Simply because the push for cheaper juniors in places where they do not belong. Less bad but still potentially life altering I see every day in healthcare, insurance and payments: software is so horribly patched together that money just gets lost or misdirected. I am biased as I was raised by a pupil of Dijkstra and studied under 2 of them as well, but for me there are solutions; tla+, Coq and languages with static typing. Not because these are silver bullets in their own right, but more because you cannot ‘move fast, break things’ in them; you have to think and plan a lot more as refactoring is very hard. Companies like Amazon uses tla+ a lot so I wonder why it is not standard practice in any place where lives are affected (aviation but also money and healthcare).

Pyxl101 8y ago

How can we be sure that this is programmers killing people, and not merely a busted angle of attack sensor, or an inadequate specification for how the plane should be designed (number of sensors or sensor robustness) or how the software should behave? This conclusion feels premature.

Slartie 8y ago

I am pretty sure the software here did exactly what it was supposed to do. There was no buffer overrun, no NullPointerException, not any other kind of bug. There was just software that reacted exactly as specified. The problem is: those specifications were bad. It's a bad idea to just rely on one single sensor when you actually have two, it's a bad idea to write software that disengages on manual input counteracting its decisions, then resume overriding pilot decisions out of the blue with a delay of a few seconds again. And it's an absurdly bad idea to do all of this while not telling the humans that will have to operate the machinery in the end ANYTHING about how all of this works. I totally get why Boeing thought all of those obviously bad ideas were "good" - because they helped their bottom line. It wouldn't be possible to completely hide the existence of the MCAS if there was a potential for a sensor disagreement that has to be solved manually by pilots, and if it hadn't been possible to completely hide the existence of MCAS, it wouldn't have been possible to get the new plane out on the market without requiring expensive re-training. And that would have made a very convenient marketing claim of Boeing impossible to make: "this new plane can be flown by all those pilots familiar with the old plane, no re-training necessary, just read these few pages on an iPad and you're good!". But this clearly makes this disaster not a "programmers killing people" incident. If any "xxx killing people" classification is to be made here, it's "managers killing people".

kharms 8y ago
Bad programming was not the sole or root cause but it was a cause, and people died. You can't punt on that unless you want software engineer to actually mean code monkey. The same way a building architect has to sign off on the design demands of a a client, a software engineer on safety-critical systems must evaluate the spec and consider the effects of edge cases.
- peteradio 8y ago
  What do you suppose the spec was here and how did the Engineer neglect to fulfill it? These specs are very compartmentalized, you think the Engineer is responsible for the big picture understanding and gets to take it up with the VP? No.

CloudNetworking 8y ago

> There you have it, folks, programmers killing people In fact it is more like "unscrupulous capitalists killing people", though. The more we know about the incident the clearer it becomes.

ajuc 8y ago

> There you have it, folks, programmers killing people. Programmers were killing people since before we had electronic computers. IBM did punch-card system for mechanical computers for managing people in nazi death camps.

deleted 8y ago

[deleted]