4 ms·
People are responsible for their actions. There were many things wrong with this project, but one of them was fatally bad code, which this guy took money to wri
by wpietri 7y ago
People are responsible for their actions. There were many things wrong with this project, but one of them was fatally bad code, which this guy took money to write. Having him be named strikes me as the very minimum in accountability.
I think it's also important to name him, to interview him. To understand how he came to kill. So that anybody writing life-critical code today can say, "I'd better not end up infamous like that guy. I can't make the same mistakes."
Maybe if he had been held accountable, the programmers at Uber wouldn't have been lax enough to code up the negligent homicide of Elaine Herzberg.
- adrianN 7y agoA culture of blame leads to programmers of safety critical software hiding their mistakes.
- Jwarder 7y agoI can see why you would say that, but isn't classical engineering a counter example? Professional engineers have a culture of accountability and responsibility. No system is perfect, but (to my eyes at least) classical engineering works safely without a culture of blame.
- mcguire 7y agoA culture of refusing responsibility is better how?
- adrianN 7y agoThe opposite of a blame culture is not a culture of refusing responsibility, but a system where you look for problems in the process when a fault makes it into the final product. It's always a mixture of specification, implementation, and QA that makes mistakes possible. Putting the blame on the engineer is not very helpful, it does little to prevent future problems. Trying to figure out how it was possible that an implementation bug made it through QA allows you to prevent similar mistakes in the future (at least if you're lucky).
- wpietri 7y agoJust to be clear, I'm not putting the blame on the engineer. Many people are responsible for this failure. What I'm saying is that the engineer is one of the responsible people, and that as fellow engineers we should be especially concerned with making sure that responsibility is correctly performed and has appropriate accountability.
- wpietri 7y agoOn a project? Sure. I'm a big fan of blameless retrospectives as a way to maximize quality. But that only works in the context of a quality-oriented culture that is strongly focused on continually improving safety. However, a) none of that was true here, and b) in the wider societal frame, I think blamenessless must not always trump accountability. When we're talking about egregious negligence leading to death, I think naming the culprits is the very least we should do.
- curmudgeon48 7y agoPreferable to the "mistakes were made" fog bank of diffuse nonblame. People respond to incentives whether cash or heads on pikes.
- toomanybeersies 7y agoObviously we don't know who wrote the code, so we can't ask him, but did he even know the consequences of what he was writing? Was he even aware of the potential lethality? The very fact that we don't know who wrote the code is exactly why it doesn't matter. The complete lack of traceability and accountability is what caused these deaths. There's so many things that went wrong in order for Therac 25 to kill people that it's irrelevant who wrote the code. It could've just as easily been you or me.
- crispyambulance 7y ago> Was he even aware of the potential lethality? It could be worse than that. He might not have even known the actual purpose and complete function of the system for which he was writing code. That's common when big projects get outsourced. One might call it "decomposing the problem into small tasks" but it ends up with people writing code for machinery that they have no knowledge of. I've seen this myself. These kinds of projects depend upon layers and layers of oversight and process. I think that the folks who are looking for "a perpetrator" are utterly missing the key-value of studying the Therac-25 case study.
- wpietri 7y agoI am personally not looking for "a perpetrator". Many people share responsibility for a failure like this. But that's all the more reason that we should clearly describe who did what, when, and why. Please note that you are working very hard to make up a fantasy situation that entirely exonerates the person like you. Your fantasy does not match the known facts. Think hard about why you think that's the most important thing for you to do here.
- wpietri 7y agoHe knew or should have known the potential consequences of the code he was writing. Maybe this could have been you, but it definitely wouldn't have been me, because I would not have committed the first line of code until I was reasonably sure that a) I was not going to kill somebody accidentally, and b) the project had sufficient safeguards. I'd also like you to note that you're pointing out the lack of accountability as a problem, and then dismissing one of the fundamental mechanisms of accountability: naming people who harm others. Yes, other people on the project also should have pushed for traceability and accountability. But responsibility is not zero sum. "I was just following orders" is never an excuse for killing people.
- edoo 7y agoWow I hope you are never on a jury. You get both completely reversed. For uber the worthless lady who was hired to sit behind the wheel to take over as needed was texting. The death is entirely attributable to her failing to perform her job duties. The person who wrote the code for this is last person you'd blame, not the first. Programmers go in and do work asked of them. In the therac situation it would be the company at fault. If an uber self driving car really did kill someone (it hasn't, it was the lady), the company would be at fault. The only way you could blame an engineer is if they lied about meeting the requirements they were asked to do.
- hhas01 7y agoTypical code cowboy mindset: expects all of the glory, none of the responsibility. Kindly piss off and don’t come back until you’ve learned proper systems design. https://en.wikipedia.org/wiki/Fail-safe https://en.wikipedia.org/wiki/Fail-safe Self-driving software might find its way around a controlled test track okay but is nowhere near fit to be in control on public roads. Safe way to test it the is to have human drive and have the software simulate the same drive, then compare the two for discrepancies. Rinse and repeat until the software consistently equals or betters human decision-making. Then you can consider putting the software in charge. Programmers are supposed to be the technical experts in the room. So if they aren’t pushing back against ignorant uninformed hubristic Management’s bad decision-making and making it right, how the hell do they think their software will do any better? Fucking Worse than Useless, the whole cowardly bloody bunch.
- edoo 7y agoI don't think you understand the basic concept let alone how the self driving systems are comprised and how large software development projects work. The uber car wasn't self driving. It required a human occupant to watch the road at all times. That is certainly not on the developers.
- hhas01 7y agoYou appear to be laboring under the gross misassumption that the human is there to take control when the machine fucks up. They are not. They are there to take the blame. You cannot realistically expect a human spectator to switch modes at zero notice and save the machine from a deadly fuckup that the machine has already put them into. That’s not how human minds work. We just don’t bootstrap that fast. Jebus, we’re bad enough at salvaging bad situations we’ve knowingly manoeuvered ourselves into while already fully engaged in command and control mode. At least when you—the human pilot—fucks up, you already know the decision chain that got you there because it is your own. With the machine you have first to determine it has gone catastrophically wrong, then determine how it has gone wrong, and finally calculate and execute the recovery strategy before…oh, whoops, too late: you just wiped out all the executive bonuses for this quarter. Also, there’s a blood streak on the street. As for how large software development projects work, I think the trail of corpses already testifies more than suf as to how they don’t. Industrial institutionalization of incompetence is no defense, and any “professional” who hides behind it can go get fucked.
- mapcars 7y agoBut why you want to keep him accountable and not the ones who did the testing and released the product?
- wpietri 7y agoI never said he should be the only one. Responsibility is not zero sum. But as a professional software developer, I am pointing out the professional responsibility we all have.
- gwd 7y ago> There were many things wrong with this project, but one of them was fatally bad code, which this guy took money to write. If I remember correctly, the bug in this case was a race condition between "normally" running code and an interrupt handler. The race was only triggered if an interrupt happened in just the right window between two instructions. I'd be willing to bet that 99% of programmers, if simply given the code and asked, "Is there a bug here?" would have answered "No". Should people writing operating-system-level medical equipment software be required to have a basic training about race conditions and how to prevent them? Yes. Is it fair to expect a random engineer who was ordered by his company to rewrite the code to work without a hardware interlock to know what he doesn't know? No.
- starfallg 7y ago> Is it fair to expect a random engineer who was ordered by his company to rewrite the code to work without a hardware interlock to know what he doesn't know? No. Looking back at when I did Safety Critical Systems at Uni way back in another era, the most important points that we got out of the Therac-25 case study were not to do with bugs at all, but to do with the deficiencies of the system architecture and methodology, especially the decisions that led up to it.
- mcguire 7y agoIs said "random engineer" an engineer? No. He's a code monkey and should be expected to be paid like a code monkey.
- wpietri 7y agoIt is 100% fair to expect people working on life-critical systems to either a) know that they are competent to work on life-critical code, or b) to not work on life-critical code. The guy wasn't ordered to do this at the point of a gun. His bosses asked him to do something, he said yes because he liked the money, and people died. The bosses are also responsible, but that doesn't mean that his negligence didn't kill people.
- epx 7y agoIMHO whoever decided to let out the hardware protection, present in older machines, is the 'culprit'.