6 ms·
So management bears no blame for requiring illegal work be done, on pain of termination? Said another way, engineers now need to be technical and legal experts
by paulryanrogers 6y ago
So management bears no blame for requiring illegal work be done, on pain of termination? Said another way, engineers now need to be technical and legal experts in the business domain?
(Remember employees in the US depend on the company for health insurance. Saying 'no' could cost a lot more than just ones position.)
Most software engineers are not like doctors. We have little autonomy over what is created. Our responsibility is primarily the how. And with devops sometimes the actual deployment and maintenance itself.
- skissane 6y ago> Said another way, engineers now need to be technical and legal experts in the business domain? Consider something like the 737 MAX debacle – did the programmers writing the MCAS code actually have enough aviation domain knowledge and understanding of where the component fit in the overall system to realise it was a threat to people's lives? I don't know, but my guess is the most likely answer is "No".
- MattGaiser 6y agoI can’t imagine software engineers staying in a company long enough to acquire it. For this to work, software engineer compensation would need to change radically.
- watt 6y agoMaybe that's the point of this oath? Software engineers will refuse to work on these areas, unless demands of the oath can be met.
- Chris2048 6y agoIf a doctor says no, it's because of legal liability and risk to licence. For the same reason, it's harder to hire some rando budget doctor because the field is gate-keeped by the requirement of a licence, and liability. You can't magically make Engineering the same without the same conditions. Add barriers top entry that see my pay rise, or have cheap programmers with no liability.
- bumby 6y agoEngineering already has these barriers, they just aren’t required or enforced. I’ve never been on a project that requires a software product stamped by a licensed engineer. NCEES dropped the software license because there was so little demand, compared to, say, civil engineers who consider a license a rite of passage to career growth.
- Chris2048 6y agoYou need a licence to practice medicine, if there is something called a "license" for engineering but isn't required for practise, then it's not the same. If we have the same barriers, but not enforced, then we don't have the same barriers.
- bumby 6y agoThat’s not quite right. You actually do need a state license in the US to practice engineering for the public except for a few basic instances: 1) you work for the federal government 2) you work under a licensed engineer 3) you work under an industrial exemption There’s differences depending on state. There has been a more concerted effort to remove #3 recently due to both political reasons and the technical issues in this thread. Most people performing engineering work under an industrial exemption don’t actually realize it. Again, this is different state to state. For example, in some states you can’t start a business with “engineering” in the name unless you have a certain percentage of owners/principals with an engineering license.[1] What actually happens is conflating terms in common parlance. “Engineer” and “engineer” are not necessarily the same. For example, a computer engineer may work under an industrial exemption (due to working in a manufacturing service) while a software engineer does not. Legally, an “Engineer” claims an explicit responsibility to public safety.[2] Apropos to the headline article, there is a distinction with this difference. [1] https://fxbinc.com/wp-content/uploads/2016/10/state-by-state-summary-licensure-law-exemptions.pdf https://fxbinc.com/wp-content/uploads/2016/10/state-by-state... [2] https://www.theatlantic.com/technology/archive/2015/11/programmers-should-not-call-themselves-engineers/414271/ https://www.theatlantic.com/technology/archive/2015/11/progr...
- wonderwonder 6y agoA doctor refusing to do a procedure because he worries for his patient is seen as a good guy doing what is right, he is in his opinion saving a life. In addition he is trying to avoid the massive cost of a medical malpractice suit. A software developer refusing a job because it does not meet his ethical parameters is just an unemployed software engineer.
- bumby 6y agoI think one of the issues is the domain of is vast. One developer may be working on a basic CRUD app while the next is working safety critical code on a vessel going to Mars. There are definitely areas where the prudent thing for a developer to do is raise a dissenting opinion, if not halting work. What seems lacking is clear industry consensus standards to back up that decision.
- bumby 6y agoFrom my experience, software engineers don't rotate nearly as often in traditional cyclical and defense businesses (auto, aerospace) as they do in the FAANG technology sector. There are a lot of grey beards who weren't so grey when they started.
- ilaksh 6y agoThat's completely the wrong comparison and context. The reason MCAS came about was because management wanted to try to fudge a larger engine into an outdated design created for a different purpose rather than do the engineering and certification necessary for the new requirements and to update the system. Management wanted to save money. Of course the engineering leadership did not want to fudge something -- they wanted to do proper engineering. But the people in charge just wanted to save money, and the engineering leadership could not do anything otherwise, even they knew that just making the engine larger and compensating did not make sense from an engineering standpoint. By the time it got to the MCAS, that was far down the line of the decision to not do proper engineering.
- chii 6y agoWhich demonstrates the irrelevancy of a hippocratic oath for engineers. Because the complexity of the system is such that no single engineer understands it all, and thus no single engineer is (or feels) responsible. Blaming it on management is also irrelevant, since management merely takes the advice of engineers, and do financial/business trade offs to maximize profit. If the engineers cannot tell that MCAS system could fail this way (due to the complexity), management will not question it.
- moron4hire 6y agoThat's like saying burglars who break windows demonstrate the irrelevancy of door locks.
- ilaksh 6y agoHippocratic oath for engineers is irrelevant, correct. But management does not take the advice of engineers. As I said, the engineers wanted to do proper engineering, but management wanted to save a buck, so they instructed engineering to fudge a cheap solution. Its very common.
- kittiepryde 6y agoSo.. if the engineering leadership said, this is not a good idea. Management said, it saves money. Engineering failed to convince management. Management didn't have the understanding that it was a bad idea. It is now, no one's fault? If management merely takes the advice of engineers ( and other who specialize in the things that they do not ), and they choose to ignore it because they do not understand the things they do not specialize in. I believe it's a reasonable to assume that management is more at fault than engineering ( I'm not sure they're is a situation here where any party is fault less )
- imtringued 6y agoFrom my limited information the MCAS code was primarily causing problems in association of incorrect readings by damaged sensors. Of course one could argue that this is an engineering failure because the MCAS failed to account for wrong sensor input but when you consider the legal implications of a MCAS fallback there is actually not much that can be done on the software side. The MCAS is an optional component that reduces certification and training costs. It is definitively possible to fly the plane without accidents even with a disabled MCAS. So why can't the MCAS be turned off automatically when sensors fail? Because that changes the classification of the plane and therefore requires pilots to be certified for a new machine and receive new flight training for both MCAS and no MCAS modes. If the software engineer was under a hippocratic oath then he would have to refuse to build the MCAS entirely but not because the idea of an MCAS is inherently unethical, no, he would have to refuse because the company he works at wants to use the MCAS for a non ethical purpose (namely operate and hide the existence of MCAS even when it is unsafe to do so). This is basically a reverse audit but the software engineer has no authority conduct such an audit and even if he was allowed to, the business has no obligation to give him the necessary information to determine whether the MCAS will be used unethically.
- skissane 6y ago> he works at wants to use the MCAS for a non ethical purpose (namely operate and hide the existence of MCAS even when it is unsafe to do so). You think a programmer, handed a spec and asked to implement it, can be expected to know that their employer (or the employer's customer) wants to use it for a "non ethical purpose"? Again, I can't know for sure, but I doubt the programmers who wrote MCAS (who most likely didn't even work for Boeing, but rather some subcontractor) actually knew, or could have known, how the code fit into Boeing's larger purposes
- UnpossibleJim 6y agoWith all of that being said, were these software engineers (probably subcontractors) even given access to actual MCAS readouts or, more likely, virtualizations of expected readouts. These people probably didn't account for this type of faulty readout because the virtual machine never put out that type of fault. Most types of development in these large companies is so compartmentalized that it's next to impossible to see the whole structure from a software engineers prospective. You need to be at a management level to understand how most of the pieces really come together, which is the only place where one of these "oaths" might make an influence. At that point, however, the selection is so goal oriented, I have a doubt as to whether or not people would take that oath.
- qyv 6y ago"I was only following orders." Edit to add: If you are being asked to perform illegal or unethical acts as part of your employment, then perhaps termination is an ideal course of action? Unless of course your personal enrichment outweighs legalities or ethics in your worldview?
- paulryanrogers 6y agoAnd I wasn't implying engineers should be entirely blameless. Everyone has a limited understanding of legal systems too complex for one person to fully grasp. And workers far below the level of decision makers should be judged according to evidence of their knowledge and responsibility. Likewise those who give orders should bear more responsibly. All these "companies take on a life of their own" arguments sound a lot like executives priming the pump of potential jurors with excuses. If decision makers cannot bear responsibility because of a company size or organizational structure then we can make some sizes and structures illegal before they stumble/march into devastating incompetence.
- drivingmenuts 6y agoIt may be fine in some countries, but saying that you’ll make some organization sizes and structures illegal, barring other criminal activity, smells like a violation of the freedom of association.
- kazagistar 6y agoWhy is that a good freedom?
- drivingmenuts 6y agoBecause it allows people to associate with whom they choose to. Remove that and you’ve opened the gate to legal racism, legally institutionalized homophobia, banning of religion; the list is endless. The five freedoms are the pillars of our Constitution. Without them, we are no better than China or Russia or even any third-world hellhole you care to mention.
- tfehring 6y ago> So management bears no blame for requiring illegal work be done, on pain of termination? No, I wouldn't say that. In many cases, management and engineering share the blame jointly and severally since they both have an opportunity to stop it. > Said another way, engineers now need to be technical and legal experts in the business domain? Engineers should know enough about their business domains to understand the ethical impacts of their work. Ethics and law are orthogonal, so thankfully this is generally much easier than being a legal expert. > (Remember employees in the US depend on the company for health insurance. Saying 'no' could cost a lot more than just ones position.) Thankfully, the healthcare safety net in much of the US is far better than it gets credit for, and the pay and availability of opportunities for software engineers in the US has generally been quite good. I'm sympathetic to this argument in general, which is one reason I don't think there should be an oath for, like, Amazon warehouse workers, but I'm far less sympathetic for anyone making 5+ times the median US income. > Most software engineers are not like doctors. We have little autonomy over what is created. Our responsibility is primarily the how. And with devops sometimes the actual deployment and maintenance itself. Your responsibility as framed to you by the business is the how, but upstream of the how is the question of whether or not to do it at all. If you contribute to a piece of software, you've tacitly answered yes to that question.
- paulryanrogers 6y ago> Engineers should know enough about their business domains to understand the ethical impacts of their work. This would be more reasonable in the era of 10-20 years in the same company or industry. Needing to job hop every 2-3 years for a decent raise, and software skills applying to a vast array of industries, makes it less reasonable IMO. > ...but I'm far less sympathetic for anyone making 5+ times the median US income. Not everyone here or in software makes that kind of money. Some of us in the Midwest--or who aren't as skilled at negotiating--don't pull down nearly that much.
- tfehring 6y ago> This would be more reasonable in the era of 10-20 years in the same company or industry. Needing to job hop every 2-3 years for a decent raise, and software skills applying to a vast array of industries, makes it less reasonable IMO. Yeah, to be clear, I don't think software engineers have an infinite level of responsibility for understanding the ethical implications of their work. If you were a software engineer at a credit rating agency in 2006, and you didn't see the ethical dilemma because you didn't anticipate that contagion would be exacerbated by the shadow banking system to bring down the global economy, you get a pass. But if your prospective employer is, like, locking children in cages, or spreading disinformation on political candidates, you should probably find that out during the interview process. > Not everyone here or in software makes that kind of money. Some of us in the Midwest--or who aren't as skilled at negotiating--don't pull down nearly that much. Good point - I'm also in the Midwest and make less than that, for what it's worth. I've naturally had FAANG in mind as I type these comments, and more generally I think salaries for the more unethical roles tend to skew higher.
- BeetleB 6y ago> So management bears no blame for requiring illegal work be done, on pain of termination? Said another way, engineers now need to be technical and legal experts in the business domain? This is a red herring. The oath is not needed merely for illegal work. In fact, the more common use cases will likely be legal. It's a common sentiment, but: Don't conflate ethics with legal. > Most software engineers are not like doctors. We have little autonomy over what is created. Our responsibility is primarily the how. And with devops sometimes the actual deployment and maintenance itself. This is not a dichotomy - there can be a spectrum. You can restrict it to those who do know what the product is used for, or at least have good guesses for them. And while not everyone is this way, I wouldn't really want to work in a job for long if I'm not told what the code I'm writing is for. It's not even an ethical concern for me - it just makes for a boring job. Ideally I want people to tell me the problem they are solving and give me some leeway in crafting a solution. Don't come to me with a solution and ask me to implement it.
- an_opabinia 6y ago> I wouldn't really want to work in a job for long if I'm not told what the code I'm writing is for. But money dude.
- Conan_Kudo 6y ago> Most software engineers are not like doctors. We have little autonomy over what is created. Our responsibility is primarily the how. And with devops sometimes the actual deployment and maintenance itself. If you're an actual engineer of any kind, you always have some choice on this. You make architectural decisions every day, and you generally work for places that do, in fact, take your input into consideration. If you don't work for any of those kinds of places, then you are still responsible because you wrote the code to enable it. You can always say "no". There are consequences for that, for sure. You can always quit as well. And it may still get made. But it won't be by you. And sometimes, that's still better than the alternative.
- ClumsyPilot 6y agoYou are missing the point, the idea is that a average developer at a major bank has very poor understanding exactly what impact of his code will be. He generally has neiglther the information (secrecy, need to knwo basis) nor the understanding of the financial system. On the contrary, a doctor's desision affects the life of an individual patient in a very clear and understandable manner.
- ozim 6y agoJust check out cases of VW software devs who added code for emission test cheating. Check out the case of dev who was copy-pasting code at Toyota. Illegal stuff is illegal because not knowing the law is no excuse. For your own safety and good, you better have some grasp on legal stuff in your domain. Don't have to be an expert.
- sneak 6y agoVery few software engineers in the US, despite receiving health insurance as compensation, do not have sufficient cash comp to purchase health insurance on their own should they need to. I believe that this is a red herring. Yes, the drone operator that pushes the button is a murderer, even if it was the POTUS that gave the order for it to happen.
- imtringued 6y agoI don't think the drone operator is a good example. There are situations in which unethical things must happen, no matter what. If wars could be fought without killing we'd be doing that already.
- sneak 6y agoThat’s called peace, and most nations are doing that already. Just not the minority of violent ones.
- darkerside 6y agoHow much autonomy do you think doctors have? They're not in the operating room inventing new procedures. They are following careful scripts, adapted for the intricacies of one particular human body. There are times where they need to qualitatively improvise, yes, but that's generally only when something has gone horribly wrong. Also, the Hippocratic Oath is not terribly complicated. It basically says, I will not furtively and maliciously hurt people in an abuse of my authority, and I will try to heal them when they are sick. I don't think it's a lot to ask that software engineers to agree not to create knowingly malicious software. It actually addresses exactly the problem you describe. If everyone has taken this oath, and adheres to it, there is no "someone else" to do that evil work. Last point, there is a wide variety of software engineering work out there. Some of it may be mindless of the bigger picture of what is actually happening, but for any sufficiently advanced behavior to emerge out of a complex software product, some engineer at some level has to have some idea of the path they are going down to create or allow that behavior. And every engineer has the ultimate autonomy over how and what is created because it is our hands on the code. If you don't understand that, you don't understand the power of the profession.
- bumby 6y agoWe can take lessons from other engineering domains. Disregarding exemptions, engineers who design and build for “the public good” have someone who is ultimately responsible. A newly minted civil engineer, for example, may be low on the hierarchy but has to work under the direct supervision of a licensed Professional Engineer. That licensed PE is the one responsible, legally and ethically, for now just the “what” but also the “how”. They may work for a project manager but ultimately it’s their stamp that allows the build. As someone who works in safety critical code nothing irks me more than when people absolve themselves by saying “I’m just a programmer, that’s not my job/problem”. We need to hold ourselves to a higher professional standard