17 ms·

I’m not a fan of comparisons to the Hippocratic oath. The greatest risk to AI ethics is not the ethics of software engineers but the ethics of the software engi

by madrox 6y ago

I’m not a fan of comparisons to the Hippocratic oath. The greatest risk to AI ethics is not the ethics of software engineers but the ethics of the software engineering process. By the time tasks are handed to engineers, most of the ethical decisions have been made by product managers, designers, and business stakeholders who are focused on their own goals. Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. The Hippocratic oath sounds more altruistic than the alternatives, but good legislation, including business audits and incentives, will have far more impact than a software engineer swearing they won’t be evil.

batiudrami 6y ago
Being expected to behave ethically is part of being an Engineer regardless of what your boss expects. Now we increasingly see real world negative consequences of the work of software engineers I don't see why they should be any different.
- spunker540 6y ago
  Is behaving ethically not a part of being a designer or a PM or a director/boss?
  - brobdingnagians 6y ago
    And, what's more, isn't behaving ethically not part of being human? Aristotle and a while host of philosophers certainly thought so. Extending this, why don't we expect everyone in society to behave ethically? But then we get into arguments about what is ethical, because people disagree on that, and people naturally will disagree on what is ethical in software development, which isn't always a problem. The is a diversity of opinion, though since are clearly extreme. One of the issues with a "Hippocratic oath" for software developers is that software spans the while spectrum of human activity and thought. Constraining software is nearly equivalent to constraining human thought in more ways than one.
    - Ygg2 6y ago
      Ethic is subjective. Across time and space. Want to see an example - Ask Aristotle are women same or lower than men?
      - pixl97 6y ago
        And relevant to this conversation, software can be written anywhere especially if you need alternative ethics.
    - makomk 6y ago
      Yeah, the problem with this idea is that it assumes there's one fixed set of right ethical values everyone should agree on. Take for example the actual, original Hippocratic Oath - it required, amongst other things, doctors not to provide abortions to women. There's still a large number of Americans who hold this ethical belief, but I'd venture a guess that almost all the people pushing for ethics in software development hold the exact diametrically opposite view. A lot of the demands for software ethics get into even fuzzier and more complicated territory, like whether the companies which control our mass communications should use that control to decide which political views people should be able to share and what they should target, whether this is good or bad for democracy, etc. Also, many of them seem to be things that would've been incredibly niche viewpoints even a decade ago. I've seen people in another thread thinking that Google promised not to use e-mail contents to sell ads originally and then snuck that in, but in reality basing ads on the contents of e-mails was part of their business model all along and it's just that no-one cared when they launched because it was so much less obnoxious than the ads on other providers. Somewhere along the way, we got this meme about big tech selling our personal information, and everyone seems to project it backwards in time onto how people felt before the meme.
  - Polylactic_acid 6y ago
    Ethics are subjective. Plenty of people working at facebook will tell you their work is ethical.
  - jimmySixDOF 6y ago
    That's why Smith also talks about a new DIGITAL GENEVA CONVENTION - not just a code for devs - so there is a defined framework for everyone in the chain. Pls excuse the caps, but too many people seem to ignore this very important part of his argument here.
    - Nasrudith 6y ago
      I thought everyone ignored it because that is what would happen in practice. Like the actual Geneva convention only applying in an unconditional surrender or complete defeat.
- p1necone 6y ago
  I think the point is that we can't just rely on individual ethics to enact change. People have bills to pay and kids to feed, if it's all on the man at the bottom to say no then a lot of bad software is still going to be created.
  - Legogris 6y ago
    The argument made is that it's necessary, not that it's sufficient.
  - jrumbut 6y ago
    I think making a statement as an industry helps avoid some of this stuff even being asked. With research you can easily find doctors who are brought in to assist with torture or other heinous crimes, but in general I would expect it to be harder to find a doctor to help with my project of creating a more addictive cigarette versus finding a web developer to help me market them. There's some (not a lot, but some) power in a profession just claiming to have ethics, even if there is no enforcement and change in the hearts of practitioners. Plus, at some point, we all just have to own what we do with our lives. It's great to advocate for systemic change but doing your own small part is at least as important.
    - wolco 6y ago
      You would lose the ability to practise medicine as a license is required. Do you want to license software developers?
      - p1necone 6y ago
        I think we should license some software developers - if you're developing software to be used in industries where the people using your software need to be licensed (i.e medical equipment, structural engineering, industrial control systems) then perhaps the software engineers should be similarly rigorously qualified. But if you're working on video games, or pizza delivery, or really probably like 90% of software then no.
- Tehdasi 6y ago
  The consequences of the decisions made by Microsoft Presidents and their ilk, since they are the ones who actually decide what software will look like. But still it would be good when MS gets found to do unethical things that a programmer could be scapegoated instead of their president, at least from the presidents perspective.
- gonzo41 6y ago
  Are you expecting engineers to quit FB and IG and Twitter en masse? Because I'm not. People like money more than each other.
  - imtringued 6y ago
    It wouldn't surprise me if there are people specializing in ethics violations in exchange for lots of money.
- Nasrudith 6y ago
  Well there is one major assumption error there - that negative real world consequences are only linked to negative ethics. That is so wrong it isn't even funny. If the car was invented as powered by a Mr. Fusion the buggy whip makers going out of business would be a negative real world consequence.
  - imtringued 6y ago
    Cars made us wealthier but they also contributed to obesity. It's not exactly clear if reducing obesity by banning cars is worth it.
- tomjen3 6y ago
  In some Engineering societies maybe, in general you may be liable if you construct a building you shouldn't have. Software is fundamentally different because until you run it it has no consequences, and even if you run it, it can be contained. I can write a worm and not release it on world. In that regard, it is more like engineering _plans_. I can draw up plans for a building that is designed to collapse with X number of persons inside -- in fact I can imagine either of the two assignments given as an exercise in University. No reason to make more laws: it should be immaterial whether I chop down the Christmas tree at the local town square or program a robot to do it.
  - tech2 6y ago
    It should be immaterial whether you chop down the Christmas tree or set a pre-programmed robot capable of doing the same on its way. The programming of said robot isn't the bad act here, it's the act the robot actually performs. The danger of being castigated for having written something that wasn't used is that we then get into the area of thought-crimes.
asperous 6y ago
I agree, however not all ethics can be legislated. This sort of thing works in other professions like medicine because malpractice can cause doctors to lose their license. Same with civil engineers. This changes things because the choice is now quitting or possibly never being able to work in the field again. Perhaps principal software engineers in charge of life or death software should be licensed for accountability, “engineer on record”.
- optimiz3 6y ago
  Doctors don't typically have bosses directing how they treat patients; Doctors have full agency, front line grunt workers do not.
  - SpicyLemonZest 6y ago
    But that's not an independent fact. That's one of the major goals of medical oaths; they establish that doctors must be able to determine the right course of action on their own, that ethical doctors won't work for an organization that won't give them that agency.
  - TeaDrunk 6y ago
    I don’t think this is true in America. Doctors are limited on what will benefit the system monetarily. Healthcare administrators will decline suggested procedures from doctors due to expense.
walrus01 6y ago
> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. I agree to a very limited extent about the hierarchical nature of a typical corporation, but I also disagree. Software engineers at a certain level of their career and with relatively uncommon skills can pick and choose what companies they want to work at. In my opinion people of good moral character and conscience need to be prepared to refuse to accept a position at companies known to engage in activities against their principles. And further need to be prepared to resign if they are asked to do something clearly unethical. From my particular specialization in network engineering, I would never accept a role at an ISP in an environment where I had to implement something like the GFW in China, or further walled-garden/censorship of the global Internet. It's directly contradictory to my principles. I sincerely hope that the best and the brightest of my colleagues would never choose to aid and abet internet-fuckery by autocratic regimes. If people from my field look at a project and could reasonably say "Vint Cerf would be really disappointed if he saw me implementing this...", I hope they will choose to walk away.
- peteradio 6y ago
  Would you accept any job under penalty of imprisonment for violating grey ethical rules?
  - walrus01 6y ago
    Under which nation-state's legal system are we basing this theoretical scenario?
- xdavidliu 6y ago
  > Software engineers at a certain level of their career and with relatively uncommon skills can pick and choose what companies they want to work at. What percent of software engineers does this statement apply to?
tfehring 6y ago
In sufficiently complex systems, you can always blame someone else if you're motivated to do so. https://www.youtube.com/watch?v=IssR_J0QWr4 https://www.youtube.com/watch?v=IssR_J0QWr4 But ultimately there's always a software engineer involved in the creation of software - and that's not true of any of the other roles you mentioned. Since software engineers are necessary and sufficient to produce software, they should always be held responsible, and any oath should fall on engineers. > To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. Well, yes - if there were no tradeoffs there would be no point in having an oath to begin with. But there are software engineers today, including some on HN, who do things more harmful and unethical than medical malpractice, and they are personally culpable for the decision to do so - just as their replacements would be if they refused. I would also like to see laws criminalizing those individual engineers' conduct - maybe you're alluding to the same thing? - but an oath is a good start.
- Aerroon 6y ago
  How about a compromise then? Software engineers are responsible for the negative consequences of the software. Any and all responsibility they take is then also shared with every person that is above them in the hierarchy. Eg a developer does something and society finds this unethical and punishes them. The developer's boss, the boss's boss, the boss's boss's boss etc up to the CEO all get punished in the same way. Furthermore, to avoid companies trying to shield themselves from this by putting their developers into a different company, it will apply to software that you get from someone else too. Suddenly this doesn't sound very appealing anymore, does it?
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
  - bjo590 6y ago
    > Any and all responsibility they take is then also shared with every person that is above them in the hierarchy. Currently, if I write software that performs illegal actions -- let's say software that allows me to use unlicensed Adobe products -- at the request of my boss and their boss, all three of us would be legally liable.
sjwright 6y ago
Corporate strategists blaming software engineers for the consequences of corporate strategy is a fairly brazen kind of blame-shifting. A system of ethics within the health system are necessary for customers to retain trust in the health industry. It's also strongly aligned with the selfish interests of workers who must enact that system of ethics. These properties do not neatly translate to software engineering—mostly because the most difficult ethical dilemmas in technology are rarely obvious when looking at source code. The problems with Facebook (for example) are not always inherent in code; many are only revealed after being deployed at scale and external groups begin exploiting the system.
peterkelly 6y ago
> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be I've quit jobs in the past because of ethical concerns about the way in which those above me have been acting. In one case this involved bribery of senior government officials to push through a project that put at risk the privacy of hundreds of thousands of people. If you go along with shit like that, you're an accomplice and share partial responsibility. As professionals we have a responsibility to stand up for what is right. It's not good enough to fall back to the lazy excuse of "just doing my job".
- pixl97 6y ago
  And I mean, that is good of you to have a moral backbone, unfortunately there are many people behind you that will do the same job in software and they can be located anywhere in the world.
  - walrus01 6y ago
    In the current state of things, that is factually true. However the same argument could be applied to labor abuses in the textile/garment and shoe manufacturing industry. Most people who follow news are probably aware that about 20 years ago Nike, Adidas and other brands went through a period of terrible public relations disasters, after the working conditions in some of their shoe factories in developing nations were exposed by journalists. The argument that could have been made at that time would also have been "well but if we don't employ these people, somebody else will just do the same thing with equally terrible labor/human rights violations somewhere else in the world, with even cheaper factories". The situation today is not great, but it is significantly improved from how it was twenty years ago. There are third party neutral inspection/oversight agencies. Companies in the garment industry are forced to make public commitments to labor rights and reasonable working conditions, and to allow external auditing. They can't just hand wave away the problem and say "but if we don't do it , someone else will.."
    - quickthrower2 6y ago
      The more abstract the problem the less people care. Police suffocating someone to death > 1,000,000 people being subtly spied on. "I got nothing to hide" after all.
tlb 6y ago
The Hippocratic oath is similarly local in scope. Individual doctors try hard not to cause harm to individual patients, but the medical establishment causes massive amounts of harm, by: - developing treatments for chronic symptoms instead of curing diseases - being unprepared for pandemics - making health care unaffordable except through employer plans - promoting wrong nutrition guidelines for decades after the evidence was in and more. To have good outcomes you need ethics at both individual and system-wide levels.
replyifuagree 6y ago
Agreed, the vast majority of software engineering disasters can be laid at the feet of bad management, not software engineers. Let's clean house in management first, just like Deming did when he straightened out Ford. https://en.m.wikipedia.org/wiki/W._Edwards_Deming https://en.m.wikipedia.org/wiki/W._Edwards_Deming
BeetleB 6y ago
> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. Software engineers are accountable to themselves before they are accountable to their bosses.
jbay808 6y ago
> To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. This is where a strict licensing requirement, like Canada's P. Eng, can empower the engineer. If you think what you're being asked to do would violate your professional ethics, not only can you decline to do it, but you have a system to ensure that you won't just get replaced by someone who will do it.
- chii 6y ago
  a licensing system is just a form of regulation as alluded to in the comment you're replying to. And in the end, if software engineers are to conduct themselves in moral and ethical ways, they must be empowered to do so without having to sacrifice their personal wellbeing or livelihood. Regulation, it seems, is the only way to achieve that end.
  - austincheney 6y ago
    The necessary regulation is imposed and enforced via licensing. An example simply look at every other professional industry.
- jimnotgym 6y ago
  Indeed it is the same with Chartered Accountants. Management can make whatever decisions they think they can make, but an accountant won't inact things that are unethical at the risk of their charter and professional standing.
  - coldcode 6y ago
    You really believe many people with won't violate their oaths and associations and licenses for enough money? Accountants audited Bernie Madoff and Wirecard for years and "saw nothing". Michael Jackson hired a surgeon to pump him full of drugs for $150K a month. Money > Ethics for a lot of people.
    - 6y ago
deleted 6y ago
[deleted]
dmarchand90 6y ago
To me this argument sounds a bit like a "only following orders" defence. I'm sure smart engineers ,I.e., all of them, can figure out what the application of their work will be.
noncoml 6y ago
Don’t sell out yo FB or Google or some other “not-evil(R)” Corp and you will never have to follow orders from your boss that don’t meet your ethics.
kazagistar 6y ago
Agreed. Let's look at it from a simple emperical standpoint: where does the problem mostly arise, and who has the power to fix it? Organizations that went through true iterative process to reduce failure rate like NASA figured out that they needed to allow true authority to specific domain experts to blow the whistle and not face reprisal or suffer for it. Oaths fix nothing, you need organizational change, and if someone is going to do that, its the management in charge.
znpy 6y ago
not to mention that software is reusable: a technology might be invented with all the good intentions and ideals... and later be used for evil purposes.
sjogress 6y ago
> To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. If we were to introduce an oath we would have to take further inspiration from doctors, e.g having a certification required to do the job, or, failing that at least having an industry-wide union/guild protecting the position.
- tech2 6y ago
  How then do you manage things like outsourcing, or the purchase of overseas companies, or multinational mergers?
nicoburns 6y ago
> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. Isn't that what a hipppocratic oath would solve. They'd be accountable to the oath before their bosses, and that would give them reasonable grounds to refuse unethical work.
- imtringued 6y ago
  How do you guarantee that the boss will cooperate with the software engineers and tell them about all the unethical business practices that are currently happening?
  - TeaDrunk 6y ago
    Eventually it would have to be told to a CTO, president or Vice President it some technological product, software architect, etc. Software Engineers generally need to understand the use case of a product to be able to develop something effective. It would be exceptionally difficult to convince someone to code something that creates fake profiles of people who don’t use your site and no one have any idea that that’s what’s happening.
  - nicoburns 6y ago
    Well if it's something that the software engineers are expected to implement, then they're going to have to be told about it. If it's not, then it seems out of the realm of software engineering.
- coldcode 6y ago
  ... and be an easy way to get rid of any engineer who fails to do the work asked of them. Think you can just get a new one?
  - wtetzner 6y ago
    The Hippocratic oath for software engineers should come along with legislation that makes it illegal to fire a software engineer for refusing to break the oath.
jlnthws 6y ago
Exactly. An oath means nothing: what you need is skin in the game. Some health practitioners are literally bought by Big Pharma, by their hospital accountant, etc. How would an oath fix that? Same with engineers or any other discipline. You need to make sure that everyone in the process has skin in the game. For me it's less about control (legislation) than about responsibility and accountability (assessments, eating your own dog food).
- ethanbond 6y ago
  What does “skin in the game” mean precisely in this context?
  - jlnthws 6y ago
    Having personal exposure, taking accountability for your actions. From Taleb of course. The case of Snowden shows how bad it can turn when not everybody in the loop has skin in the game (asymmetry). Ironically, his behavior tells us he has been faithful to some kind of oath, but apparently none of his coworkers or supervisors.
bryanrasmussen 6y ago
>To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. obviously for a hippocratic type oath to work you need the same kind of system in place for qualifying engineers that you do for doctors and not allowing anyone to work as an engineer who failed the ethics board.
eie7eu3hrruur 6y ago
I think people assume the Hippocratic oath actually makes a difference. It's culturally important to many doctors of course, but how often has that hindered the military or the CIA from hiring enough doctors to facilitate a new interrogation program? Of the illegal human experiments that happened in the US many had doctors working for them as well. It's a good guiding principle but the idea that it would actually make an impact of any kind is debateable. It doesn't matter if 90% of coders follow the oath if 10% or even 5% are enough to handle the demand for oath breaking.
Cthulhu_ 6y ago
Pretty much. Those that take the oath would quickly see themselves not hired for certain positions, in favor of those that didn't take the oath.
- TeaDrunk 6y ago
  This assumes all of the vast majority of engineers are easily replaceable. Given that several tech companies prefer the chance of not hiring a good fit over the chance of needing to fire someone, I think the cost to a company if a significant subset of its engineers declined unethical work (especially engineers who at this point are high enough to have a good idea of the broad scope of the software) would make it difficult to perform what you suggest.
bumby 6y ago
Maybe the doctor analogy is flawed, but isn’t there ethics of a licensed (capital P) Professional Engineer more apt? They are responsible to their bosses as well as the public who are the end users of their designs/products
coffeefirst 6y ago
This thread is littered with this attitude, that the developers don't actually have a seat at the table where "what to build" decisions are made. This was not always the case. How the hell did we let that happened?
- xnyan 6y ago
  Here's a prospective from someone who became a software dev in my late 20s after other jobs. Relatively speaking, if you can write working code and show up on time, you have a lot of leverage in getting and keeping employment, more than most other middle to upper middle class professions. This means org politics has less effect on you, and less incentive to get involved. As a dev you can, but don't have to, think as much about the politics and operations of your org for all kinds of reasons. You are relatively harder to replace so internal politics tends to matter less, and if the org makes decisions you don't like, you can be confident that you can leave and find something else versus the long and often unfruitful process of trying to change an org from within. Politics can and often is messy, how often have you heard something like "I just want to build things" (it's how I feel for sure), if you can get paid well to do that, why get involved with a messy decision making process?
  - 52-6F-62 6y ago
    To extend upon this (though personally I feel a little differently than yourself, but inclusive of how you feel as well): My personal experience has seen individual contributing software developers have little voice in the matter regardless. Outside of "tech"-forward companies they are of little consequence in the larger political structure of a company. I've seen it and been it—speak up about a desired direction, voice concerns about a decided direction, concerns about faulty legacy software, etc. Those voices, unless amplified by political clout mean little to nothing to anyone else. In a lot of organizations, title is everything when it comes to moving discussion. So my thinking, if we're continuing the comparison to the medical profession, is hierarchies must follow similarly. A head of Surgery in a hospital ward is going to be a doctor. Hospital directors are going to be doctors. Sure, the CEO may not be, but they rely on the expertise of their directors. If the directors don't follow the same code as the rest of the professionals under them, then they can theoretically impose any ethics they choose and the onus falls to the IC/surgeon/etc which for the intended purposes of the oath/license/regulation at all is tenuous.
mcv 6y ago
The oath should be for everybody involved in the process. After the 2008 mortgage crisis, Netherland required everybody working at banks to take the banker's oath, which is mostly about balancing the interests of the 4 main stakeholders of the bank: shareholders, customers, employee, and society. It's pretty broad, it doesn't magically fix everything, but it does make everybody more aware of their responsibilities. Maybe software companies should require something similar, where everybody needs to be aware of their responsibilities towards, well, primarily user data, I guess. And that goes for not just software engineers themselves, but for everybody involved in the process.
- GordonS 6y ago
  Where do you draw the line though? Does everyone employed in any capacity at any kind of business have to take an oath? It seems to me that if everyone has to promise to do no evil, the meaning of such oaths would become diluted.
  - mcv 6y ago
    Do all those businesses have a problem with unethical behaviour causing serious problems in society? Doctors, bankers and apparently software engineers have a pretty big impact on society, and often in ways that aren't very transparent to most other people. It's quite possible there are other professions that have a similar impact, but I'm pretty sure it's not all. I think registered accountants also have some sort of oath, again because various stakeholders including society as a whole has to be able to trust them.
Heliosmaster 6y ago
There is the problem that everything can be subdivided in a bunch of menial task so every cog in the system isn't aware of their impact. It already happens a lot in software. the idea of Hippocratic Oath reminds me of Asimov's Three Laws of Robotics in "The Naked sun" (SPOILERS ahead): the detective realises that the normally quoted First Law of Robotics ("A robot may not injure a human being or, through inaction, allow a human being to come to harm.") is actually just an approximation, he argues that the real Law is "A robot may do nothing that, TO ITS KNOWLEDGE, will harm a human being; nor, through inaction, KNOWINGLY allow a human being to come to harm." This is important because even though robots really try their best, different robots could perform sub-tasks that look very harmless by themselves, but combined kill a human being: - A robot is instructed to pour this bottle of poison into a caraffe of water and then leave the room - Another robot is instructed to enter the room, take the caraffe of water and give it to a human to drink The human is poisoned, but none of the robots are directly responsible (in the first law sense). Is the act of connecting the two dots the evil deed.
- DetroitThrow 6y ago
  > There is the problem that everything can be subdivided in a bunch of menial task so every cog in the system isn't aware of their impact. It already happens a lot in software. Precisely the issue with the original proposal. Would it have mattered whatsoever if PhD's took a Hippocratic oath when developing the Manhattan project? I feel like this MSFT executive may already know that swearing engineer to "do no harm" is fruitless after reading the article, but it's still unfortunate that statements like his diverts attention from more meaningful proposals.
giancarlostoro 6y ago
Kind of funny to hear this from a company known for harvesting tons and tons of telemetry from customers, with no true way to fully opt-out which is pretty damn unethical and probably feeds their AI.
Zenbit_UX 6y ago
> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. The nazi officers who committed most of the atrocities used similar arguments. "I was just following orders!" I expect better from a software engineer on hacker news. You've single handedly convinced most here - through your weak logic - that such an oath is necessary.
- watwut 6y ago
  But we still blame the Nazi leadership the most. The nazi officers you talk about here were also in fact management typically.
deleted 6y ago
[deleted]
Osiris 6y ago
Doctors can have their licenses revoked. Software engineers cannot. There's no downside to being involved in unethical businesses.
GuB-42 6y ago
Agree. A software engineer is more like a chemist working for the pharmaceutical industry than a doctor treating patients. And chemists typically don't have an Hippocratic Oath. Pharmacists sometimes have their own version, but it is mostly about giving good advise to patients and respecting them as human beings. But it doesn't stop the pharmaceutical industry from being heavily regulated, and while their business practices are often criticized, the drugs that come out of it are generally safe and effective. Many countries also have regulations making important drugs (ex: vaccines) accessible to everyone.

by madrox 6y ago

batiudrami 6y ago

Being expected to behave ethically is part of being an Engineer regardless of what your boss expects. Now we increasingly see real world negative consequences of the work of software engineers I don't see why they should be any different.

spunker540 6y ago
Is behaving ethically not a part of being a designer or a PM or a director/boss?
- brobdingnagians 6y ago
  And, what's more, isn't behaving ethically not part of being human? Aristotle and a while host of philosophers certainly thought so. Extending this, why don't we expect everyone in society to behave ethically? But then we get into arguments about what is ethical, because people disagree on that, and people naturally will disagree on what is ethical in software development, which isn't always a problem. The is a diversity of opinion, though since are clearly extreme. One of the issues with a "Hippocratic oath" for software developers is that software spans the while spectrum of human activity and thought. Constraining software is nearly equivalent to constraining human thought in more ways than one.
  - Ygg2 6y ago
    Ethic is subjective. Across time and space. Want to see an example - Ask Aristotle are women same or lower than men?
    - pixl97 6y ago
      And relevant to this conversation, software can be written anywhere especially if you need alternative ethics.
  - makomk 6y ago
    Yeah, the problem with this idea is that it assumes there's one fixed set of right ethical values everyone should agree on. Take for example the actual, original Hippocratic Oath - it required, amongst other things, doctors not to provide abortions to women. There's still a large number of Americans who hold this ethical belief, but I'd venture a guess that almost all the people pushing for ethics in software development hold the exact diametrically opposite view. A lot of the demands for software ethics get into even fuzzier and more complicated territory, like whether the companies which control our mass communications should use that control to decide which political views people should be able to share and what they should target, whether this is good or bad for democracy, etc. Also, many of them seem to be things that would've been incredibly niche viewpoints even a decade ago. I've seen people in another thread thinking that Google promised not to use e-mail contents to sell ads originally and then snuck that in, but in reality basing ads on the contents of e-mails was part of their business model all along and it's just that no-one cared when they launched because it was so much less obnoxious than the ads on other providers. Somewhere along the way, we got this meme about big tech selling our personal information, and everyone seems to project it backwards in time onto how people felt before the meme.
- Polylactic_acid 6y ago
  Ethics are subjective. Plenty of people working at facebook will tell you their work is ethical.
- jimmySixDOF 6y ago
  That's why Smith also talks about a new DIGITAL GENEVA CONVENTION - not just a code for devs - so there is a defined framework for everyone in the chain. Pls excuse the caps, but too many people seem to ignore this very important part of his argument here.
  - Nasrudith 6y ago
    I thought everyone ignored it because that is what would happen in practice. Like the actual Geneva convention only applying in an unconditional surrender or complete defeat.
p1necone 6y ago
I think the point is that we can't just rely on individual ethics to enact change. People have bills to pay and kids to feed, if it's all on the man at the bottom to say no then a lot of bad software is still going to be created.
- Legogris 6y ago
  The argument made is that it's necessary, not that it's sufficient.
- jrumbut 6y ago
  I think making a statement as an industry helps avoid some of this stuff even being asked. With research you can easily find doctors who are brought in to assist with torture or other heinous crimes, but in general I would expect it to be harder to find a doctor to help with my project of creating a more addictive cigarette versus finding a web developer to help me market them. There's some (not a lot, but some) power in a profession just claiming to have ethics, even if there is no enforcement and change in the hearts of practitioners. Plus, at some point, we all just have to own what we do with our lives. It's great to advocate for systemic change but doing your own small part is at least as important.
  - wolco 6y ago
    You would lose the ability to practise medicine as a license is required. Do you want to license software developers?
    - p1necone 6y ago
      I think we should license some software developers - if you're developing software to be used in industries where the people using your software need to be licensed (i.e medical equipment, structural engineering, industrial control systems) then perhaps the software engineers should be similarly rigorously qualified. But if you're working on video games, or pizza delivery, or really probably like 90% of software then no.
Tehdasi 6y ago
The consequences of the decisions made by Microsoft Presidents and their ilk, since they are the ones who actually decide what software will look like. But still it would be good when MS gets found to do unethical things that a programmer could be scapegoated instead of their president, at least from the presidents perspective.
gonzo41 6y ago
Are you expecting engineers to quit FB and IG and Twitter en masse? Because I'm not. People like money more than each other.
- imtringued 6y ago
  It wouldn't surprise me if there are people specializing in ethics violations in exchange for lots of money.
Nasrudith 6y ago
Well there is one major assumption error there - that negative real world consequences are only linked to negative ethics. That is so wrong it isn't even funny. If the car was invented as powered by a Mr. Fusion the buggy whip makers going out of business would be a negative real world consequence.
- imtringued 6y ago
  Cars made us wealthier but they also contributed to obesity. It's not exactly clear if reducing obesity by banning cars is worth it.
tomjen3 6y ago
In some Engineering societies maybe, in general you may be liable if you construct a building you shouldn't have. Software is fundamentally different because until you run it it has no consequences, and even if you run it, it can be contained. I can write a worm and not release it on world. In that regard, it is more like engineering _plans_. I can draw up plans for a building that is designed to collapse with X number of persons inside -- in fact I can imagine either of the two assignments given as an exercise in University. No reason to make more laws: it should be immaterial whether I chop down the Christmas tree at the local town square or program a robot to do it.
- tech2 6y ago
  It should be immaterial whether you chop down the Christmas tree or set a pre-programmed robot capable of doing the same on its way. The programming of said robot isn't the bad act here, it's the act the robot actually performs. The danger of being castigated for having written something that wasn't used is that we then get into the area of thought-crimes.

asperous 6y ago

I agree, however not all ethics can be legislated. This sort of thing works in other professions like medicine because malpractice can cause doctors to lose their license. Same with civil engineers. This changes things because the choice is now quitting or possibly never being able to work in the field again. Perhaps principal software engineers in charge of life or death software should be licensed for accountability, “engineer on record”.

optimiz3 6y ago
Doctors don't typically have bosses directing how they treat patients; Doctors have full agency, front line grunt workers do not.
- SpicyLemonZest 6y ago
  But that's not an independent fact. That's one of the major goals of medical oaths; they establish that doctors must be able to determine the right course of action on their own, that ethical doctors won't work for an organization that won't give them that agency.
- TeaDrunk 6y ago
  I don’t think this is true in America. Doctors are limited on what will benefit the system monetarily. Healthcare administrators will decline suggested procedures from doctors due to expense.

walrus01 6y ago

> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. I agree to a very limited extent about the hierarchical nature of a typical corporation, but I also disagree. Software engineers at a certain level of their career and with relatively uncommon skills can pick and choose what companies they want to work at. In my opinion people of good moral character and conscience need to be prepared to refuse to accept a position at companies known to engage in activities against their principles. And further need to be prepared to resign if they are asked to do something clearly unethical. From my particular specialization in network engineering, I would never accept a role at an ISP in an environment where I had to implement something like the GFW in China, or further walled-garden/censorship of the global Internet. It's directly contradictory to my principles. I sincerely hope that the best and the brightest of my colleagues would never choose to aid and abet internet-fuckery by autocratic regimes. If people from my field look at a project and could reasonably say "Vint Cerf would be really disappointed if he saw me implementing this...", I hope they will choose to walk away.

peteradio 6y ago
Would you accept any job under penalty of imprisonment for violating grey ethical rules?
- walrus01 6y ago
  Under which nation-state's legal system are we basing this theoretical scenario?
xdavidliu 6y ago
> Software engineers at a certain level of their career and with relatively uncommon skills can pick and choose what companies they want to work at. What percent of software engineers does this statement apply to?

tfehring 6y ago

In sufficiently complex systems, you can always blame someone else if you're motivated to do so. https://www.youtube.com/watch?v=IssR_J0QWr4 https://www.youtube.com/watch?v=IssR_J0QWr4 But ultimately there's always a software engineer involved in the creation of software - and that's not true of any of the other roles you mentioned. Since software engineers are necessary and sufficient to produce software, they should always be held responsible, and any oath should fall on engineers. > To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. Well, yes - if there were no tradeoffs there would be no point in having an oath to begin with. But there are software engineers today, including some on HN, who do things more harmful and unethical than medical malpractice, and they are personally culpable for the decision to do so - just as their replacements would be if they refused. I would also like to see laws criminalizing those individual engineers' conduct - maybe you're alluding to the same thing? - but an oath is a good start.

Aerroon 6y ago
How about a compromise then? Software engineers are responsible for the negative consequences of the software. Any and all responsibility they take is then also shared with every person that is above them in the hierarchy. Eg a developer does something and society finds this unethical and punishes them. The developer's boss, the boss's boss, the boss's boss's boss etc up to the CEO all get punished in the same way. Furthermore, to avoid companies trying to shield themselves from this by putting their developers into a different company, it will apply to software that you get from someone else too. Suddenly this doesn't sound very appealing anymore, does it?
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- bjo590 6y ago
  > Any and all responsibility they take is then also shared with every person that is above them in the hierarchy. Currently, if I write software that performs illegal actions -- let's say software that allows me to use unlicensed Adobe products -- at the request of my boss and their boss, all three of us would be legally liable.

sjwright 6y ago

Corporate strategists blaming software engineers for the consequences of corporate strategy is a fairly brazen kind of blame-shifting. A system of ethics within the health system are necessary for customers to retain trust in the health industry. It's also strongly aligned with the selfish interests of workers who must enact that system of ethics. These properties do not neatly translate to software engineering—mostly because the most difficult ethical dilemmas in technology are rarely obvious when looking at source code. The problems with Facebook (for example) are not always inherent in code; many are only revealed after being deployed at scale and external groups begin exploiting the system.

peterkelly 6y ago

> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be I've quit jobs in the past because of ethical concerns about the way in which those above me have been acting. In one case this involved bribery of senior government officials to push through a project that put at risk the privacy of hundreds of thousands of people. If you go along with shit like that, you're an accomplice and share partial responsibility. As professionals we have a responsibility to stand up for what is right. It's not good enough to fall back to the lazy excuse of "just doing my job".

pixl97 6y ago
And I mean, that is good of you to have a moral backbone, unfortunately there are many people behind you that will do the same job in software and they can be located anywhere in the world.
- walrus01 6y ago
  In the current state of things, that is factually true. However the same argument could be applied to labor abuses in the textile/garment and shoe manufacturing industry. Most people who follow news are probably aware that about 20 years ago Nike, Adidas and other brands went through a period of terrible public relations disasters, after the working conditions in some of their shoe factories in developing nations were exposed by journalists. The argument that could have been made at that time would also have been "well but if we don't employ these people, somebody else will just do the same thing with equally terrible labor/human rights violations somewhere else in the world, with even cheaper factories". The situation today is not great, but it is significantly improved from how it was twenty years ago. There are third party neutral inspection/oversight agencies. Companies in the garment industry are forced to make public commitments to labor rights and reasonable working conditions, and to allow external auditing. They can't just hand wave away the problem and say "but if we don't do it , someone else will.."
  - quickthrower2 6y ago
    The more abstract the problem the less people care. Police suffocating someone to death > 1,000,000 people being subtly spied on. "I got nothing to hide" after all.

tlb 6y ago

The Hippocratic oath is similarly local in scope. Individual doctors try hard not to cause harm to individual patients, but the medical establishment causes massive amounts of harm, by: - developing treatments for chronic symptoms instead of curing diseases - being unprepared for pandemics - making health care unaffordable except through employer plans - promoting wrong nutrition guidelines for decades after the evidence was in and more. To have good outcomes you need ethics at both individual and system-wide levels.

replyifuagree 6y ago

Agreed, the vast majority of software engineering disasters can be laid at the feet of bad management, not software engineers. Let's clean house in management first, just like Deming did when he straightened out Ford. https://en.m.wikipedia.org/wiki/W._Edwards_Deming https://en.m.wikipedia.org/wiki/W._Edwards_Deming

BeetleB 6y ago

> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. Software engineers are accountable to themselves before they are accountable to their bosses.

jbay808 6y ago

> To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. This is where a strict licensing requirement, like Canada's P. Eng, can empower the engineer. If you think what you're being asked to do would violate your professional ethics, not only can you decline to do it, but you have a system to ensure that you won't just get replaced by someone who will do it.

chii 6y ago
a licensing system is just a form of regulation as alluded to in the comment you're replying to. And in the end, if software engineers are to conduct themselves in moral and ethical ways, they must be empowered to do so without having to sacrifice their personal wellbeing or livelihood. Regulation, it seems, is the only way to achieve that end.
- austincheney 6y ago
  The necessary regulation is imposed and enforced via licensing. An example simply look at every other professional industry.
jimnotgym 6y ago
Indeed it is the same with Chartered Accountants. Management can make whatever decisions they think they can make, but an accountant won't inact things that are unethical at the risk of their charter and professional standing.
- coldcode 6y ago
  You really believe many people with won't violate their oaths and associations and licenses for enough money? Accountants audited Bernie Madoff and Wirecard for years and "saw nothing". Michael Jackson hired a surgeon to pump him full of drugs for $150K a month. Money > Ethics for a lot of people.
  - 6y ago

deleted 6y ago

[deleted]

dmarchand90 6y ago

To me this argument sounds a bit like a "only following orders" defence. I'm sure smart engineers ,I.e., all of them, can figure out what the application of their work will be.

noncoml 6y ago

Don’t sell out yo FB or Google or some other “not-evil(R)” Corp and you will never have to follow orders from your boss that don’t meet your ethics.

kazagistar 6y ago

Agreed. Let's look at it from a simple emperical standpoint: where does the problem mostly arise, and who has the power to fix it? Organizations that went through true iterative process to reduce failure rate like NASA figured out that they needed to allow true authority to specific domain experts to blow the whistle and not face reprisal or suffer for it. Oaths fix nothing, you need organizational change, and if someone is going to do that, its the management in charge.

znpy 6y ago

not to mention that software is reusable: a technology might be invented with all the good intentions and ideals... and later be used for evil purposes.

sjogress 6y ago

> To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. If we were to introduce an oath we would have to take further inspiration from doctors, e.g having a certification required to do the job, or, failing that at least having an industry-wide union/guild protecting the position.

tech2 6y ago
How then do you manage things like outsourcing, or the purchase of overseas companies, or multinational mergers?

nicoburns 6y ago

> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. Isn't that what a hipppocratic oath would solve. They'd be accountable to the oath before their bosses, and that would give them reasonable grounds to refuse unethical work.

imtringued 6y ago
How do you guarantee that the boss will cooperate with the software engineers and tell them about all the unethical business practices that are currently happening?
- TeaDrunk 6y ago
  Eventually it would have to be told to a CTO, president or Vice President it some technological product, software architect, etc. Software Engineers generally need to understand the use case of a product to be able to develop something effective. It would be exceptionally difficult to convince someone to code something that creates fake profiles of people who don’t use your site and no one have any idea that that’s what’s happening.
- nicoburns 6y ago
  Well if it's something that the software engineers are expected to implement, then they're going to have to be told about it. If it's not, then it seems out of the realm of software engineering.
coldcode 6y ago
... and be an easy way to get rid of any engineer who fails to do the work asked of them. Think you can just get a new one?
- wtetzner 6y ago
  The Hippocratic oath for software engineers should come along with legislation that makes it illegal to fire a software engineer for refusing to break the oath.

jlnthws 6y ago

Exactly. An oath means nothing: what you need is skin in the game. Some health practitioners are literally bought by Big Pharma, by their hospital accountant, etc. How would an oath fix that? Same with engineers or any other discipline. You need to make sure that everyone in the process has skin in the game. For me it's less about control (legislation) than about responsibility and accountability (assessments, eating your own dog food).

ethanbond 6y ago
What does “skin in the game” mean precisely in this context?
- jlnthws 6y ago
  Having personal exposure, taking accountability for your actions. From Taleb of course. The case of Snowden shows how bad it can turn when not everybody in the loop has skin in the game (asymmetry). Ironically, his behavior tells us he has been faithful to some kind of oath, but apparently none of his coworkers or supervisors.

bryanrasmussen 6y ago

>To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. obviously for a hippocratic type oath to work you need the same kind of system in place for qualifying engineers that you do for doctors and not allowing anyone to work as an engineer who failed the ethics board.

eie7eu3hrruur 6y ago

I think people assume the Hippocratic oath actually makes a difference. It's culturally important to many doctors of course, but how often has that hindered the military or the CIA from hiring enough doctors to facilitate a new interrogation program? Of the illegal human experiments that happened in the US many had doctors working for them as well. It's a good guiding principle but the idea that it would actually make an impact of any kind is debateable. It doesn't matter if 90% of coders follow the oath if 10% or even 5% are enough to handle the demand for oath breaking.

Cthulhu_ 6y ago

Pretty much. Those that take the oath would quickly see themselves not hired for certain positions, in favor of those that didn't take the oath.

TeaDrunk 6y ago
This assumes all of the vast majority of engineers are easily replaceable. Given that several tech companies prefer the chance of not hiring a good fit over the chance of needing to fire someone, I think the cost to a company if a significant subset of its engineers declined unethical work (especially engineers who at this point are high enough to have a good idea of the broad scope of the software) would make it difficult to perform what you suggest.

bumby 6y ago

Maybe the doctor analogy is flawed, but isn’t there ethics of a licensed (capital P) Professional Engineer more apt? They are responsible to their bosses as well as the public who are the end users of their designs/products

coffeefirst 6y ago

This thread is littered with this attitude, that the developers don't actually have a seat at the table where "what to build" decisions are made. This was not always the case. How the hell did we let that happened?

xnyan 6y ago
Here's a prospective from someone who became a software dev in my late 20s after other jobs. Relatively speaking, if you can write working code and show up on time, you have a lot of leverage in getting and keeping employment, more than most other middle to upper middle class professions. This means org politics has less effect on you, and less incentive to get involved. As a dev you can, but don't have to, think as much about the politics and operations of your org for all kinds of reasons. You are relatively harder to replace so internal politics tends to matter less, and if the org makes decisions you don't like, you can be confident that you can leave and find something else versus the long and often unfruitful process of trying to change an org from within. Politics can and often is messy, how often have you heard something like "I just want to build things" (it's how I feel for sure), if you can get paid well to do that, why get involved with a messy decision making process?
- 52-6F-62 6y ago
  To extend upon this (though personally I feel a little differently than yourself, but inclusive of how you feel as well): My personal experience has seen individual contributing software developers have little voice in the matter regardless. Outside of "tech"-forward companies they are of little consequence in the larger political structure of a company. I've seen it and been it—speak up about a desired direction, voice concerns about a decided direction, concerns about faulty legacy software, etc. Those voices, unless amplified by political clout mean little to nothing to anyone else. In a lot of organizations, title is everything when it comes to moving discussion. So my thinking, if we're continuing the comparison to the medical profession, is hierarchies must follow similarly. A head of Surgery in a hospital ward is going to be a doctor. Hospital directors are going to be doctors. Sure, the CEO may not be, but they rely on the expertise of their directors. If the directors don't follow the same code as the rest of the professionals under them, then they can theoretically impose any ethics they choose and the onus falls to the IC/surgeon/etc which for the intended purposes of the oath/license/regulation at all is tenuous.

mcv 6y ago

The oath should be for everybody involved in the process. After the 2008 mortgage crisis, Netherland required everybody working at banks to take the banker's oath, which is mostly about balancing the interests of the 4 main stakeholders of the bank: shareholders, customers, employee, and society. It's pretty broad, it doesn't magically fix everything, but it does make everybody more aware of their responsibilities. Maybe software companies should require something similar, where everybody needs to be aware of their responsibilities towards, well, primarily user data, I guess. And that goes for not just software engineers themselves, but for everybody involved in the process.

GordonS 6y ago
Where do you draw the line though? Does everyone employed in any capacity at any kind of business have to take an oath? It seems to me that if everyone has to promise to do no evil, the meaning of such oaths would become diluted.
- mcv 6y ago
  Do all those businesses have a problem with unethical behaviour causing serious problems in society? Doctors, bankers and apparently software engineers have a pretty big impact on society, and often in ways that aren't very transparent to most other people. It's quite possible there are other professions that have a similar impact, but I'm pretty sure it's not all. I think registered accountants also have some sort of oath, again because various stakeholders including society as a whole has to be able to trust them.

Heliosmaster 6y ago

There is the problem that everything can be subdivided in a bunch of menial task so every cog in the system isn't aware of their impact. It already happens a lot in software. the idea of Hippocratic Oath reminds me of Asimov's Three Laws of Robotics in "The Naked sun" (SPOILERS ahead): the detective realises that the normally quoted First Law of Robotics ("A robot may not injure a human being or, through inaction, allow a human being to come to harm.") is actually just an approximation, he argues that the real Law is "A robot may do nothing that, TO ITS KNOWLEDGE, will harm a human being; nor, through inaction, KNOWINGLY allow a human being to come to harm." This is important because even though robots really try their best, different robots could perform sub-tasks that look very harmless by themselves, but combined kill a human being: - A robot is instructed to pour this bottle of poison into a caraffe of water and then leave the room - Another robot is instructed to enter the room, take the caraffe of water and give it to a human to drink The human is poisoned, but none of the robots are directly responsible (in the first law sense). Is the act of connecting the two dots the evil deed.

DetroitThrow 6y ago
> There is the problem that everything can be subdivided in a bunch of menial task so every cog in the system isn't aware of their impact. It already happens a lot in software. Precisely the issue with the original proposal. Would it have mattered whatsoever if PhD's took a Hippocratic oath when developing the Manhattan project? I feel like this MSFT executive may already know that swearing engineer to "do no harm" is fruitless after reading the article, but it's still unfortunate that statements like his diverts attention from more meaningful proposals.

giancarlostoro 6y ago

Kind of funny to hear this from a company known for harvesting tons and tons of telemetry from customers, with no true way to fully opt-out which is pretty damn unethical and probably feeds their AI.

Zenbit_UX 6y ago

> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will. That isn’t setting us up for success. The nazi officers who committed most of the atrocities used similar arguments. "I was just following orders!" I expect better from a software engineer on hacker news. You've single handedly convinced most here - through your weak logic - that such an oath is necessary.

watwut 6y ago
But we still blame the Nazi leadership the most. The nazi officers you talk about here were also in fact management typically.

deleted 6y ago

[deleted]

Osiris 6y ago

Doctors can have their licenses revoked. Software engineers cannot. There's no downside to being involved in unethical businesses.

GuB-42 6y ago

Agree. A software engineer is more like a chemist working for the pharmaceutical industry than a doctor treating patients. And chemists typically don't have an Hippocratic Oath. Pharmacists sometimes have their own version, but it is mostly about giving good advise to patients and respecting them as human beings. But it doesn't stop the pharmaceutical industry from being heavily regulated, and while their business practices are often criticized, the drugs that come out of it are generally safe and effective. Many countries also have regulations making important drugs (ex: vaccines) accessible to everyone.