13 ms·

In sufficiently complex systems, you can always blame someone else if you're motivated to do so. https://www.youtube.com/watch?v=IssR_J0QWr4 https://www.youtube

• Aerroon 6y ago
  How about a compromise then? Software engineers are responsible for the negative consequences of the software. Any and all responsibility they take is then also shared with every person that is above them in the hierarchy. Eg a developer does something and society finds this unethical and punishes them. The developer's boss, the boss's boss, the boss's boss's boss etc up to the CEO all get punished in the same way. Furthermore, to avoid companies trying to shield themselves from this by putting their developers into a different company, it will apply to software that you get from someone else too. Suddenly this doesn't sound very appealing anymore, does it?
  • deleted 6y ago
    [deleted]
  • deleted 6y ago
    [deleted]
  • bjo590 6y ago
    > Any and all responsibility they take is then also shared with every person that is above them in the hierarchy. Currently, if I write software that performs illegal actions -- let's say software that allows me to use unlicensed Adobe products -- at the request of my boss and their boss, all three of us would be legally liable.
    • ivanche 6y ago
      It seems you're conflating illegal and unethical.
      • jki275 6y ago
        Illegal and unethical can be the same -- they don't have to be, but it's not wrong to discuss cases where they are.
  • imtringued 6y ago
    That makes no sense. Management just needs to decide on an ethics standard for the entire company and install an audit process that maintains that standard. That's the entire problem. Don't ask a handful of employees to do some charity.
• wolco 6y ago
  Not sure hou understand how software is made. A programmer doesn't decide what to write, when to write it and they are lucky to be included in how it's made. Programmers get specs and write programs to match those. At no point is it the programmers responsibility to talk about the moral compass of the project and where it fits into society. An oath to do no harm? You first need to give programmers the power to decide the fate of projects on their own the way only a doctor can decide medicine or treatment.
  • justinclift 6y ago
    It's probably not an "either/or" thing. Although it would be optimal for the management/leadership to not be pursuing unethical developments, a software engineer having the fallback of "I can't implement this in good faith" is another layer of defence (to society). It would probably also allow for legal push back against being terminated for refusing to implementing the unethical thing.
  • tfehring 6y ago
    The programmer still decides whether that code gets written, since they’re the one writing it! If you write or review a piece of software, even if the spec was written by the PM/business, you’re endorsing whatever that spec says and all of its ethical implications. “Just following orders” is a famously poor defense at this point.
    • ryandrake 6y ago
      Exactly. I can’t believe all the blame-shifting I’m reading in this thread! It’s as if software engineers are suddenly these powerless victims, lacking agency over their work, only capable of saying “yes, boss, whatever you say, boss!” If a civil engineer’s manager told them to design an unsafe building or bridge, they’re not going to just say, “Sure thing manager! One death trap coming right up!” It is their ethical duty to build it safely.
      • pixl97 6y ago
        Eh, this isn't a great analogy either. If I'm an engineer that develops a single beam in a bridge, is it my fault if someone assembles those beams together in such a way that is dangerous? At which point does a function become unethical? Do you now have to only use software made in your country by ethical developers? Software isn't a bridge and comparisons fall apart quickly.
• paulryanrogers 6y ago
  So management bears no blame for requiring illegal work be done, on pain of termination? Said another way, engineers now need to be technical and legal experts in the business domain? (Remember employees in the US depend on the company for health insurance. Saying 'no' could cost a lot more than just ones position.) Most software engineers are not like doctors. We have little autonomy over what is created. Our responsibility is primarily the how. And with devops sometimes the actual deployment and maintenance itself.
  • skissane 6y ago
    > Said another way, engineers now need to be technical and legal experts in the business domain? Consider something like the 737 MAX debacle – did the programmers writing the MCAS code actually have enough aviation domain knowledge and understanding of where the component fit in the overall system to realise it was a threat to people's lives? I don't know, but my guess is the most likely answer is "No".
    • MattGaiser 6y ago
      I can’t imagine software engineers staying in a company long enough to acquire it. For this to work, software engineer compensation would need to change radically.
      • watt 6y ago
        Maybe that's the point of this oath? Software engineers will refuse to work on these areas, unless demands of the oath can be met.
        • Chris2048 6y ago
          If a doctor says no, it's because of legal liability and risk to licence. For the same reason, it's harder to hire some rando budget doctor because the field is gate-keeped by the requirement of a licence, and liability. You can't magically make Engineering the same without the same conditions. Add barriers top entry that see my pay rise, or have cheap programmers with no liability.
• ezoe 6y ago
  The problem is, software can be really complex it's possible to make it so no one programmer can understand the whole picture. The tasks can be divided so the individual programmers are given order like "do X in condition of Y" which itself seems harmless and lawful, but combining them lead to malicious behaviours.
  • leeoniya 6y ago
    this is precisely the problem. few physicists knew they were all collectively building a nuclear bomb because the big picture was well compartmentalized. feynman only found out because he picked the locks of his colleagues to piece it all together.
    • andi999 6y ago
      Can you point to the part of the book where it is stated that only few physicist (not including Feynman) knew this? I remember it very differently. From my memory: - lock picking was for fun and didnt gain info - some warehouse workers didnt know about critical mass of uranium and stored the stuff to closely together
      • mrunkel 6y ago
        This is my recollection too.
      • leeoniya 6y ago
        ah maybe you're right. memory might be foggy. time for a fresh re-read!
        • andi999 6y ago
          Maybe you meant the part where it is clarified that the weapon was not developed against Germany but strategically against the UDSSR. (if I remember that part correctly)
    • spacemanmatt 6y ago
• cheez 6y ago
  I'd take the oath if there are no more dumbass interviews, I get paid like a surgeon and work 20 hours a week. Until then, management falls on the sword, thanks.
• didibus 6y ago
  > Since software engineers are necessary and sufficient to produce software, they should always be held responsible, and any oath should fall on engineers. That's not true in any way. Lots of software is written by people who don't even have a degree, others by some who have a computer science degree, but not an engineering one, etc. The other issue is that software is rarely unethical. The unethical bit often comes from the way it is used. And I'd have to agree with OP. In an idealist world you could assume software engineers would all be ready to quit their job at any sight of unethical affair, even say, launching something to production with a known vulnerability, or without anything but the most rigorous security review process having passed. But in practice, you're not going to achieve this result, unless you put a framework to incentivize software engineers towards being ethical. If you allowed them to sue their employer, and made it that they more often win the lawsuit, for asking them to build something unethical, or insisting that they do so even after the SE said it was unethical, or to retaliate in any way to an SE refusing to build something on ground of ethics, then you'd maybe start to see results. Otherwise, won't happen, and you've only created a scape goat to make it even easier for companies to push for unethical software, since they can now just blame SE they coerce into building it anyways.
• raxxorrax 6y ago
  I think it something should already kick in if you create tracking pixels to read canvas data to identify users or generally work on fingerprinting. Especially if it is for a benign purpose as advertising, an industry that is notoriously toxic and would have no problems selling ever kind of data they get their hands on. It is fine to generalize it that way in my opinion. The directly conflict with any spirit of the law in most countries regarding privacy. Aside from that, the quantification of attributes/properties of people can have negative implications for many people. Oversharing is a problem on the net, but at least here people just endanger themselves.
• manuelabeledo 6y ago
  > But there are software engineers today, including some on HN, who do things more harmful and unethical than medical malpractice... I'm having a hard time trying to find examples of this, outside the field of armament development. And in those fields where a software failure may result in death, e.g. aircraft development, proof of a software engineer willingly causing it, would likely result in jail time already.
• coliveira 6y ago
  The big question is: who has ultimate visibility on the consequences of a particular project? Very frequently software engineers are asked to work on projects where they only know one side of the picture. The executives in the company are the ones who know the ultimate context of what they're doing.
• kelvin0 6y ago
  Here's how this oath thing would play out: 1) New regulation forces some branches of software engineering to have some type of oath. 2) Now some software jobs will require only oath takers to do. 3) A well payed and powerful new cast of software engineers is born. 4) They are highly paid and have a powerful lobby working for them. 5) The oath takers become very frisky and only work on jobs with minimal risk. The ones that do screw up have an armada of lawyers, because of course they have a new association with deep pockets. 6) Innovation stalls for a while. 7) Big corps start outsourcing some of the oath-taking jobs. These engineers are not bound by the same regulation. Screw ups happen, people die at some point. 8) Maybe we should have the outsourced engineers also take an oath? Back to square 1 This is exactly what I found happened for medical Doctors in Canada (don't know for US). Not saying doctors are not doing a good job, and I can't imagine the stress and pressure they operate from. But suing for malpractice in Canada can be challenging to say the least. I have personal account of a Family member who was grossly mistreated, and all the Doctor did was changed hospitals, nothing more than a slap on the wrist. https://diamondlaw.ca/blog/how-canadian-law-discourages-patients-from-suing-physicians-for-medical-malpractice https://diamondlaw.ca/blog/how-canadian-law-discourages-pati...
• rstuart4133 6y ago
  > But ultimately there's always a software engineer involved in the creation of software - and that's not true of any of the other roles you mentioned. Since software engineers are necessary and sufficient to produce software, they should always be held responsible, and any oath should fall on engineers. Nah, it's not the same at all. The fundamental difference between creating a program and medicine is creating a program only has to be done once, or at least a only by a few. Medicine on the other hand: it has to be redone with each new patient. If the Hippocratic Oath works to prevent 99.9% doctors from doing a harmful procedure then you've hit a home run. Sure, you will never completely stop some bad egg removing a perfectly good limb because a patient suffering from Xenomelia offered enough money. But who wouldn't call a thousand fold reduction a huge win. We demonstrably have the 0.1% of programmers who are willing to break any oath. They make malware, and willingly take out Sony as mercenaries because Kim Jong-Un got pissed off at a movie. All that 0.1% has to do is write the program once. Thereafter you are not trying to discourage hoards of high skilled professional from doing it again, you are trying to stop a legion of dark net operators copying the thing and selling it to anyone. An oath is a waste of time under those circumstances.
• m463 6y ago
  With a combination of local engineers, remote engineers in other countries, mechanical turk and some sleight of hand, I wonder if you could craft a nefarious project where nobody knows the whole picture.