6 ms·

The programmer still decides whether that code gets written, since they’re the one writing it! If you write or review a piece of software, even if the spec was

The programmer still decides whether that code gets written, since they’re the one writing it! If you write or review a piece of software, even if the spec was written by the PM/business, you’re endorsing whatever that spec says and all of its ethical implications. “Just following orders” is a famously poor defense at this point.

ryandrake 6y ago
Exactly. I can’t believe all the blame-shifting I’m reading in this thread! It’s as if software engineers are suddenly these powerless victims, lacking agency over their work, only capable of saying “yes, boss, whatever you say, boss!” If a civil engineer’s manager told them to design an unsafe building or bridge, they’re not going to just say, “Sure thing manager! One death trap coming right up!” It is their ethical duty to build it safely.
- pixl97 6y ago
  Eh, this isn't a great analogy either. If I'm an engineer that develops a single beam in a bridge, is it my fault if someone assembles those beams together in such a way that is dangerous? At which point does a function become unethical? Do you now have to only use software made in your country by ethical developers? Software isn't a bridge and comparisons fall apart quickly.
  - ryandrake 6y ago
    Wow, I’m realizing what an unpopular opinion this is on HN! Yes, as a software developer you should absolutely be accountable for the ethical concerns around what that protobuf you’re moving around from one API layer to the other is used for. You’re not a code monkey. Ask, and refuse if it’s unacceptable. I have quit jobs where the ultimate purpose of what I was building was evil. EDIT: Forum the original OP: > Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. They are accountable to themselves and their own conscience before both their bosses and their users. I understand this is an uncomfortable line of thinking if your employer asks for ethically questionable project work, but I’d argue that if this is the case for you, it warrants career introspection.
    - deleted 6y ago
      [deleted]
    - deleted 6y ago
      [deleted]
    - bcrosby95 6y ago
      So the people that wrote java.util.List are liable for all unethical software that makes use of java.util.List?
      - jon-wood 6y ago
        No, of course they're not. Just like the scientists that developed morphine aren't responsible for a doctor later using it to kill a patient.
        ryandrake 6y ago
        Sure, same as the LLVM example someone else pointed out. Good points. I’ll qualify my opinion then. To the extent that the engineer can know the ultimate application of their work, he or she should be responsible for ensuring it is being used ethically. So, the engineer writing a binary search, knowingly working on “Project Orbital Death Ray” or “Voter Suppression 2.1” should know better. I hope we can at least agree on that one. The engineer writing a linked list or moving around Protobufs for their some open source toolset gets a pass because their project as they understand it is ethically neutral. BUT there will be that engineer who then takes those tools and integrates it into “Project Orbital Death Ray”. That’s maybe where accountability should begin. Everyone’s talking about the managers taking the blame and yes they’re culpable too. But at the end of the day an actual software developer’s fingers type the code in. If that developer knows what he is working on, he needs to bear the responsibility, too.
    - imtringued 6y ago
      There is a gigantic difference between building an unethical software product and abusing an ethical software product for unethical purposes. The developer is not the user. Do you not understand that?
  - jbay808 6y ago
    Engineers at the beam companies just certify that the beam meets its specs. I'm not sure why software would be any different. Bridges are complicated and made up of versatile submodules, just like software. Some other software engineer eventually designs the "bridge" and selects "beams" for the structure. If those beams fail to meet their specs, then the engineers who stood by them are at fault. If the bridge fails because the beams weren't used in accordance with their spec, or didn't have a spec at all, then the engineers who approved their use in the bridge is at fault.
    - imtringued 6y ago
      What if the engineer isn't allowed to certify his beams to save on costs? That's basically what happened with 737 MAX.
      - jbay808 6y ago
        Then the engineer designing the bridge should probably choose a different beam supplier who can stand by their product.
- erik_seaberg 6y ago
  A bridge is limited to a single purpose, like an appliance. If you insist on veto power over every outcome of what you build, that means you can only ever build sealed appliances for hapless consumers, not unfettered tools that empower clever human beings who will use them in unanticipated ways. Having sworn a Hippocratic oath, are you allowed to work on LLVM, which half of all evil apps probably depend on? I could get behind a requirement that code be reliable and fit for purpose, though very few of us have any experience with the formal methods that might get us there, and most don't want to work that way.
- imtringued 6y ago
  Imagine if your manager copied the safe bridge you designed with a magic replicator and now uses that exact same design somewhere else. You tell him that the bridge was not designed for this location and that the bridge will collapse in 5 years. Your boss fires you but you are still responsible for the collapse of the bridge. Let's go further into absurdity. The engineer is kidnapping the daughter of the manager and blackmailing the manager to take the bridge down. Is it ethical to force someone else to be ethical even if its only possible through unethical means? What if there is a hero saves the daughter? Will the hero be liable for the collapsed bridge?
- pixelbash 6y ago
  This argument is more akin to saying its the builders responsibility to decide whether a bad architectural decision should be built or not. They might bring it up, but it's not really expected that they get to decide.
Ygg2 6y ago
Except that the start and goal are so far away you can legitimately claim you can't see the actual harm in writing something. You write a tool for let's say recognizing faces. Will it be used for login onto computer? Tracking dissidents? IDing corpses? Who knows. What if you start as something 100% ethical. But your company pivots to unethical application?
- tfehring 6y ago
  This is a great point, and reasonable people can disagree about when an application's abuse or potential for abuse crosses the line. The same goes for pivots or for general-purpose code that's used elsewhere. (Is it ethical to contribute to internal tools at Facebook? What if those tools make other engineers way more effective at doing things that ultimately undermine democratic systems?) My point here isn't to dictate what software is or isn't ethical, but to argue that if a program is unethical, its ethical implications are the responsibility of the engineer(s) who wrote it.
  - deleted 6y ago
    [deleted]
- BeetleB 6y ago
  > You write a tool for let's say recognizing faces. Will it be used for login onto computer? Tracking dissidents? IDing corpses? Who knows. I mentioned this in another comment, but I'll say it again: Irrespective of any legal/ethical concerns, yes, I would like to know! If my boss just came to me and said "build a facial recognition system" I certainly will ask how it is going to be used. Not because I care about ethics, but it's a basic aspect of the job. You can replace "facial recognition" with "CMS" and I'd still ask. If they tell me the facial recognition is for logging into computers, and then later decide to use it to track dissidents, that is a different concern. But I'll at least ask! > What if you start as something 100% ethical. But your company pivots to unethical application? If they pivot after my work is done, I won't feel responsible. If they never used it for the original application and pivoted to this, I may get upset and quit, but my conscience would be clear.
deleted 6y ago
[deleted]
marmaduke 6y ago
Are you kidding? Do you like to get paid? Btw I have tried this, and I was just replaced by someone who would just follow orders. Then I get to come in after and clean up the mess ..

ryandrake 6y ago

Exactly. I can’t believe all the blame-shifting I’m reading in this thread! It’s as if software engineers are suddenly these powerless victims, lacking agency over their work, only capable of saying “yes, boss, whatever you say, boss!” If a civil engineer’s manager told them to design an unsafe building or bridge, they’re not going to just say, “Sure thing manager! One death trap coming right up!” It is their ethical duty to build it safely.

pixl97 6y ago
Eh, this isn't a great analogy either. If I'm an engineer that develops a single beam in a bridge, is it my fault if someone assembles those beams together in such a way that is dangerous? At which point does a function become unethical? Do you now have to only use software made in your country by ethical developers? Software isn't a bridge and comparisons fall apart quickly.
- ryandrake 6y ago
  Wow, I’m realizing what an unpopular opinion this is on HN! Yes, as a software developer you should absolutely be accountable for the ethical concerns around what that protobuf you’re moving around from one API layer to the other is used for. You’re not a code monkey. Ask, and refuse if it’s unacceptable. I have quit jobs where the ultimate purpose of what I was building was evil. EDIT: Forum the original OP: > Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. They are accountable to themselves and their own conscience before both their bosses and their users. I understand this is an uncomfortable line of thinking if your employer asks for ethically questionable project work, but I’d argue that if this is the case for you, it warrants career introspection.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
  - bcrosby95 6y ago
    So the people that wrote java.util.List are liable for all unethical software that makes use of java.util.List?
    - jon-wood 6y ago
      No, of course they're not. Just like the scientists that developed morphine aren't responsible for a doctor later using it to kill a patient.
      - ryandrake 6y ago
        Sure, same as the LLVM example someone else pointed out. Good points. I’ll qualify my opinion then. To the extent that the engineer can know the ultimate application of their work, he or she should be responsible for ensuring it is being used ethically. So, the engineer writing a binary search, knowingly working on “Project Orbital Death Ray” or “Voter Suppression 2.1” should know better. I hope we can at least agree on that one. The engineer writing a linked list or moving around Protobufs for their some open source toolset gets a pass because their project as they understand it is ethically neutral. BUT there will be that engineer who then takes those tools and integrates it into “Project Orbital Death Ray”. That’s maybe where accountability should begin. Everyone’s talking about the managers taking the blame and yes they’re culpable too. But at the end of the day an actual software developer’s fingers type the code in. If that developer knows what he is working on, he needs to bear the responsibility, too.
  - imtringued 6y ago
    There is a gigantic difference between building an unethical software product and abusing an ethical software product for unethical purposes. The developer is not the user. Do you not understand that?
- jbay808 6y ago
  Engineers at the beam companies just certify that the beam meets its specs. I'm not sure why software would be any different. Bridges are complicated and made up of versatile submodules, just like software. Some other software engineer eventually designs the "bridge" and selects "beams" for the structure. If those beams fail to meet their specs, then the engineers who stood by them are at fault. If the bridge fails because the beams weren't used in accordance with their spec, or didn't have a spec at all, then the engineers who approved their use in the bridge is at fault.
  - imtringued 6y ago
    What if the engineer isn't allowed to certify his beams to save on costs? That's basically what happened with 737 MAX.
    - jbay808 6y ago
      Then the engineer designing the bridge should probably choose a different beam supplier who can stand by their product.
erik_seaberg 6y ago
A bridge is limited to a single purpose, like an appliance. If you insist on veto power over every outcome of what you build, that means you can only ever build sealed appliances for hapless consumers, not unfettered tools that empower clever human beings who will use them in unanticipated ways. Having sworn a Hippocratic oath, are you allowed to work on LLVM, which half of all evil apps probably depend on? I could get behind a requirement that code be reliable and fit for purpose, though very few of us have any experience with the formal methods that might get us there, and most don't want to work that way.
imtringued 6y ago
Imagine if your manager copied the safe bridge you designed with a magic replicator and now uses that exact same design somewhere else. You tell him that the bridge was not designed for this location and that the bridge will collapse in 5 years. Your boss fires you but you are still responsible for the collapse of the bridge. Let's go further into absurdity. The engineer is kidnapping the daughter of the manager and blackmailing the manager to take the bridge down. Is it ethical to force someone else to be ethical even if its only possible through unethical means? What if there is a hero saves the daughter? Will the hero be liable for the collapsed bridge?
pixelbash 6y ago
This argument is more akin to saying its the builders responsibility to decide whether a bad architectural decision should be built or not. They might bring it up, but it's not really expected that they get to decide.

Ygg2 6y ago

Except that the start and goal are so far away you can legitimately claim you can't see the actual harm in writing something. You write a tool for let's say recognizing faces. Will it be used for login onto computer? Tracking dissidents? IDing corpses? Who knows. What if you start as something 100% ethical. But your company pivots to unethical application?

tfehring 6y ago
This is a great point, and reasonable people can disagree about when an application's abuse or potential for abuse crosses the line. The same goes for pivots or for general-purpose code that's used elsewhere. (Is it ethical to contribute to internal tools at Facebook? What if those tools make other engineers way more effective at doing things that ultimately undermine democratic systems?) My point here isn't to dictate what software is or isn't ethical, but to argue that if a program is unethical, its ethical implications are the responsibility of the engineer(s) who wrote it.
- deleted 6y ago
  [deleted]
BeetleB 6y ago
> You write a tool for let's say recognizing faces. Will it be used for login onto computer? Tracking dissidents? IDing corpses? Who knows. I mentioned this in another comment, but I'll say it again: Irrespective of any legal/ethical concerns, yes, I would like to know! If my boss just came to me and said "build a facial recognition system" I certainly will ask how it is going to be used. Not because I care about ethics, but it's a basic aspect of the job. You can replace "facial recognition" with "CMS" and I'd still ask. If they tell me the facial recognition is for logging into computers, and then later decide to use it to track dissidents, that is a different concern. But I'll at least ask! > What if you start as something 100% ethical. But your company pivots to unethical application? If they pivot after my work is done, I won't feel responsible. If they never used it for the original application and pivoted to this, I may get upset and quit, but my conscience would be clear.

deleted 6y ago

[deleted]

marmaduke 6y ago

Are you kidding? Do you like to get paid? Btw I have tried this, and I was just replaced by someone who would just follow orders. Then I get to come in after and clean up the mess ..