5 ms·

Those are first order effects connected directly to the immediate aim of allowing transport over an obstacle. A civil engineer does not consider "who" will cros

Those are first order effects connected directly to the immediate aim of allowing transport over an obstacle. A civil engineer does not consider "who" will cross or what their aims will be. For example a bridge may allow an enemy army to move their tanks faster during wartime. Similarly tech shouldn't have to consider whether someone with unrelated goals will use their tech for unwanted aims.

glitchc 8y ago
If you know your software will increase emissions to illegal levels, and still write it, you are culpable. If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable. In aviation, catastrophic failure means at least one death and the system should have an overall probability of failure at 10^-9. Critical software is frequently written at one or two orders below. That means 95%, 99%, 99.9%, are not enough.
- Milner08 8y ago
  Do you understand the difference between a first order effect and a second order effect?
  - dekhn 8y ago
    I don't (btw, I'm a software engineer and scientist with extensive training). So I looked it up. Did you mean first order or second order kind of like the way physics people mean it (direct linear relationship with strong effect on the unknown variable). Or the way that MBAs mean it (they have their own definition). Or did you mean structural engineering, same as physics: https://www.quora.com/In-structural-engineering-what-are-first-order-effects https://www.quora.com/In-structural-engineering-what-are-fir... I think what you probably mean is a colloquial interpretation, where first order means something is directly causal in an obvious to see way, where second order is more of an indirect effect that occurs through complex dynamics. Either way, I don't think those terms are well defined, and they're kind of complex enough that using them to argue for morality, and calling people out for not knowing your definitions, makes you look caddish.
    - jonnycomputer 8y ago
      I agree. I doubt that they could mean causally indirect. I think it must mean something like: not used in accordance to their the engineer's intent, especially when that intent covers a a general affordance (e.g. affords communication over long distances), but is being judged for a more specific affordance (affords communication over long distances to order a terror attack). I dunno. I'd have to be a philosopher to work it out.
    - learc83 8y ago
      I think that's spot on. When people say something is a first/ second/nth, order effect they are really just arbitrarily drawing discrete lines in a continuous chain of causality. The problem of where to draw those lines is so difficult that legal systems resort to constructing tests based on whether a "reasonable person" could have foreseen the specific consequence of an act.
- slededit 8y ago
  > If you know your software will increase emissions to illegal levels, and still write it, you are culpable. Again a first order effect > If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable. Again a first order effect connected directly with the immediate goal. However if you write communications software and people use it to say bad things, those are second order effects. You provided communication software that works as advertised and has beneficial purposes. Others misused it.
- stlHusker 8y ago
  What are the limits of culpability and what are you responsible for? The article is pushing that definition beyond the first order effects of safety. And the proposed solution is that gate keeping is done by the press and the public. How well do you think that will go? If you write software that replaces someone's job are you "culpable"? If you create a phone and people are killed when someone texts and drives are you "culpable"? If you create a better facial recognition program as part of building a secure access systems and someone uses that system to discriminate are you "culpable"? I find this unending process of creating "victims" just so they can be "saved" as tiresome.
  - glitchc 8y ago
    That's what standards are for. The threshold is determined by harm. Software practices are still in their infancy, which is is why it is hard to take them seriously as mature disciplines. Now that software is having a direct impact on people's lives on a large scale, it's time to set some standards, starting with autonomous driving and mass surveillance. Then have people trained to understand those standards and adhere to them as best practices.
  - jonnycomputer 8y ago
    I find engineers and entrepreneurs who shirk their responsibilities as moral agents tiresome. If a thief comes and steals money under the mattress meant to pay for your kid's college tuition, and your kid is dropped because she can no longer pay, your kid's loss is a second order effect of the thief's theft, but I still hold the thief responsible* * this is a hypothetical story; please don't save that kind of money under your mattress.
    - slededit 8y ago
      Your argument has some basis in law and I do find it compelling. The nuance to my argument is that the original creator did not have nefarious aims. I believe that does make a difference.
      - CJKinni 8y ago
- scarejunba 8y ago
  Is the guy who invented valves responsible for the Holocaust? Not trying to be inflammatory. This is specifically trying to find if you think there's a point where culpability disappears. I would appreciate a yes/no answer. There's no nuance to the question. I'm not talking about the specific valves, just the notion of a valve.
- deklerk 8y ago
  Those are also, as the OP has repeatedly addressed: first order effects.
onceKnowable 8y ago
Yeah but that’s a different argument to what you’re replying to. If that civil engineer sold or allowed (for any reason from negligence, insufficient security protocols, malice, ignorance etc) blueprints for his design to end up in enemy hands, he most certainly would be held liable. The thrust of that point is the very idea that software engineers on projects involving user data, regardless of how and what they think they are building, are not simply building “bridges”. They are building systems that have the potential to be misused as mass-surveillance apparatuses. And they need to build their systems accordingly, with appropriate data-security and with acknowledgement that their work could leave them personally liable to action should their work be found to be used as a threat-vector towards their users. The current attitude of “well people know what they’re signing up for and since they signed the User Agreement, we don’t have any liability here” towards users in the industry is sick.
- smoony 8y ago
  I literally signed up for HN, something I've been uninterested in doing, just because of this comment. Saying that engineers need to make responsible decisions in their software design is one thing. But you imply that misuse is a liability for the designer? So if someone takes an engineer's code and modifies it for malicious intent the engineer is liable? That's absurd. That's like saying the person who designed the airplanes used in 9/11 are partially liable for 9/11.
  - onceKnowable 8y ago
    Absolutely, if that data is stolen due to the designers’ neglect in the security aspects of the systems they signed off as “adequately designed”.
    - smoony 8y ago
      Yup! It just concerns me since this topic has a lot of great discussion but many people are missing the distinction between different types of effects. Sorry if I misunderstood what you were saying! For others who are confused what I mean... Consider: The person who makes a car that someone uses to kill someone by running a bystander over. The engineer isn't responsible. The person who messed up the Prius with the issues where the gas pedal or whatever didn't work. The engineer IS responsible. The first example is analogous to a software engineer writing image recognition software for a safety app. Then someone maliciously taking the software and using it to discriminate via misuse. The engineer is NOT responsible. If someone takes your software (could even be open source) and applies it or modifies it in a malicious way they are responsible not the engineer. The second is example is where if an engineer makes a widget app and has a vulnerability but signs off on it, and then widget purchaser credit card info is stolen. The engineer is responsible for negligence and should be liable for the misuse.

glitchc 8y ago

If you know your software will increase emissions to illegal levels, and still write it, you are culpable. If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable. In aviation, catastrophic failure means at least one death and the system should have an overall probability of failure at 10^-9. Critical software is frequently written at one or two orders below. That means 95%, 99%, 99.9%, are not enough.

Milner08 8y ago
Do you understand the difference between a first order effect and a second order effect?
- dekhn 8y ago
  I don't (btw, I'm a software engineer and scientist with extensive training). So I looked it up. Did you mean first order or second order kind of like the way physics people mean it (direct linear relationship with strong effect on the unknown variable). Or the way that MBAs mean it (they have their own definition). Or did you mean structural engineering, same as physics: https://www.quora.com/In-structural-engineering-what-are-first-order-effects https://www.quora.com/In-structural-engineering-what-are-fir... I think what you probably mean is a colloquial interpretation, where first order means something is directly causal in an obvious to see way, where second order is more of an indirect effect that occurs through complex dynamics. Either way, I don't think those terms are well defined, and they're kind of complex enough that using them to argue for morality, and calling people out for not knowing your definitions, makes you look caddish.
  - jonnycomputer 8y ago
    I agree. I doubt that they could mean causally indirect. I think it must mean something like: not used in accordance to their the engineer's intent, especially when that intent covers a a general affordance (e.g. affords communication over long distances), but is being judged for a more specific affordance (affords communication over long distances to order a terror attack). I dunno. I'd have to be a philosopher to work it out.
  - learc83 8y ago
    I think that's spot on. When people say something is a first/ second/nth, order effect they are really just arbitrarily drawing discrete lines in a continuous chain of causality. The problem of where to draw those lines is so difficult that legal systems resort to constructing tests based on whether a "reasonable person" could have foreseen the specific consequence of an act.
slededit 8y ago
> If you know your software will increase emissions to illegal levels, and still write it, you are culpable. Again a first order effect > If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable. Again a first order effect connected directly with the immediate goal. However if you write communications software and people use it to say bad things, those are second order effects. You provided communication software that works as advertised and has beneficial purposes. Others misused it.
stlHusker 8y ago
What are the limits of culpability and what are you responsible for? The article is pushing that definition beyond the first order effects of safety. And the proposed solution is that gate keeping is done by the press and the public. How well do you think that will go? If you write software that replaces someone's job are you "culpable"? If you create a phone and people are killed when someone texts and drives are you "culpable"? If you create a better facial recognition program as part of building a secure access systems and someone uses that system to discriminate are you "culpable"? I find this unending process of creating "victims" just so they can be "saved" as tiresome.
- glitchc 8y ago
  That's what standards are for. The threshold is determined by harm. Software practices are still in their infancy, which is is why it is hard to take them seriously as mature disciplines. Now that software is having a direct impact on people's lives on a large scale, it's time to set some standards, starting with autonomous driving and mass surveillance. Then have people trained to understand those standards and adhere to them as best practices.
- jonnycomputer 8y ago
  I find engineers and entrepreneurs who shirk their responsibilities as moral agents tiresome. If a thief comes and steals money under the mattress meant to pay for your kid's college tuition, and your kid is dropped because she can no longer pay, your kid's loss is a second order effect of the thief's theft, but I still hold the thief responsible* * this is a hypothetical story; please don't save that kind of money under your mattress.
  - slededit 8y ago
    Your argument has some basis in law and I do find it compelling. The nuance to my argument is that the original creator did not have nefarious aims. I believe that does make a difference.
    - CJKinni 8y ago
scarejunba 8y ago
Is the guy who invented valves responsible for the Holocaust? Not trying to be inflammatory. This is specifically trying to find if you think there's a point where culpability disappears. I would appreciate a yes/no answer. There's no nuance to the question. I'm not talking about the specific valves, just the notion of a valve.
deklerk 8y ago
Those are also, as the OP has repeatedly addressed: first order effects.

onceKnowable 8y ago

Yeah but that’s a different argument to what you’re replying to. If that civil engineer sold or allowed (for any reason from negligence, insufficient security protocols, malice, ignorance etc) blueprints for his design to end up in enemy hands, he most certainly would be held liable. The thrust of that point is the very idea that software engineers on projects involving user data, regardless of how and what they think they are building, are not simply building “bridges”. They are building systems that have the potential to be misused as mass-surveillance apparatuses. And they need to build their systems accordingly, with appropriate data-security and with acknowledgement that their work could leave them personally liable to action should their work be found to be used as a threat-vector towards their users. The current attitude of “well people know what they’re signing up for and since they signed the User Agreement, we don’t have any liability here” towards users in the industry is sick.

smoony 8y ago
I literally signed up for HN, something I've been uninterested in doing, just because of this comment. Saying that engineers need to make responsible decisions in their software design is one thing. But you imply that misuse is a liability for the designer? So if someone takes an engineer's code and modifies it for malicious intent the engineer is liable? That's absurd. That's like saying the person who designed the airplanes used in 9/11 are partially liable for 9/11.
- onceKnowable 8y ago
  Absolutely, if that data is stolen due to the designers’ neglect in the security aspects of the systems they signed off as “adequately designed”.
  - smoony 8y ago
    Yup! It just concerns me since this topic has a lot of great discussion but many people are missing the distinction between different types of effects. Sorry if I misunderstood what you were saying! For others who are confused what I mean... Consider: The person who makes a car that someone uses to kill someone by running a bystander over. The engineer isn't responsible. The person who messed up the Prius with the issues where the gas pedal or whatever didn't work. The engineer IS responsible. The first example is analogous to a software engineer writing image recognition software for a safety app. Then someone maliciously taking the software and using it to discriminate via misuse. The engineer is NOT responsible. If someone takes your software (could even be open source) and applies it or modifies it in a malicious way they are responsible not the engineer. The second is example is where if an engineer makes a widget app and has a vulnerability but signs off on it, and then widget purchaser credit card info is stolen. The engineer is responsible for negligence and should be liable for the misuse.