4 ms·

If you know your software will increase emissions to illegal levels, and still write it, you are culpable. If you write software for autonomous cars that is no

If you know your software will increase emissions to illegal levels, and still write it, you are culpable. If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable. In aviation, catastrophic failure means at least one death and the system should have an overall probability of failure at 10^-9. Critical software is frequently written at one or two orders below. That means 95%, 99%, 99.9%, are not enough.

Milner08 8y ago
Do you understand the difference between a first order effect and a second order effect?
- dekhn 8y ago
  I don't (btw, I'm a software engineer and scientist with extensive training). So I looked it up. Did you mean first order or second order kind of like the way physics people mean it (direct linear relationship with strong effect on the unknown variable). Or the way that MBAs mean it (they have their own definition). Or did you mean structural engineering, same as physics: https://www.quora.com/In-structural-engineering-what-are-first-order-effects https://www.quora.com/In-structural-engineering-what-are-fir... I think what you probably mean is a colloquial interpretation, where first order means something is directly causal in an obvious to see way, where second order is more of an indirect effect that occurs through complex dynamics. Either way, I don't think those terms are well defined, and they're kind of complex enough that using them to argue for morality, and calling people out for not knowing your definitions, makes you look caddish.
  - jonnycomputer 8y ago
    I agree. I doubt that they could mean causally indirect. I think it must mean something like: not used in accordance to their the engineer's intent, especially when that intent covers a a general affordance (e.g. affords communication over long distances), but is being judged for a more specific affordance (affords communication over long distances to order a terror attack). I dunno. I'd have to be a philosopher to work it out.
  - learc83 8y ago
    I think that's spot on. When people say something is a first/ second/nth, order effect they are really just arbitrarily drawing discrete lines in a continuous chain of causality. The problem of where to draw those lines is so difficult that legal systems resort to constructing tests based on whether a "reasonable person" could have foreseen the specific consequence of an act.
slededit 8y ago
> If you know your software will increase emissions to illegal levels, and still write it, you are culpable. Again a first order effect > If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable. Again a first order effect connected directly with the immediate goal. However if you write communications software and people use it to say bad things, those are second order effects. You provided communication software that works as advertised and has beneficial purposes. Others misused it.
stlHusker 8y ago
What are the limits of culpability and what are you responsible for? The article is pushing that definition beyond the first order effects of safety. And the proposed solution is that gate keeping is done by the press and the public. How well do you think that will go? If you write software that replaces someone's job are you "culpable"? If you create a phone and people are killed when someone texts and drives are you "culpable"? If you create a better facial recognition program as part of building a secure access systems and someone uses that system to discriminate are you "culpable"? I find this unending process of creating "victims" just so they can be "saved" as tiresome.
- glitchc 8y ago
  That's what standards are for. The threshold is determined by harm. Software practices are still in their infancy, which is is why it is hard to take them seriously as mature disciplines. Now that software is having a direct impact on people's lives on a large scale, it's time to set some standards, starting with autonomous driving and mass surveillance. Then have people trained to understand those standards and adhere to them as best practices.
- jonnycomputer 8y ago
  I find engineers and entrepreneurs who shirk their responsibilities as moral agents tiresome. If a thief comes and steals money under the mattress meant to pay for your kid's college tuition, and your kid is dropped because she can no longer pay, your kid's loss is a second order effect of the thief's theft, but I still hold the thief responsible* * this is a hypothetical story; please don't save that kind of money under your mattress.
  - slededit 8y ago
    Your argument has some basis in law and I do find it compelling. The nuance to my argument is that the original creator did not have nefarious aims. I believe that does make a difference.
    - CJKinni 8y ago
scarejunba 8y ago
Is the guy who invented valves responsible for the Holocaust? Not trying to be inflammatory. This is specifically trying to find if you think there's a point where culpability disappears. I would appreciate a yes/no answer. There's no nuance to the question. I'm not talking about the specific valves, just the notion of a valve.
deklerk 8y ago
Those are also, as the OP has repeatedly addressed: first order effects.

Milner08 8y ago

Do you understand the difference between a first order effect and a second order effect?

dekhn 8y ago
I don't (btw, I'm a software engineer and scientist with extensive training). So I looked it up. Did you mean first order or second order kind of like the way physics people mean it (direct linear relationship with strong effect on the unknown variable). Or the way that MBAs mean it (they have their own definition). Or did you mean structural engineering, same as physics: https://www.quora.com/In-structural-engineering-what-are-first-order-effects https://www.quora.com/In-structural-engineering-what-are-fir... I think what you probably mean is a colloquial interpretation, where first order means something is directly causal in an obvious to see way, where second order is more of an indirect effect that occurs through complex dynamics. Either way, I don't think those terms are well defined, and they're kind of complex enough that using them to argue for morality, and calling people out for not knowing your definitions, makes you look caddish.
- jonnycomputer 8y ago
  I agree. I doubt that they could mean causally indirect. I think it must mean something like: not used in accordance to their the engineer's intent, especially when that intent covers a a general affordance (e.g. affords communication over long distances), but is being judged for a more specific affordance (affords communication over long distances to order a terror attack). I dunno. I'd have to be a philosopher to work it out.
- learc83 8y ago
  I think that's spot on. When people say something is a first/ second/nth, order effect they are really just arbitrarily drawing discrete lines in a continuous chain of causality. The problem of where to draw those lines is so difficult that legal systems resort to constructing tests based on whether a "reasonable person" could have foreseen the specific consequence of an act.

slededit 8y ago

> If you know your software will increase emissions to illegal levels, and still write it, you are culpable. Again a first order effect > If you write software for autonomous cars that is not foolproof at detecting pedestrians, you are culpable. Again a first order effect connected directly with the immediate goal. However if you write communications software and people use it to say bad things, those are second order effects. You provided communication software that works as advertised and has beneficial purposes. Others misused it.

stlHusker 8y ago

What are the limits of culpability and what are you responsible for? The article is pushing that definition beyond the first order effects of safety. And the proposed solution is that gate keeping is done by the press and the public. How well do you think that will go? If you write software that replaces someone's job are you "culpable"? If you create a phone and people are killed when someone texts and drives are you "culpable"? If you create a better facial recognition program as part of building a secure access systems and someone uses that system to discriminate are you "culpable"? I find this unending process of creating "victims" just so they can be "saved" as tiresome.

glitchc 8y ago
That's what standards are for. The threshold is determined by harm. Software practices are still in their infancy, which is is why it is hard to take them seriously as mature disciplines. Now that software is having a direct impact on people's lives on a large scale, it's time to set some standards, starting with autonomous driving and mass surveillance. Then have people trained to understand those standards and adhere to them as best practices.
jonnycomputer 8y ago
I find engineers and entrepreneurs who shirk their responsibilities as moral agents tiresome. If a thief comes and steals money under the mattress meant to pay for your kid's college tuition, and your kid is dropped because she can no longer pay, your kid's loss is a second order effect of the thief's theft, but I still hold the thief responsible* * this is a hypothetical story; please don't save that kind of money under your mattress.
- slededit 8y ago
  Your argument has some basis in law and I do find it compelling. The nuance to my argument is that the original creator did not have nefarious aims. I believe that does make a difference.
  - CJKinni 8y ago

scarejunba 8y ago

Is the guy who invented valves responsible for the Holocaust? Not trying to be inflammatory. This is specifically trying to find if you think there's a point where culpability disappears. I would appreciate a yes/no answer. There's no nuance to the question. I'm not talking about the specific valves, just the notion of a valve.

deklerk 8y ago

Those are also, as the OP has repeatedly addressed: first order effects.