3 ms·

Presumably the realistic cases are not so clear cut, but it's not hard to imagine situations where self driving cars must make choices between courses of action

Presumably the realistic cases are not so clear cut, but it's not hard to imagine situations where self driving cars must make choices between courses of action that have some probability of killing people (and where the car might calculate that all courses of action have some non-zero probability of killing people). The contrived cases make sense as a way to guide what should be done in more realistic/ambiguous ones.

imgabe 5y ago
Maybe. But we needn't compare driverless cars against some hypothetical morally perfect agent. We compare them against humans. Some humans might instinctively sacrifice themselves to save a baby carriage or an old lady. Many more will instinctively act to preserve themselves regardless of the consequences. Even more will panic and make bad decisions that cause needless deaths that a self-driving car would have avoided. I had a cousin who died because he swerved to avoid a squirrel and drove off a cliff. I don't think anyone would argue that we should program cars to prioritize squirrels over people, but humans will end up doing that sometimes. The possible amount of lives saved by the overall reduction in accidents with a competent self-driving car is going to be far, far greater than any 1-in-a-billion moral dilemma situations that they might encounter and where they might make choices that are not "optimal" as decided by some human who is looking at the situation after the fact with the time to think coolly and composedly about it.
- HPsquared 5y ago
  There's also the practical limitations to consider. Self-driving cars have enough trouble identifying what's ON the road, never mind what's off to the side. Is it a cliff, a crowded sidewalk, a crash barrier, a thin wall with lots of people behind, an empty field, trees, a ditch? etc etc.
- penteract 5y ago
  You seem to be addressing the question of when self-driving cars should be used in place of manually driven ones, whereas the website seems to asking how we should program the self driving cars (where I would say we should be trying to make them close to a hypothetically perfect moral agent).
mindslight 5y ago
Why is "probability of killing someone", much less probability of killing specific people, a metric that the car should be calculating in the first place? By the time a car has gotten to these type of no-win no-safe-fail scenarios, you have massively screwed up and are way beyond the design space and can no longer take assumptions for granted. If the probability of sudden total brake failure is high enough to be designed for then the correct answer is to add more hardware for redundant emergency brakes, not to add software to mitigate damage from the insufficient braking system. I feel like this whole topic is basically media click bait, that some software engineers with little hardware experience latch onto because it seems edgy. The answer, like many things in life, is to set aside the abstraction and question the assumptions.

imgabe 5y ago

Maybe. But we needn't compare driverless cars against some hypothetical morally perfect agent. We compare them against humans. Some humans might instinctively sacrifice themselves to save a baby carriage or an old lady. Many more will instinctively act to preserve themselves regardless of the consequences. Even more will panic and make bad decisions that cause needless deaths that a self-driving car would have avoided. I had a cousin who died because he swerved to avoid a squirrel and drove off a cliff. I don't think anyone would argue that we should program cars to prioritize squirrels over people, but humans will end up doing that sometimes. The possible amount of lives saved by the overall reduction in accidents with a competent self-driving car is going to be far, far greater than any 1-in-a-billion moral dilemma situations that they might encounter and where they might make choices that are not "optimal" as decided by some human who is looking at the situation after the fact with the time to think coolly and composedly about it.

HPsquared 5y ago
There's also the practical limitations to consider. Self-driving cars have enough trouble identifying what's ON the road, never mind what's off to the side. Is it a cliff, a crowded sidewalk, a crash barrier, a thin wall with lots of people behind, an empty field, trees, a ditch? etc etc.
penteract 5y ago
You seem to be addressing the question of when self-driving cars should be used in place of manually driven ones, whereas the website seems to asking how we should program the self driving cars (where I would say we should be trying to make them close to a hypothetically perfect moral agent).

mindslight 5y ago

Why is "probability of killing someone", much less probability of killing specific people, a metric that the car should be calculating in the first place? By the time a car has gotten to these type of no-win no-safe-fail scenarios, you have massively screwed up and are way beyond the design space and can no longer take assumptions for granted. If the probability of sudden total brake failure is high enough to be designed for then the correct answer is to add more hardware for redundant emergency brakes, not to add software to mitigate damage from the insufficient braking system. I feel like this whole topic is basically media click bait, that some software engineers with little hardware experience latch onto because it seems edgy. The answer, like many things in life, is to set aside the abstraction and question the assumptions.