4 ms·

These contrived examples don't seem that helpful. Brakes failed? Throw the car into park or reverse. Pull the emergency/parking brake. Swerve sharply enough to

by imgabe 5y ago

These contrived examples don't seem that helpful. Brakes failed? Throw the car into park or reverse. Pull the emergency/parking brake. Swerve sharply enough to spin out. Honk to alert pedestrians (which a self-driving car would have seen long ago and already started slowing down anyway). If all else fails, build the car so that a head-on collision into a barrier at speeds where pedestrians could be present would not be fatal to the occupants. In the real world I have a hard time imagining a scenario where a case like this would actually come up.

penteract 5y ago
Presumably the realistic cases are not so clear cut, but it's not hard to imagine situations where self driving cars must make choices between courses of action that have some probability of killing people (and where the car might calculate that all courses of action have some non-zero probability of killing people). The contrived cases make sense as a way to guide what should be done in more realistic/ambiguous ones.
- imgabe 5y ago
  Maybe. But we needn't compare driverless cars against some hypothetical morally perfect agent. We compare them against humans. Some humans might instinctively sacrifice themselves to save a baby carriage or an old lady. Many more will instinctively act to preserve themselves regardless of the consequences. Even more will panic and make bad decisions that cause needless deaths that a self-driving car would have avoided. I had a cousin who died because he swerved to avoid a squirrel and drove off a cliff. I don't think anyone would argue that we should program cars to prioritize squirrels over people, but humans will end up doing that sometimes. The possible amount of lives saved by the overall reduction in accidents with a competent self-driving car is going to be far, far greater than any 1-in-a-billion moral dilemma situations that they might encounter and where they might make choices that are not "optimal" as decided by some human who is looking at the situation after the fact with the time to think coolly and composedly about it.
  - HPsquared 5y ago
    There's also the practical limitations to consider. Self-driving cars have enough trouble identifying what's ON the road, never mind what's off to the side. Is it a cliff, a crowded sidewalk, a crash barrier, a thin wall with lots of people behind, an empty field, trees, a ditch? etc etc.
  - penteract 5y ago
    You seem to be addressing the question of when self-driving cars should be used in place of manually driven ones, whereas the website seems to asking how we should program the self driving cars (where I would say we should be trying to make them close to a hypothetically perfect moral agent).
- mindslight 5y ago
  Why is "probability of killing someone", much less probability of killing specific people, a metric that the car should be calculating in the first place? By the time a car has gotten to these type of no-win no-safe-fail scenarios, you have massively screwed up and are way beyond the design space and can no longer take assumptions for granted. If the probability of sudden total brake failure is high enough to be designed for then the correct answer is to add more hardware for redundant emergency brakes, not to add software to mitigate damage from the insufficient braking system. I feel like this whole topic is basically media click bait, that some software engineers with little hardware experience latch onto because it seems edgy. The answer, like many things in life, is to set aside the abstraction and question the assumptions.
tgv 5y ago
The first scenario I got was: car drives in direction of concrete road block that blocks half the road, on the other half there are two cats walking (on a zebra crossing, if you can believe it). The implicit choice seems to be "crash car with passengers" or "kill the cats". How about breaking? You should never drive so fast that you can't avoid a static obstacle, is it?
ep103 5y ago
Correct, the Moral Machine is an inherently flawed approach to its domain. Abby Evertt Jaques goes into much more detail, but the point you are pointing out here is one of the main concerns. Why the Moral Machine is a Monster - Abby Everett Jaques https://robots.law.miami.edu/2019/wp-content/uploads/2019/03/MoralMachineMonster.pdf https://robots.law.miami.edu/2019/wp-content/uploads/2019/03...
- mindslight 5y ago
  I think that paper is still missing the bigger picture - in its own terms, the "structural effect" of framing the situation in such a paradigm to begin with. The paradigm is flawed in that it presupposes that these situations exist and could not have been avoided, thus absolving responsibility for whomever got the car into the unwinnable situation to begin with. We've already got enough of this "nobody's fault, things happen" nonsense with traditional cars. Self driving cars are capable of programmatically sticking to invariants and not becoming overwhelmed, and we should reject allowing the lazy "no fault" culture to persist.

by imgabe 5y ago

penteract 5y ago

Presumably the realistic cases are not so clear cut, but it's not hard to imagine situations where self driving cars must make choices between courses of action that have some probability of killing people (and where the car might calculate that all courses of action have some non-zero probability of killing people). The contrived cases make sense as a way to guide what should be done in more realistic/ambiguous ones.

imgabe 5y ago
Maybe. But we needn't compare driverless cars against some hypothetical morally perfect agent. We compare them against humans. Some humans might instinctively sacrifice themselves to save a baby carriage or an old lady. Many more will instinctively act to preserve themselves regardless of the consequences. Even more will panic and make bad decisions that cause needless deaths that a self-driving car would have avoided. I had a cousin who died because he swerved to avoid a squirrel and drove off a cliff. I don't think anyone would argue that we should program cars to prioritize squirrels over people, but humans will end up doing that sometimes. The possible amount of lives saved by the overall reduction in accidents with a competent self-driving car is going to be far, far greater than any 1-in-a-billion moral dilemma situations that they might encounter and where they might make choices that are not "optimal" as decided by some human who is looking at the situation after the fact with the time to think coolly and composedly about it.
- HPsquared 5y ago
  There's also the practical limitations to consider. Self-driving cars have enough trouble identifying what's ON the road, never mind what's off to the side. Is it a cliff, a crowded sidewalk, a crash barrier, a thin wall with lots of people behind, an empty field, trees, a ditch? etc etc.
- penteract 5y ago
  You seem to be addressing the question of when self-driving cars should be used in place of manually driven ones, whereas the website seems to asking how we should program the self driving cars (where I would say we should be trying to make them close to a hypothetically perfect moral agent).
mindslight 5y ago
Why is "probability of killing someone", much less probability of killing specific people, a metric that the car should be calculating in the first place? By the time a car has gotten to these type of no-win no-safe-fail scenarios, you have massively screwed up and are way beyond the design space and can no longer take assumptions for granted. If the probability of sudden total brake failure is high enough to be designed for then the correct answer is to add more hardware for redundant emergency brakes, not to add software to mitigate damage from the insufficient braking system. I feel like this whole topic is basically media click bait, that some software engineers with little hardware experience latch onto because it seems edgy. The answer, like many things in life, is to set aside the abstraction and question the assumptions.

tgv 5y ago

The first scenario I got was: car drives in direction of concrete road block that blocks half the road, on the other half there are two cats walking (on a zebra crossing, if you can believe it). The implicit choice seems to be "crash car with passengers" or "kill the cats". How about breaking? You should never drive so fast that you can't avoid a static obstacle, is it?

ep103 5y ago

Correct, the Moral Machine is an inherently flawed approach to its domain. Abby Evertt Jaques goes into much more detail, but the point you are pointing out here is one of the main concerns. Why the Moral Machine is a Monster - Abby Everett Jaques https://robots.law.miami.edu/2019/wp-content/uploads/2019/03/MoralMachineMonster.pdf https://robots.law.miami.edu/2019/wp-content/uploads/2019/03...

mindslight 5y ago
I think that paper is still missing the bigger picture - in its own terms, the "structural effect" of framing the situation in such a paradigm to begin with. The paradigm is flawed in that it presupposes that these situations exist and could not have been avoided, thus absolving responsibility for whomever got the car into the unwinnable situation to begin with. We've already got enough of this "nobody's fault, things happen" nonsense with traditional cars. Self driving cars are capable of programmatically sticking to invariants and not becoming overwhelmed, and we should reject allowing the lazy "no fault" culture to persist.