5 ms·

Self-Driving Mercedes Will Sacrifice Pedestrians to Save the Driver

21 points by braythwayt 7y ago

baldgeek 7y ago
If you are interested in a bunch of ethical commentary regarding self-drving. Radiolab did a very interesting episode awhile ago: https://www.wnycstudios.org/podcasts/radiolab/articles/driverless-dilemma https://www.wnycstudios.org/podcasts/radiolab/articles/drive...
maerF0x0 7y ago
People may be appalled by the prior decision to save the driver over others. I think the onus is, at least in part, on others to prove that a human driver wouldnt do the same thing. If something is coming at me, I'm likely to swerve (by quick reaction), even if it means hitting someone else.
- moralestapia 7y ago
  Huh? What's the point of deploying AI with the same shortcomings as humans?
  - maerF0x0 7y ago
    The AI ought to be finding itself in the situation far less frequently than humans do. Still a net reduction in issues.
- jdashg 7y ago
  We have laws that say it's not ok to hit pedestrians to avoid obstacles, which is a strong prior that society doesn't want, and enforces against, a greedy self-trolley solution. Expect legislation prohibiting this behavior.
  - Arnt 7y ago
    At the very least it'll tilt the balance further against car drivers, in favour of people who don't want traffic where they live. https://www.bussgeldkatalog-mpu.de/bussgeld-wAssets/img/verkehrszeichen/durchfahrt-verboten/weblication/wThumbnails/verkehrszeichen-250-durchfahrt-verboten-3570d73ddff018agf3d7b07fe1a9d902.jpg https://www.bussgeldkatalog-mpu.de/bussgeld-wAssets/img/verk...
- guenthert 7y ago
  I'm not sure whether that is really the case. If something is coming at you, the choices are usually to swerve or to brake (even if that turns out to be ineffective in avoiding the collision). People might swerve, if they think the newly chosen path is clear. They just might be mistaken in that, often due to tunnel vision and the unfortunate focus on the immediate and obvious danger, rather on a path avoiding disaster. These example strive me as fairly contrived however. In traffic there simply is no time to make complex moral decisions. In the fraction of a second between realizing that an accident is imminent and impact, people just react (more or less skillfully and fortuitous, depending on many circumstances, including training and sheer luck). Perhaps that's actually the way out for developers of such autonomous vehicles: just focus on what's ahead and if there's danger and braking won't be effective, swerve. If then someone is hit, it is an accident (due to incomplete information), rather than a (planned) sacrifice. Ignorance is a bliss.
- braythwayt 7y ago
  You're saying that this is no worse than the status quo. Quite possibly true. But moving forward into a world where every vehicle is autonomous, we definitely want to do better than the status quo. Debates like this shape our definition of "better."
- Gibbon1 7y ago
  Airbags in the A pillar prioritize the safety of the driver while creating a blind spot that makes it difficult to spot pedestrians.
jdashg 7y ago
The trolley problem is so overblown. Just brake. Just braking is a simple and really effective solution to the vast, vast majority of problems. Instead of trying to program complex and fraught harm minimization systems, focus on braking distance, collision safety, and enforcing a safety envelope for self-driving: Never go faster than your brakes, even if humans often do.
- mindslight 7y ago
  The problem with your comment is that it is not long enough to put any ads on it.
- xmprt 7y ago
  The trolley problem always struck me as a super contrived problem when applied to self-driving vehicles. In what scenario are your sensors and computing devices functional to the point that you can decide who you want to save but also break down in such a way that you weren't able to prevent it.
- braythwayt 7y ago
  The trolley problem as usually posed is contrived, but the "swerve instead of braking" problem happens more frequently than many people think. Consider driving along an undivided highway. An approaching car veers into your lane. You definitely want to swerve, rather than brake and risk being hit head-on. Of course, we can contrive a situation on top of that, like a group ride of cyclists to your right, so swerving will take them out... But that is besides the point. I am just saying that there are cases where braking is not the safest thing for the occupants of the car.
  - JoeAltmaier 7y ago
    But that is exactly the trolley problem! Brake and 1 occupant is at risk. Swerve and the cyclists at the side of the road are at risk. So the decision to swerve is the trolley problem either way. You know the cyclists are there, and swerve because the occupant is valued more. Or the car doesn't know if cyclists are there, swerve and these possible/virtual cyclists are sacrificed for the occupant of the vehicle.
neilalexander 7y ago
Assuming we are talking about vehicles that still have manual controls as a last resort, is saving the driver not an obvious course of action? It’s the driver that needs to be given as much time and every possible opportunity to take over and react, to attempt to correct the situation or to move the vehicle (and its occupants) out of further harms way after an accident.
Jeff_Brown 7y ago
> "... Save the one in the car,” von Hugo told Car and Driver in an interview. “If all you know for sure is that one death can be prevented, then that’s your first priority.” How is this quote relevant? THe point of the trolley problem is that you face a clear choice between who dies. Is Mercedes willfully programming its cars to understand the mortality of the car's occupants and nobody else's?
- Gibbon1 7y ago
  Mercedes has a duty towards the people that buy their cars not random jaywalkers.
  - Jeff_Brown 7y ago
    If profit motive was the only duty anybody felt, civilization would quickly collapse. That said, I don't expect business to do anything that's not in its own interest. This sort of problem, if in fact it's a problem[1], would seem to require a legislative solution. [1] https://news.ycombinator.com/item?id=21818086 https://news.ycombinator.com/item?id=21818086
Schwolop 7y ago
This guy needs some PR advice, but the approach is sound. To be ethically clear, the approach must be hard-coded in that it doesn’t evaluate and choose a control output. Instead it is forced to pick one control output that was predetermined to be safe enough. Reducing energy ASAP by braking meets that. Attempting to swerve does not - because you must _decide_ which way to do so, and that decision is fraught with ethical issues.
russellbeattie 7y ago
I wrote this as a comment a few months ago (https://news.ycombinator.com/item?id=21250661 https://news.ycombinator.com/item?id=21250661), and was downvoted, but I'll post it again because I feel it's right: Artificial Intelligence is a tool. Just like anyone who uses a hammer, backhoe or dump truck is responsible for what happens when they are using it, the same is true for AI. As a tool, all AI systems should be designed to always do the most harm to the user of the AI first. It should be embedded into every autopilot-like system, and users should be aware of that choice. It's the only moral and ethically correct solution. Let's say your AI-controlled car is driving at speed and turning the corner, there suddenly appears a little girl in the road in front of you. It can either swerve into a wall or run down the girl. A human might not be able to make the decision in time, but an AI system would. It needs to be programmed to always hit the wall. Though this might seem extreme, the opposite is completely immoral. If it was a human driving, one could assume bad luck to be put in that situation: aka innocent until proven guilty. But for an AI system, we have to assume the opposite: The AI should always be presumed to be faulty. Therefore the user of that AI is culpable of putting it in that situation. And just like any other tool, the manufacturer of that tool is legally responsible for its quality and reliability. The opposite of the above means we'll all be riding around in autonomous tanks, aggressively maneuvering around each other at higher and higher speeds, pedestrians and others be damned.
m463 7y ago
two cars: - brand A saves who it decides. - brand B saves the occupant. which would you buy? Also, I can't help but think of the brand A airplane that knew better and crashed [1] and the recent brand B airplane that in one case it thought it did and crashed [2] 1: https://en.wikipedia.org/wiki/Air_France_Flight_296 https://en.wikipedia.org/wiki/Air_France_Flight_296 2: https://en.wikipedia.org/wiki/Lion_Air_Flight_610 https://en.wikipedia.org/wiki/Lion_Air_Flight_610
thatfrenchguy 7y ago
Waiting for the lawsuit ?

21 points by braythwayt 7y ago

If you are interested in a bunch of ethical commentary regarding self-drving. Radiolab did a very interesting episode awhile ago: https://www.wnycstudios.org/podcasts/radiolab/articles/driverless-dilemma https://www.wnycstudios.org/podcasts/radiolab/articles/drive...

maerF0x0 7y ago

People may be appalled by the prior decision to save the driver over others. I think the onus is, at least in part, on others to prove that a human driver wouldnt do the same thing. If something is coming at me, I'm likely to swerve (by quick reaction), even if it means hitting someone else.

moralestapia 7y ago
Huh? What's the point of deploying AI with the same shortcomings as humans?
- maerF0x0 7y ago
  The AI ought to be finding itself in the situation far less frequently than humans do. Still a net reduction in issues.
jdashg 7y ago
We have laws that say it's not ok to hit pedestrians to avoid obstacles, which is a strong prior that society doesn't want, and enforces against, a greedy self-trolley solution. Expect legislation prohibiting this behavior.
- Arnt 7y ago
  At the very least it'll tilt the balance further against car drivers, in favour of people who don't want traffic where they live. https://www.bussgeldkatalog-mpu.de/bussgeld-wAssets/img/verkehrszeichen/durchfahrt-verboten/weblication/wThumbnails/verkehrszeichen-250-durchfahrt-verboten-3570d73ddff018agf3d7b07fe1a9d902.jpg https://www.bussgeldkatalog-mpu.de/bussgeld-wAssets/img/verk...
guenthert 7y ago
I'm not sure whether that is really the case. If something is coming at you, the choices are usually to swerve or to brake (even if that turns out to be ineffective in avoiding the collision). People might swerve, if they think the newly chosen path is clear. They just might be mistaken in that, often due to tunnel vision and the unfortunate focus on the immediate and obvious danger, rather on a path avoiding disaster. These example strive me as fairly contrived however. In traffic there simply is no time to make complex moral decisions. In the fraction of a second between realizing that an accident is imminent and impact, people just react (more or less skillfully and fortuitous, depending on many circumstances, including training and sheer luck). Perhaps that's actually the way out for developers of such autonomous vehicles: just focus on what's ahead and if there's danger and braking won't be effective, swerve. If then someone is hit, it is an accident (due to incomplete information), rather than a (planned) sacrifice. Ignorance is a bliss.
braythwayt 7y ago
You're saying that this is no worse than the status quo. Quite possibly true. But moving forward into a world where every vehicle is autonomous, we definitely want to do better than the status quo. Debates like this shape our definition of "better."
Gibbon1 7y ago
Airbags in the A pillar prioritize the safety of the driver while creating a blind spot that makes it difficult to spot pedestrians.

jdashg 7y ago

The trolley problem is so overblown. Just brake. Just braking is a simple and really effective solution to the vast, vast majority of problems. Instead of trying to program complex and fraught harm minimization systems, focus on braking distance, collision safety, and enforcing a safety envelope for self-driving: Never go faster than your brakes, even if humans often do.

mindslight 7y ago
The problem with your comment is that it is not long enough to put any ads on it.
xmprt 7y ago
The trolley problem always struck me as a super contrived problem when applied to self-driving vehicles. In what scenario are your sensors and computing devices functional to the point that you can decide who you want to save but also break down in such a way that you weren't able to prevent it.
braythwayt 7y ago
The trolley problem as usually posed is contrived, but the "swerve instead of braking" problem happens more frequently than many people think. Consider driving along an undivided highway. An approaching car veers into your lane. You definitely want to swerve, rather than brake and risk being hit head-on. Of course, we can contrive a situation on top of that, like a group ride of cyclists to your right, so swerving will take them out... But that is besides the point. I am just saying that there are cases where braking is not the safest thing for the occupants of the car.
- JoeAltmaier 7y ago
  But that is exactly the trolley problem! Brake and 1 occupant is at risk. Swerve and the cyclists at the side of the road are at risk. So the decision to swerve is the trolley problem either way. You know the cyclists are there, and swerve because the occupant is valued more. Or the car doesn't know if cyclists are there, swerve and these possible/virtual cyclists are sacrificed for the occupant of the vehicle.

neilalexander 7y ago

Assuming we are talking about vehicles that still have manual controls as a last resort, is saving the driver not an obvious course of action? It’s the driver that needs to be given as much time and every possible opportunity to take over and react, to attempt to correct the situation or to move the vehicle (and its occupants) out of further harms way after an accident.

Jeff_Brown 7y ago

> "... Save the one in the car,” von Hugo told Car and Driver in an interview. “If all you know for sure is that one death can be prevented, then that’s your first priority.” How is this quote relevant? THe point of the trolley problem is that you face a clear choice between who dies. Is Mercedes willfully programming its cars to understand the mortality of the car's occupants and nobody else's?

Gibbon1 7y ago
Mercedes has a duty towards the people that buy their cars not random jaywalkers.
- Jeff_Brown 7y ago
  If profit motive was the only duty anybody felt, civilization would quickly collapse. That said, I don't expect business to do anything that's not in its own interest. This sort of problem, if in fact it's a problem[1], would seem to require a legislative solution. [1] https://news.ycombinator.com/item?id=21818086 https://news.ycombinator.com/item?id=21818086

Schwolop 7y ago

This guy needs some PR advice, but the approach is sound. To be ethically clear, the approach must be hard-coded in that it doesn’t evaluate and choose a control output. Instead it is forced to pick one control output that was predetermined to be safe enough. Reducing energy ASAP by braking meets that. Attempting to swerve does not - because you must _decide_ which way to do so, and that decision is fraught with ethical issues.

russellbeattie 7y ago

I wrote this as a comment a few months ago (https://news.ycombinator.com/item?id=21250661 https://news.ycombinator.com/item?id=21250661), and was downvoted, but I'll post it again because I feel it's right: Artificial Intelligence is a tool. Just like anyone who uses a hammer, backhoe or dump truck is responsible for what happens when they are using it, the same is true for AI. As a tool, all AI systems should be designed to always do the most harm to the user of the AI first. It should be embedded into every autopilot-like system, and users should be aware of that choice. It's the only moral and ethically correct solution. Let's say your AI-controlled car is driving at speed and turning the corner, there suddenly appears a little girl in the road in front of you. It can either swerve into a wall or run down the girl. A human might not be able to make the decision in time, but an AI system would. It needs to be programmed to always hit the wall. Though this might seem extreme, the opposite is completely immoral. If it was a human driving, one could assume bad luck to be put in that situation: aka innocent until proven guilty. But for an AI system, we have to assume the opposite: The AI should always be presumed to be faulty. Therefore the user of that AI is culpable of putting it in that situation. And just like any other tool, the manufacturer of that tool is legally responsible for its quality and reliability. The opposite of the above means we'll all be riding around in autonomous tanks, aggressively maneuvering around each other at higher and higher speeds, pedestrians and others be damned.

m463 7y ago

two cars: - brand A saves who it decides. - brand B saves the occupant. which would you buy? Also, I can't help but think of the brand A airplane that knew better and crashed [1] and the recent brand B airplane that in one case it thought it did and crashed [2] 1: https://en.wikipedia.org/wiki/Air_France_Flight_296 https://en.wikipedia.org/wiki/Air_France_Flight_296 2: https://en.wikipedia.org/wiki/Lion_Air_Flight_610 https://en.wikipedia.org/wiki/Lion_Air_Flight_610

thatfrenchguy 7y ago

Waiting for the lawsuit ?