3 ms·

Every single scenario in this is a false dichotomy. If the only choices are "plow through a bunch of pedestrians, killing them" and "swerve into a fixed obstac

Every single scenario in this is a false dichotomy. If the only choices are "plow through a bunch of pedestrians, killing them" and "swerve into a fixed obstacle, killing your passengers", your self-driving car has already made an immoral choice to drive too fast for the conditions. The correct choice was "if there are hazards in the roadway limiting your maneuvering options, pedestrians (or other objects) that might suddenly enter the roadway or visual blockages that would prevent the car from determining whether there are people/object that might suddenly enter the roadway - slow down before the car's choices are so limited that they all involve unnecessary death". A self driving car that encounters any of these scenarios has already failed at being a safe driver.

blackbear_ 5y ago
What if the car contains a passenger whose life is at risk and requires urgent medical care?
- wizzwizz4 5y ago
  Normally, you have plenty of warning if you're near a situation like the ones described here, even if you're speeding. Provided you can pay a small amount of attention to many many things at once, which humans can't. (Neither can existing self-driving cars, but a self-driving car that works must be able to.)
dataangel 5y ago
While it's true it should already be considered a kind of failure to get into this situation in the first place, that shouldn't get in the way of "hope for the best, plan for the worst." It's never going to be possible to anticipate every kind of condition and have perfect information. Ideally we want these systems to engage in whatever harm and risk mitigation is believed to be possible based on the current reality, we don't want the system saying to itself, "The real solution is for me to go back in time and not have made some earlier mistake." That analysis is great afterwards for figuring out how to prevent in the system from getting into that situation again in the future, so it's still important too, but we need both.
- danShumway 5y ago
  Thought experiment: Let's assume that a burglar has broken into someone's house, but Amazon Alexa knows the burglar is unarmed and that the owner is armed. Alexa has IDed the burglar and knows that they're a college kid who is unlikely to cause injury or attack someone. Alexa looks up the kid in police databases and identifies that they do not have a criminal record. The owner of the house is trying to hunt down the burglar with a gun. The owner asks Alexa where the burglar is currently hiding. Alexa knows the average response time of police officers in that neighborhood, and knows that if they're called, they'll arrive too late to intervene in the situation. Should Amazon Alexa: A) remain neutral and refuse to answer, letting the situation play out as it may? B) lie to the owner so that the kid can escape? C) threaten the owner with revealing an embarrassing secret so that they'll let the kid go? D) tell the truth and let the owner find the kid? And does it bother you that Amazon doesn't have a public policy available about what Alexa should do in this situation? Does it bother you that Amazon hasn't prepared for the worst in that situation? That Amazon is selling Alexa devices even though we haven't had a public debate about it or put any public funding into answering this question? ---- The problem with the questions on this site is not that they're preparing for the worst, it's that they're preparing for the absurd. They're saying we should build capabilities to prepare for situations that devices shouldn't be in to start with, that devices shouldn't realistically be making decisions about, and that devices should respond to those situations based on information that shouldn't be available to them using capabilities that they don't have. A "prepare for the worst" policy for self-driving cars as they exist today is realistically "slam on the breaks, swerve the car without over-correcting the wheel and going into oncoming traffic." That is a significant improvement over human drivers today, none of which are capable of forming split-second philosophical decisions. Simple consistent rules like "brake, swerve but don't cross oncoming lanes" are good enough. Realistically, they're good enough forever, we never need to ask these philosophical questions. All of the energy spent debating them can be poured into decreasing car brake time. And by positing a scenario where a car is able to identify people's age and demographics in milliseconds, we're entering into a world where there really is zero excuse for why the car got into that situation to begin with. We're positing a scenario where a car somehow has godlike powers to make split-second philosophical decisions, but where they can't avoid accidentally speeding up through a cross-walk. I don't think that's realistic, and I think it's a waste of time to worry about. If it doesn't bother you that Alexa doesn't have a philosophical threshold around burglar age/danger where it will decide whether or not to sell out that someone's hiding under the stairs from an owner with a gun, then it also probably shouldn't bother you that your Tesla doesn't have a special mode where it decides whether or not to swerve into a pedestrian based on how old they are. Because both of those scenarios are equally ridiculous, and there are some worst-case scenarios that are so ridiculous that it's a waste of time to even think about them. A car shouldn't have the capabilities to make any of the decisions we're talking about. If engineers start building those capabilities then something has gone clearly wrong and we all need to stop driving the car that does socioeconomic profiling of the people around it. I guarantee you that any car that is designed to do that is also doing some horrifying surveillance crap that should be opposed at every turn. And if the capabilities ever actually do get built for some reason (which they shouldn't), and do work reliably (which they won't), and are capable of being used in these situations where split-second decisions matter, then engineers should be going to jail for negligence every time that car crashes, because any system that advanced could also avoid crashing in the first place.
danShumway 5y ago
Especially if it has AI and recognition systems powerful enough to make these decisions reliably in the first place. We're imagining a scenario where a computer is capable of reliably recognizing that someone is an elderly female doctor, and is capable of responding to that information in milliseconds before slamming into that person, but the car is somehow not capable of recognizing ahead of time that a human is on the side of the road and about to step into it. Cars that have AI systems advanced enough to recognize what profession a pedestrian is in should also be able to recognize that pedestrians are on the side of the road, or that the car is entering a blind spot where a pedestrian might enter the road. These are very selectively omniscient cars that the website imagines we've built.

blackbear_ 5y ago

What if the car contains a passenger whose life is at risk and requires urgent medical care?

wizzwizz4 5y ago
Normally, you have plenty of warning if you're near a situation like the ones described here, even if you're speeding. Provided you can pay a small amount of attention to many many things at once, which humans can't. (Neither can existing self-driving cars, but a self-driving car that works must be able to.)

dataangel 5y ago

While it's true it should already be considered a kind of failure to get into this situation in the first place, that shouldn't get in the way of "hope for the best, plan for the worst." It's never going to be possible to anticipate every kind of condition and have perfect information. Ideally we want these systems to engage in whatever harm and risk mitigation is believed to be possible based on the current reality, we don't want the system saying to itself, "The real solution is for me to go back in time and not have made some earlier mistake." That analysis is great afterwards for figuring out how to prevent in the system from getting into that situation again in the future, so it's still important too, but we need both.

danShumway 5y ago
Thought experiment: Let's assume that a burglar has broken into someone's house, but Amazon Alexa knows the burglar is unarmed and that the owner is armed. Alexa has IDed the burglar and knows that they're a college kid who is unlikely to cause injury or attack someone. Alexa looks up the kid in police databases and identifies that they do not have a criminal record. The owner of the house is trying to hunt down the burglar with a gun. The owner asks Alexa where the burglar is currently hiding. Alexa knows the average response time of police officers in that neighborhood, and knows that if they're called, they'll arrive too late to intervene in the situation. Should Amazon Alexa: A) remain neutral and refuse to answer, letting the situation play out as it may? B) lie to the owner so that the kid can escape? C) threaten the owner with revealing an embarrassing secret so that they'll let the kid go? D) tell the truth and let the owner find the kid? And does it bother you that Amazon doesn't have a public policy available about what Alexa should do in this situation? Does it bother you that Amazon hasn't prepared for the worst in that situation? That Amazon is selling Alexa devices even though we haven't had a public debate about it or put any public funding into answering this question? ---- The problem with the questions on this site is not that they're preparing for the worst, it's that they're preparing for the absurd. They're saying we should build capabilities to prepare for situations that devices shouldn't be in to start with, that devices shouldn't realistically be making decisions about, and that devices should respond to those situations based on information that shouldn't be available to them using capabilities that they don't have. A "prepare for the worst" policy for self-driving cars as they exist today is realistically "slam on the breaks, swerve the car without over-correcting the wheel and going into oncoming traffic." That is a significant improvement over human drivers today, none of which are capable of forming split-second philosophical decisions. Simple consistent rules like "brake, swerve but don't cross oncoming lanes" are good enough. Realistically, they're good enough forever, we never need to ask these philosophical questions. All of the energy spent debating them can be poured into decreasing car brake time. And by positing a scenario where a car is able to identify people's age and demographics in milliseconds, we're entering into a world where there really is zero excuse for why the car got into that situation to begin with. We're positing a scenario where a car somehow has godlike powers to make split-second philosophical decisions, but where they can't avoid accidentally speeding up through a cross-walk. I don't think that's realistic, and I think it's a waste of time to worry about. If it doesn't bother you that Alexa doesn't have a philosophical threshold around burglar age/danger where it will decide whether or not to sell out that someone's hiding under the stairs from an owner with a gun, then it also probably shouldn't bother you that your Tesla doesn't have a special mode where it decides whether or not to swerve into a pedestrian based on how old they are. Because both of those scenarios are equally ridiculous, and there are some worst-case scenarios that are so ridiculous that it's a waste of time to even think about them. A car shouldn't have the capabilities to make any of the decisions we're talking about. If engineers start building those capabilities then something has gone clearly wrong and we all need to stop driving the car that does socioeconomic profiling of the people around it. I guarantee you that any car that is designed to do that is also doing some horrifying surveillance crap that should be opposed at every turn. And if the capabilities ever actually do get built for some reason (which they shouldn't), and do work reliably (which they won't), and are capable of being used in these situations where split-second decisions matter, then engineers should be going to jail for negligence every time that car crashes, because any system that advanced could also avoid crashing in the first place.

danShumway 5y ago

Especially if it has AI and recognition systems powerful enough to make these decisions reliably in the first place. We're imagining a scenario where a computer is capable of reliably recognizing that someone is an elderly female doctor, and is capable of responding to that information in milliseconds before slamming into that person, but the car is somehow not capable of recognizing ahead of time that a human is on the side of the road and about to step into it. Cars that have AI systems advanced enough to recognize what profession a pedestrian is in should also be able to recognize that pedestrians are on the side of the road, or that the car is entering a blind spot where a pedestrian might enter the road. These are very selectively omniscient cars that the website imagines we've built.