6 ms·

The Robot Car of Tomorrow May Just Be Programmed to Hit You

47 points by arikrak 12y ago

angersock 12y ago
The basic thought experiment is this: "If the robot car that hit me on my bike did so with provably the best intentions, should I still be angry?" At least with a person driving, I can criticize their thought patterns, their reactions, their choices--I can laud them for dodging the stroller, or curse them for drinking and swerving into me (provided I can still talk). With a machine, though, we lose any chance at morality. They act exactly as programmed, and we all know how reliable programs are. If, following such an accident, I were to be in a wheelchair for the rest of my life, I would want the entire stack trace that led up to my collision. I would want the unabridged source code and issue trackers. I would want the core dump. I would want every scrap of information that went into the vehicle's decision to zig instead of zagging. I'm not sure that we want to hand over so much blanket responsibility to black-boxes designed by committee and implemented by antisocial 20-something engineers working weird hours. EDIT: Reflecting on this a little has suggested, perhaps, that we should have one standard algorithm and package for doing this sort of thing, open-source and in the public domain. There shouldn't even be a question about allowing closed-source autonomous vehicles on the road.
- eurleif 12y ago
  You would want all that information for the purpose of proving the program was at fault, right? What if the company behind the software was automatically liable for all accidents? Wouldn't that answer that?
  - angersock 12y ago
    I would want all that information as closure, since I wouldn't be able to walk again. Whether or not the program is functioning correctly, I want to know exactly how things went down--something that is finally possible if you automate driving. Holding the companies responsible is nice (and might as well be done), but until all cars are autonomous (and even then) there will be many cases where the initial conditions of an accident preclude anything but a harmful outcome. For example, a robot car doing the right thing could be boxed in by two malicious drivers and "forced" into hurting a bystander.
- jotux 12y ago
  >implemented by antisocial 20-something engineers working weird hours. I feel like this is a myth we should actively work to squash. The vast majority of engineers and coders are 30/40-somethings with normal social lives, families, and working 9-5 weekdays.
  - angersock 12y ago
    Quite true--and yet, look at the companies making the news. Hell, in many cases your motion-planning is being done by people who were graduate students, and even then the stereotype...well, it fits, you know?
  - FatalError 12y ago
    And even if they were, surely this is only a bad thing in that it might indicate that some potentially good engineers had been excluded from the workforce. Is reverse-bigotry really so far gone that some people genuinely think that being an "antisocial 20-something" makes a person a worse engineer?
    - dagw 12y ago
      One of the key factors in becoming a good engineer is experience. a 20-something, antisocial or otherwise, basically by definition is lacking in this experience. Sure they may be really really smart and write incredibly clever code, but it takes more than that to be a good engineer.
- MiguelHudnandez 12y ago
  I think another way to look at it is: if all cars were on auto-pilot, you would have a much smaller chance of being injured or killed on your bicycle. So if it happens a hundred thousand times more with human drivers, but there's a person you can point your finger at afterwards, does that make anyone more comfortable?
- thenmar 12y ago
  I'm having trouble understanding what point you're trying to make. Is it that if a computer bug causes an accident, the victim wouldn't have an individual to fantasize about getting revenge on? That the company that developed the software wouldn't be held accountable? That humans have "morals" that make them avoid accidents, but computers don't, so there will be more accidents? Please clarify this for us.
- rando289 12y ago
  > we lose any chance at morality Our morality is mostly learned. It can grow. Yes, we do need the source to be available.
  - npizzolato 12y ago
    Do you? Where's the source for the programs flying commercial jets? Do you still fly?
    - rando289 12y ago
      Good point. However, I don't own the jet, nor have any liability/responsibility if it's software fails. I do own the car, and I do have liability/responsibility for its operation.
NPC82 12y ago
I think this problem is nonexistent to an autonomous car. A situation where the car has a choice to hit something would never happen. The only times humans have that choice is if we don't look far ahead enough or our field of view is impaired. Sure, autonomous cars may eventually hit something. But if they do it will be totally out of their control, by design. We can make them see further, stay focused, and be cautious. Humans can't always do that, and that's why we have to constantly choose our lesser evils.
- rando289 12y ago
  It is a real problem. It should be fairly obvious that it is. More realistically, It won't be a real time decision of hit this or that, it will be that it is programmed to do behavior X instead of Y. X is known to be 30% more likely to hit a pedestrian, but 10% less likely overall to be in any accident. Then you dramatize it by saying the car decided to hit a pedestrian instead of a car. Just substitute the behavior described in the story or some variation.
  - thenmar 12y ago
    I think it will probably be much more complicated than that. Suppose the car is controlled by an artificial neural network that has been trained with a combination of real world experience and human-generated driving data. There is no giant switch statement that someone wrote for the outcome. Who is to blame then? Would using that data to train the model further be sufficient accountability?
- aninhumer 12y ago
  There will always be unexpected situations, especially while we still have human drivers on the road. Ultimately, if you want to actually get anywhere while there are other cars on the road, you have to make some assumptions about how they're going to behave. And even when you're completely automated, there may still be unexpected failures that cause these situations. However, I'm not sure the ethics of this are such a big deal, because the decision is such a tiny part of the causation of the eventual accident. I think it's far more likely that people will write code that just keeps trying to dodge until it can't any more, and I don't think that's a bad thing. Ultimately you have to make some compromises on safety vs resources, and deciding not to spend time writing special behaviour for pathological cases is no more ethically suspect than deciding not to spend any more time improving the brakes.
jfoster 12y ago
"As a matter of physics, you should choose a collision with a heavier vehicle that can better absorb the impact of a crash, which means programming the car to crash into the Volvo." Better for the car that is being hit, but not the car that we have control of. I'd like my robotic car to hit the car with more crumple or smaller relative momentum.
- couchand 12y ago
  You raise an interesting point: does the software controlling an automated car optimize for the owner of that car (as capitalism would suggest) or does it optimize for universal utility (as this article and many write-ups assume). I'd guess we wouldn't see as significant of gains if the cars optimize for their individual driver; however, it won't be long before some plucky manufacturer sees an untapped market and builds a selfish car.
NAFV_P 12y ago
> In the name of crash-optimization, you should program the car to crash into whatever can best survive the collision. In the last scenario, that meant smashing into the Volvo SUV. Here, it means striking the motorcyclist who’s wearing a helmet. A good algorithm would account for the much-higher statistical odds that the biker without a helmet would die, and surely killing someone is one of the worst things auto manufacturers desperately want to avoid. This could encourage bikers to refrain from donning protective gear. This reminds me of "Fight Club", where the central (anonymous) character describes his job in crash investigations. He has to decide whether recalling all instances of a make of car is less expensive than getting sued by the drivers in court.
DigitalSea 12y ago
The kind of scenario Wired propose between hitting two cars based on which would absorb the impact the better or cause the least harm aren't the kind of decisions humans make during a crash. I've been in a couple of car accidents caused by other people luckily and not once during any of those crashes did I weigh up my options between which car or which direction to swerve. Maybe we will expect more from an autonomous for its ability to make these kind of decisions, but I think it's an argument with a few holes. A lot of cars are currently at a point where they come standard with sensors that can allow a car to automatically parallel park itself between two other cars on a busy street. Smart cruise control which will know when to brake if the distance between your car and the car in-front gets smaller, cameras that follow the lines on a road. It's all there, car technology is already insanely smart. The questions people aren't asking here about autonomous cars is not what if it has to choose between two targets to hit, but rather what happens when a bug in the software controlling the vehicle rears its head and the car gets confused and reacts to something that isn't really happening? A faulty sensor thinks you're about to hit something when you're driving on a straight road and swerves into a tree trying to avoid an accident that wasn't even about to take place, now that's a scary thought. That would be my biggest worry. I am aware modern jetliner planes are essentially fly by wire except during takeoff and landing, but if the computer fails, another takes its place and on the rare occasion redundancy fails, the pilot and crew are there to manually guide the plane. The arguments presented here aren't even realistic. Why should autonomous computers be expected to make such decisions based on environmental factors like if a car can handle the impact or not? For accidents where the outcome will be the same (choosing between two vehicles for example) does it matter what the thinking behind the decision is? There is going to be an accident and no matter how in-depth the analysis is, depending on the speed and location, someone is bound to get hurt to an extent or killed. I have a much better idea and it's far cheaper and more realistic: if an accident is about to take place, the car slams on the brakes and the car slides to a stop. Technology like that already exists in some cars. Swerving in a car accident is most of the time the wrong choice as well, you're most likely to cause more harm to someone swerving in an unpredictable direction then you are straight on, unless of course you can see someone ahead, something on the road or about to have a head-on collision with another car.
- sounds 12y ago
Schwolop 12y ago
I have personally written algorithms to select paths for 800kg autonomous vehicles operating around humans and livestock, and that minimise some cost function. My cost functions always treated crashes as (effectively) infinite cost. In the limit case where all possible paths had such an infinite cost, the failure mode was to revoke any electrical system that could increase the kinetic energy of the vehicle, apply maximum braking power (and/or any other subsystem that would serve reduce the kinetic energy of the vehicle), disconnect any chemical stores of energy from any activation system (i.e. stall the engine), and then finally - and as a direct consequence of these actions - maintain heading/steering angle. Since we elected to revoke all power systems, we lose the ability to steer. We're attempting to reduce the kinetic energy of the system to zero as rapidly as possible (since we've computed that there are no viable controlled manoeuvres available), and we concluded that the best (and in fact only) option was to maintain trajectory whilst trying to slow down. As a result, I don't think this argument has merit. There is no concept of "who should we crash into" because both are unacceptable, and we'd let physics take over instead.
- TheSoftwareGuy 12y ago
  Not to be pedantic, but I think you mean to reduce the kinetic energy, not potential. kinetic energy is associated with movement, potential is not.
  - Schwolop 12y ago
    Yes, you're totally right and I've edited my phrasing. I was trying to cover the issue of not allowing any more petrol to be turned into kinetic energy, and muddled my own brain... :-)
- stickfigure 12y ago
  Opting to hold direction is a technical limitation of your system; it's not obvious that disconnecting all power systems is inherently the safest choice - especially as other vehicles react to the situation. Or at the very least, there is a long ethical decision tree of probabilities that must be evaluated before a crash is 100% certain. Do you pick a 48% chance of hitting the Volvo or a 26% chance of hitting the motorcycle? I predict that the ultimate solution will be to pick the path that optimizes for survival of the occupants of the automated vehicle. After all, they paid for it.
jerf 12y ago
Another edge case I've wondered about is whether there will be driving circumstances during inclement weather in which a sensibly-programmed autonomous car will simply pull over and refuse to operate, even as human drivers happily drive forth. Autonomous cars will be capable of producing reasonably objective measurements of driving condition safety... I wouldn't be surprised winter storms are more routinely capable of inducing conditions that are worse than we humans really realize. Or, alternatively, that we realize are dangerous but choose to drive in anyhow, but no autonomous care programmer would ever choose to accept.
tunesmith 12y ago
This reminds me of the Trolley Problem [1]. It could be argued it is the difference between utilitarianism and value ethics. Or around a recent dinner table, the star trek metaphor was "The needs of the many outweigh the needs of the view" versus the "Prime Directive". The former would actively choose whatever crash would do the least damage by some utility function, while the latter would see that a crash would happen either way and thereby turn off any guidance technology since it could then be argued to not be responsible for an active choice in either direction. [1] http://en.wikipedia.org/wiki/Trolley_problem http://en.wikipedia.org/wiki/Trolley_problem
- Schwolop 12y ago
  The second option (turning off all guidance) is effectively what I did (see https://news.ycombinator.com/item?id=7708105 https://news.ycombinator.com/item?id=7708105). Upon recognising that we had no controlled trajectories that were viable, we just tried to reduce kinetic energy as fast as possible without making any trajectory changes at all.
bagels 12y ago
Why program the car to choose to crash in to things? This makes no sense. What if the sensors are wrong and there was no second car? It seems more prudent to just program it to avoid crashing in to things, even if that is an impossible goal in some circumstances. In those cases, a better fallback would be to reduce velocity, and keep the car on the road or road shoulder perhaps?
ArchD 12y ago
In an unexpected complex situation for which there is very little training data and the behavioral dynamics are not merely about physics, namely impending accident with other cars controlled by other entities that can also respond to this car, any kind of simplistic reasoning like "hit the SUV because it can survive better" has no scientific basis. It's not clear what is being optimized for. For example, how do you know that SUVs are not more likely to have babies who are more fragile, so hitting the SUV is not necessarily better in terms of reducing fatalities. Also, the other cars are responding to this car. This is not about shooting stationary targets.
glomph 12y ago
The other day I was thinking how easy it would be to DoS a robotic car. You could just walk in front of it and then out the way again, over and over. How would they deal with malicious pedestrians?
- IvyMike 12y ago
  Of course you could already do this to a human car. In which case you'd call a human cop. And thus the answer becomes obvious. Robocar is DoS'd? Call Robocop.
sdfjkl 12y ago
This headline got me thinking, what if such a car gets infected with malware? Let's see. I don't know much about self-driving cars, but I imagine they use plenty of cameras and image processing software. That might make it pretty suitable to run some facial recognition software. Now multiply that with the botnets we already have today. The result might be assassination by malware: Upload a set of facial features, distribute by botnet to all infected cars, wait until target gets in range, accelerate. Optionally report success and uninstall malware in the milliseconds between inevitable outcome and actual impact to destroy evidence.
- arikrak 12y ago
  Or what if people take advantage of knowing what it will do...
dm2 12y ago
What if an autonomous vehicle see a pedestrian walking towards the road in front of the car (X meters in front but on a collision course with the car)? It calculates that if the human continues this path, they will collide. Should the car slow down so that no matter what the car can stop if the human decides to run out in front of the car or assume that the human will stop? What if you change human to deer? Autonomous vehicles should never be in the situation of hitting something. They should always calculate ahead of time what could possibly move into their path. If there is an obstacle then the car should slow down sufficiently to prevent an accident no matter what happens. I realize that there are near unlimited use-cases that will have to be accounted for. How would it handle a squirrel that is just sitting in the road? Stop and honk? Would it ever go around the obstacle by moving into an oncoming lane if it's deemed safe? Should they program in the ability to recognize the difference between a piece of carpet that has fallen onto the road and a board with nails or should it stop in the middle of the street for both?
vog 12y ago
My proposal for the solution of this dilemma: The car traffic development should go a route that is similar to air traffic controlling. That is, the security measures and controlling are increased so much that accidents happen very seldom, possibly once per month or less. Then, almost all of these moral issues would become irrelevant, simply because each crash would become an isolated case, literally.

47 points by arikrak 12y ago

The basic thought experiment is this: "If the robot car that hit me on my bike did so with provably the best intentions, should I still be angry?" At least with a person driving, I can criticize their thought patterns, their reactions, their choices--I can laud them for dodging the stroller, or curse them for drinking and swerving into me (provided I can still talk). With a machine, though, we lose any chance at morality. They act exactly as programmed, and we all know how reliable programs are. If, following such an accident, I were to be in a wheelchair for the rest of my life, I would want the entire stack trace that led up to my collision. I would want the unabridged source code and issue trackers. I would want the core dump. I would want every scrap of information that went into the vehicle's decision to zig instead of zagging. I'm not sure that we want to hand over so much blanket responsibility to black-boxes designed by committee and implemented by antisocial 20-something engineers working weird hours. EDIT: Reflecting on this a little has suggested, perhaps, that we should have one standard algorithm and package for doing this sort of thing, open-source and in the public domain. There shouldn't even be a question about allowing closed-source autonomous vehicles on the road.

eurleif 12y ago
You would want all that information for the purpose of proving the program was at fault, right? What if the company behind the software was automatically liable for all accidents? Wouldn't that answer that?
- angersock 12y ago
  I would want all that information as closure, since I wouldn't be able to walk again. Whether or not the program is functioning correctly, I want to know exactly how things went down--something that is finally possible if you automate driving. Holding the companies responsible is nice (and might as well be done), but until all cars are autonomous (and even then) there will be many cases where the initial conditions of an accident preclude anything but a harmful outcome. For example, a robot car doing the right thing could be boxed in by two malicious drivers and "forced" into hurting a bystander.
jotux 12y ago
>implemented by antisocial 20-something engineers working weird hours. I feel like this is a myth we should actively work to squash. The vast majority of engineers and coders are 30/40-somethings with normal social lives, families, and working 9-5 weekdays.
- angersock 12y ago
  Quite true--and yet, look at the companies making the news. Hell, in many cases your motion-planning is being done by people who were graduate students, and even then the stereotype...well, it fits, you know?
- FatalError 12y ago
  And even if they were, surely this is only a bad thing in that it might indicate that some potentially good engineers had been excluded from the workforce. Is reverse-bigotry really so far gone that some people genuinely think that being an "antisocial 20-something" makes a person a worse engineer?
  - dagw 12y ago
    One of the key factors in becoming a good engineer is experience. a 20-something, antisocial or otherwise, basically by definition is lacking in this experience. Sure they may be really really smart and write incredibly clever code, but it takes more than that to be a good engineer.
MiguelHudnandez 12y ago
I think another way to look at it is: if all cars were on auto-pilot, you would have a much smaller chance of being injured or killed on your bicycle. So if it happens a hundred thousand times more with human drivers, but there's a person you can point your finger at afterwards, does that make anyone more comfortable?
thenmar 12y ago
I'm having trouble understanding what point you're trying to make. Is it that if a computer bug causes an accident, the victim wouldn't have an individual to fantasize about getting revenge on? That the company that developed the software wouldn't be held accountable? That humans have "morals" that make them avoid accidents, but computers don't, so there will be more accidents? Please clarify this for us.
rando289 12y ago
> we lose any chance at morality Our morality is mostly learned. It can grow. Yes, we do need the source to be available.
- npizzolato 12y ago
  Do you? Where's the source for the programs flying commercial jets? Do you still fly?
  - rando289 12y ago
    Good point. However, I don't own the jet, nor have any liability/responsibility if it's software fails. I do own the car, and I do have liability/responsibility for its operation.

NPC82 12y ago

I think this problem is nonexistent to an autonomous car. A situation where the car has a choice to hit something would never happen. The only times humans have that choice is if we don't look far ahead enough or our field of view is impaired. Sure, autonomous cars may eventually hit something. But if they do it will be totally out of their control, by design. We can make them see further, stay focused, and be cautious. Humans can't always do that, and that's why we have to constantly choose our lesser evils.

rando289 12y ago
It is a real problem. It should be fairly obvious that it is. More realistically, It won't be a real time decision of hit this or that, it will be that it is programmed to do behavior X instead of Y. X is known to be 30% more likely to hit a pedestrian, but 10% less likely overall to be in any accident. Then you dramatize it by saying the car decided to hit a pedestrian instead of a car. Just substitute the behavior described in the story or some variation.
- thenmar 12y ago
  I think it will probably be much more complicated than that. Suppose the car is controlled by an artificial neural network that has been trained with a combination of real world experience and human-generated driving data. There is no giant switch statement that someone wrote for the outcome. Who is to blame then? Would using that data to train the model further be sufficient accountability?
aninhumer 12y ago
There will always be unexpected situations, especially while we still have human drivers on the road. Ultimately, if you want to actually get anywhere while there are other cars on the road, you have to make some assumptions about how they're going to behave. And even when you're completely automated, there may still be unexpected failures that cause these situations. However, I'm not sure the ethics of this are such a big deal, because the decision is such a tiny part of the causation of the eventual accident. I think it's far more likely that people will write code that just keeps trying to dodge until it can't any more, and I don't think that's a bad thing. Ultimately you have to make some compromises on safety vs resources, and deciding not to spend time writing special behaviour for pathological cases is no more ethically suspect than deciding not to spend any more time improving the brakes.

jfoster 12y ago

"As a matter of physics, you should choose a collision with a heavier vehicle that can better absorb the impact of a crash, which means programming the car to crash into the Volvo." Better for the car that is being hit, but not the car that we have control of. I'd like my robotic car to hit the car with more crumple or smaller relative momentum.

couchand 12y ago
You raise an interesting point: does the software controlling an automated car optimize for the owner of that car (as capitalism would suggest) or does it optimize for universal utility (as this article and many write-ups assume). I'd guess we wouldn't see as significant of gains if the cars optimize for their individual driver; however, it won't be long before some plucky manufacturer sees an untapped market and builds a selfish car.

NAFV_P 12y ago

> In the name of crash-optimization, you should program the car to crash into whatever can best survive the collision. In the last scenario, that meant smashing into the Volvo SUV. Here, it means striking the motorcyclist who’s wearing a helmet. A good algorithm would account for the much-higher statistical odds that the biker without a helmet would die, and surely killing someone is one of the worst things auto manufacturers desperately want to avoid. This could encourage bikers to refrain from donning protective gear. This reminds me of "Fight Club", where the central (anonymous) character describes his job in crash investigations. He has to decide whether recalling all instances of a make of car is less expensive than getting sued by the drivers in court.

DigitalSea 12y ago

The kind of scenario Wired propose between hitting two cars based on which would absorb the impact the better or cause the least harm aren't the kind of decisions humans make during a crash. I've been in a couple of car accidents caused by other people luckily and not once during any of those crashes did I weigh up my options between which car or which direction to swerve. Maybe we will expect more from an autonomous for its ability to make these kind of decisions, but I think it's an argument with a few holes. A lot of cars are currently at a point where they come standard with sensors that can allow a car to automatically parallel park itself between two other cars on a busy street. Smart cruise control which will know when to brake if the distance between your car and the car in-front gets smaller, cameras that follow the lines on a road. It's all there, car technology is already insanely smart. The questions people aren't asking here about autonomous cars is not what if it has to choose between two targets to hit, but rather what happens when a bug in the software controlling the vehicle rears its head and the car gets confused and reacts to something that isn't really happening? A faulty sensor thinks you're about to hit something when you're driving on a straight road and swerves into a tree trying to avoid an accident that wasn't even about to take place, now that's a scary thought. That would be my biggest worry. I am aware modern jetliner planes are essentially fly by wire except during takeoff and landing, but if the computer fails, another takes its place and on the rare occasion redundancy fails, the pilot and crew are there to manually guide the plane. The arguments presented here aren't even realistic. Why should autonomous computers be expected to make such decisions based on environmental factors like if a car can handle the impact or not? For accidents where the outcome will be the same (choosing between two vehicles for example) does it matter what the thinking behind the decision is? There is going to be an accident and no matter how in-depth the analysis is, depending on the speed and location, someone is bound to get hurt to an extent or killed. I have a much better idea and it's far cheaper and more realistic: if an accident is about to take place, the car slams on the brakes and the car slides to a stop. Technology like that already exists in some cars. Swerving in a car accident is most of the time the wrong choice as well, you're most likely to cause more harm to someone swerving in an unpredictable direction then you are straight on, unless of course you can see someone ahead, something on the road or about to have a head-on collision with another car.

sounds 12y ago

Schwolop 12y ago

I have personally written algorithms to select paths for 800kg autonomous vehicles operating around humans and livestock, and that minimise some cost function. My cost functions always treated crashes as (effectively) infinite cost. In the limit case where all possible paths had such an infinite cost, the failure mode was to revoke any electrical system that could increase the kinetic energy of the vehicle, apply maximum braking power (and/or any other subsystem that would serve reduce the kinetic energy of the vehicle), disconnect any chemical stores of energy from any activation system (i.e. stall the engine), and then finally - and as a direct consequence of these actions - maintain heading/steering angle. Since we elected to revoke all power systems, we lose the ability to steer. We're attempting to reduce the kinetic energy of the system to zero as rapidly as possible (since we've computed that there are no viable controlled manoeuvres available), and we concluded that the best (and in fact only) option was to maintain trajectory whilst trying to slow down. As a result, I don't think this argument has merit. There is no concept of "who should we crash into" because both are unacceptable, and we'd let physics take over instead.

TheSoftwareGuy 12y ago
Not to be pedantic, but I think you mean to reduce the kinetic energy, not potential. kinetic energy is associated with movement, potential is not.
- Schwolop 12y ago
  Yes, you're totally right and I've edited my phrasing. I was trying to cover the issue of not allowing any more petrol to be turned into kinetic energy, and muddled my own brain... :-)
stickfigure 12y ago
Opting to hold direction is a technical limitation of your system; it's not obvious that disconnecting all power systems is inherently the safest choice - especially as other vehicles react to the situation. Or at the very least, there is a long ethical decision tree of probabilities that must be evaluated before a crash is 100% certain. Do you pick a 48% chance of hitting the Volvo or a 26% chance of hitting the motorcycle? I predict that the ultimate solution will be to pick the path that optimizes for survival of the occupants of the automated vehicle. After all, they paid for it.

jerf 12y ago

Another edge case I've wondered about is whether there will be driving circumstances during inclement weather in which a sensibly-programmed autonomous car will simply pull over and refuse to operate, even as human drivers happily drive forth. Autonomous cars will be capable of producing reasonably objective measurements of driving condition safety... I wouldn't be surprised winter storms are more routinely capable of inducing conditions that are worse than we humans really realize. Or, alternatively, that we realize are dangerous but choose to drive in anyhow, but no autonomous care programmer would ever choose to accept.

tunesmith 12y ago

This reminds me of the Trolley Problem [1]. It could be argued it is the difference between utilitarianism and value ethics. Or around a recent dinner table, the star trek metaphor was "The needs of the many outweigh the needs of the view" versus the "Prime Directive". The former would actively choose whatever crash would do the least damage by some utility function, while the latter would see that a crash would happen either way and thereby turn off any guidance technology since it could then be argued to not be responsible for an active choice in either direction. [1] http://en.wikipedia.org/wiki/Trolley_problem http://en.wikipedia.org/wiki/Trolley_problem

Schwolop 12y ago
The second option (turning off all guidance) is effectively what I did (see https://news.ycombinator.com/item?id=7708105 https://news.ycombinator.com/item?id=7708105). Upon recognising that we had no controlled trajectories that were viable, we just tried to reduce kinetic energy as fast as possible without making any trajectory changes at all.

bagels 12y ago

Why program the car to choose to crash in to things? This makes no sense. What if the sensors are wrong and there was no second car? It seems more prudent to just program it to avoid crashing in to things, even if that is an impossible goal in some circumstances. In those cases, a better fallback would be to reduce velocity, and keep the car on the road or road shoulder perhaps?

ArchD 12y ago

In an unexpected complex situation for which there is very little training data and the behavioral dynamics are not merely about physics, namely impending accident with other cars controlled by other entities that can also respond to this car, any kind of simplistic reasoning like "hit the SUV because it can survive better" has no scientific basis. It's not clear what is being optimized for. For example, how do you know that SUVs are not more likely to have babies who are more fragile, so hitting the SUV is not necessarily better in terms of reducing fatalities. Also, the other cars are responding to this car. This is not about shooting stationary targets.

glomph 12y ago

The other day I was thinking how easy it would be to DoS a robotic car. You could just walk in front of it and then out the way again, over and over. How would they deal with malicious pedestrians?

IvyMike 12y ago
Of course you could already do this to a human car. In which case you'd call a human cop. And thus the answer becomes obvious. Robocar is DoS'd? Call Robocop.

sdfjkl 12y ago

This headline got me thinking, what if such a car gets infected with malware? Let's see. I don't know much about self-driving cars, but I imagine they use plenty of cameras and image processing software. That might make it pretty suitable to run some facial recognition software. Now multiply that with the botnets we already have today. The result might be assassination by malware: Upload a set of facial features, distribute by botnet to all infected cars, wait until target gets in range, accelerate. Optionally report success and uninstall malware in the milliseconds between inevitable outcome and actual impact to destroy evidence.

arikrak 12y ago
Or what if people take advantage of knowing what it will do...

dm2 12y ago

What if an autonomous vehicle see a pedestrian walking towards the road in front of the car (X meters in front but on a collision course with the car)? It calculates that if the human continues this path, they will collide. Should the car slow down so that no matter what the car can stop if the human decides to run out in front of the car or assume that the human will stop? What if you change human to deer? Autonomous vehicles should never be in the situation of hitting something. They should always calculate ahead of time what could possibly move into their path. If there is an obstacle then the car should slow down sufficiently to prevent an accident no matter what happens. I realize that there are near unlimited use-cases that will have to be accounted for. How would it handle a squirrel that is just sitting in the road? Stop and honk? Would it ever go around the obstacle by moving into an oncoming lane if it's deemed safe? Should they program in the ability to recognize the difference between a piece of carpet that has fallen onto the road and a board with nails or should it stop in the middle of the street for both?

vog 12y ago

My proposal for the solution of this dilemma: The car traffic development should go a route that is similar to air traffic controlling. That is, the security measures and controlling are increased so much that accidents happen very seldom, possibly once per month or less. Then, almost all of these moral issues would become irrelevant, simply because each crash would become an isolated case, literally.