5 ms·

Yes, those (very rare, mostly hypothetical) situations are trolley problems, but OP was making an entirely different argument, referring to self driving as a wh

Yes, those (very rare, mostly hypothetical) situations are trolley problems, but OP was making an entirely different argument, referring to self driving as a whole being a form of trolley problem. I think the concerns of self driving “moral dilemma” problems are largely overblown. First, they are such rare situations: I’ve been driving for 20 years and never experienced one, I imagine few people have. Second, the solution is simple anyway: favor the passengers of the current car.

ghaff 5y ago
I almost have trouble imagining realistic scenarios that involve deliberately endangering the occupants of the vehicle in order to protect others. (And any manufacturer who designed such a system is probably not going to find many takers.) And, for that matter, there are very few scenarios where swerving and running into something at speed is a better decision than braking hard.
- Digit-Al 5y ago
  I'm the same, I can't really come up with any realistic vehicle accident scenario that would be a classic trolley problem. I may just be lacking in imagination though. I'd be interested if anyone could present a realistic example of such a scenario, and one that isn't wildly unlikely ever to happen. Something that does occur to me is that since humans tend to just automatically react in extreme situations whereas an AI would presumably have precious milliseconds to consider different scenarios, maybe an AI could create a trolley problem where it wouldn't exist for a human. For example, if someone steps out from behind a bus a human will most likely slam the brakes on instantly and will likely slide into the pedestrian at a slower but possibly still fatal pace; whereas an AI having a bit of time to think calmly about it might slam the car hard into the bus whilst also braking, which might slow the car down enough to prevent too much injury to the pedestrian whilst causing damage to the bus, writing off the car, and possibly injuring the occupant (though the car safety features should help).
  - bumby 5y ago
    >possibly injuring the occupant (though the car safety features should help) This seems to take the opposite viewpoint of the OP which said the "obvious" answer is to do whatever is best for the passengers. >one that isn't wildly unlikely ever to happen. A lot of the comments here are along these lines and, to a certain extent, they miss the point. Risk is a combination of probability and severity even before considering ethics. In the cases where probability can be reliably calculated, there needs to be threshold about what risks are accepted in order to implement informed decisions. I'll try to illustrate a couple examples. Say there is software that decides to take a specific action based on sensor input. Maybe the action is to accelerate and swerve to the left if an obstacle is detected to be oncoming from on the right at an intersection. Let's make it somewhat more complicated by having a jaywalker stepping off the curb on the left. There are (at least) two prospective events: 1) The car does not perform evasive maneuvers and increases the risk of being hit by the other vehicle 2) The car does perform evasive maneuvers and increases the risk of hitting the pedestrian Each has a probability and a severity. To make things simple, say the probability of each is 5%. However, they have different severities. In scenario 1) the severity to the pedestrian may be risk of injury or death even if the severity to the driver is fairly low (probably repair work covered by insurance). But in scenario 2) the severity to the pedestrian is negligible while the severity to the driver(s) is moderate. With modern cars, it's unlikely either driver will be killed is fairly small but there is an increased risk of sub-fatal injuries and likely much more damage to the vehicles. Remember, the risk = probability x severity. So if we take the simplistic approach of only taking in the perspective of the driver ("do whatever favors the passengers of the car") we have effectively minimized the passengers risk of moderate injury by increasing the pedestrian risk of grievance injury. Adding ethics gets even more complicated; does it change if the pedestrian is a child? A mother pushing a stroller? A homeless person? Another example would be what if the car is driving by a school playground and notices a ball roll between parked cars and into the road? A human can intuit playground + ball = a higher probability a child may run into the street after the ball. This might be enough to cause a human driver to apply the brakes hard even if a child isn't immediately visible. Would software intuit the same? Even if it did, would the car do the same if it increased the risk of a collision from behind? The safety-critical software world spend a lot of time on small-probability events. The more recent 737-Max issue has similar through-lines, though there are many confounding issues as is typical with safety mishaps. The MCAS was classified as "hazardous" severity and by company policy required two AOA sensors to reduce the probability of failure. This would effectively reduce the risk since risk = probability x severity. However, since the severity was "hazardous" and not "catastrophic" (i.e., they didn't think it could cause a plane to crash), the extra sensor was an optional software feature. Had the severity been appropriately attributed, I think there is a greater likelihood that the second AOA sensor would be a mandatory feature because they (I assume) have a specific risk threshold they are aiming for. All this to say, it's not enough just to say "These are small probability events so they don't have to be addressed" but rather probability is just a part of an overall risk strategy. The decision space with safety critical software is extremely large precisely because you must mitigate low probability events if the severity is large enough. I sometimes wonder if the fact that self-driving technology targets are missed is attributable to naivety of oversimplifying the problem.
bumby 5y ago
Managing those edge cases effectively is extremely important in safety critical software. Addressing these is often what separates quality critical software from low-quality. Would you want an aircraft software engineer or nuclear power plant engineer to disregard low probability events? It worries me when I get the impression the SV mindset of glorifying “moving fast and breaking things” infiltrating safety critical applications, particularly those that impact the general public. >Second, the solution is simple anyway: favor the passengers of the current car. I don’t think this is a given. Would the ethical software, for instance, drive a car through a crowd in order to save a lone passenger? Would you be okay with a human driver being absolved of responsibility for the same choice? My intuition is most would not, because we recognize there is an obligation to others as part of the social contract. It seems a naive oversimplification to not extend the same obligations to software that makes the choices in our place.
- MattRix 5y ago
  How is the car saving the passenger by driving through the crowd? That seems like an entirely imaginary scenario. In almost any real world scenario the solution is going to be to apply the brakes and come to a stop.
  - bumby 5y ago
    It is an imaginary scenario. But so is the scenario where they can always apply the brakes and come to a stop. What if there is a cement truck tailgaiting and they can't hard brake? What if there are jersey barriers constraining the maneuverability? Either of those would risk the passenger more than running into pedestrians. Performing failure-mode-effects-analysis on software is largely an exercise in creating imaginary scenarios and then effectively mitigating the risks to within an acceptable level. I agree, in most real-world scenarios the tough problems can be mitigated largely by "slow down and re-assess". But that's not what safety-critical software risk mitigation is always about. You have to mitigate low-probability events as well, especially in a domain with governmental regulation. According to the NTSB report, the Uber accident that killed a pedestrian applied the "slow down and re-assess" scenario that attributed to the death. The software was misclassifying enough that an artificial delay was put in to avoid (I assume) nuisance braking. So in the real-world, they didn't want to always apply the "just brake if in trouble" strategy because of trade-offs to the passengers and it didn't work out well.

ghaff 5y ago

I almost have trouble imagining realistic scenarios that involve deliberately endangering the occupants of the vehicle in order to protect others. (And any manufacturer who designed such a system is probably not going to find many takers.) And, for that matter, there are very few scenarios where swerving and running into something at speed is a better decision than braking hard.

Digit-Al 5y ago
I'm the same, I can't really come up with any realistic vehicle accident scenario that would be a classic trolley problem. I may just be lacking in imagination though. I'd be interested if anyone could present a realistic example of such a scenario, and one that isn't wildly unlikely ever to happen. Something that does occur to me is that since humans tend to just automatically react in extreme situations whereas an AI would presumably have precious milliseconds to consider different scenarios, maybe an AI could create a trolley problem where it wouldn't exist for a human. For example, if someone steps out from behind a bus a human will most likely slam the brakes on instantly and will likely slide into the pedestrian at a slower but possibly still fatal pace; whereas an AI having a bit of time to think calmly about it might slam the car hard into the bus whilst also braking, which might slow the car down enough to prevent too much injury to the pedestrian whilst causing damage to the bus, writing off the car, and possibly injuring the occupant (though the car safety features should help).
- bumby 5y ago
  >possibly injuring the occupant (though the car safety features should help) This seems to take the opposite viewpoint of the OP which said the "obvious" answer is to do whatever is best for the passengers. >one that isn't wildly unlikely ever to happen. A lot of the comments here are along these lines and, to a certain extent, they miss the point. Risk is a combination of probability and severity even before considering ethics. In the cases where probability can be reliably calculated, there needs to be threshold about what risks are accepted in order to implement informed decisions. I'll try to illustrate a couple examples. Say there is software that decides to take a specific action based on sensor input. Maybe the action is to accelerate and swerve to the left if an obstacle is detected to be oncoming from on the right at an intersection. Let's make it somewhat more complicated by having a jaywalker stepping off the curb on the left. There are (at least) two prospective events: 1) The car does not perform evasive maneuvers and increases the risk of being hit by the other vehicle 2) The car does perform evasive maneuvers and increases the risk of hitting the pedestrian Each has a probability and a severity. To make things simple, say the probability of each is 5%. However, they have different severities. In scenario 1) the severity to the pedestrian may be risk of injury or death even if the severity to the driver is fairly low (probably repair work covered by insurance). But in scenario 2) the severity to the pedestrian is negligible while the severity to the driver(s) is moderate. With modern cars, it's unlikely either driver will be killed is fairly small but there is an increased risk of sub-fatal injuries and likely much more damage to the vehicles. Remember, the risk = probability x severity. So if we take the simplistic approach of only taking in the perspective of the driver ("do whatever favors the passengers of the car") we have effectively minimized the passengers risk of moderate injury by increasing the pedestrian risk of grievance injury. Adding ethics gets even more complicated; does it change if the pedestrian is a child? A mother pushing a stroller? A homeless person? Another example would be what if the car is driving by a school playground and notices a ball roll between parked cars and into the road? A human can intuit playground + ball = a higher probability a child may run into the street after the ball. This might be enough to cause a human driver to apply the brakes hard even if a child isn't immediately visible. Would software intuit the same? Even if it did, would the car do the same if it increased the risk of a collision from behind? The safety-critical software world spend a lot of time on small-probability events. The more recent 737-Max issue has similar through-lines, though there are many confounding issues as is typical with safety mishaps. The MCAS was classified as "hazardous" severity and by company policy required two AOA sensors to reduce the probability of failure. This would effectively reduce the risk since risk = probability x severity. However, since the severity was "hazardous" and not "catastrophic" (i.e., they didn't think it could cause a plane to crash), the extra sensor was an optional software feature. Had the severity been appropriately attributed, I think there is a greater likelihood that the second AOA sensor would be a mandatory feature because they (I assume) have a specific risk threshold they are aiming for. All this to say, it's not enough just to say "These are small probability events so they don't have to be addressed" but rather probability is just a part of an overall risk strategy. The decision space with safety critical software is extremely large precisely because you must mitigate low probability events if the severity is large enough. I sometimes wonder if the fact that self-driving technology targets are missed is attributable to naivety of oversimplifying the problem.

bumby 5y ago

Managing those edge cases effectively is extremely important in safety critical software. Addressing these is often what separates quality critical software from low-quality. Would you want an aircraft software engineer or nuclear power plant engineer to disregard low probability events? It worries me when I get the impression the SV mindset of glorifying “moving fast and breaking things” infiltrating safety critical applications, particularly those that impact the general public. >Second, the solution is simple anyway: favor the passengers of the current car. I don’t think this is a given. Would the ethical software, for instance, drive a car through a crowd in order to save a lone passenger? Would you be okay with a human driver being absolved of responsibility for the same choice? My intuition is most would not, because we recognize there is an obligation to others as part of the social contract. It seems a naive oversimplification to not extend the same obligations to software that makes the choices in our place.

MattRix 5y ago
How is the car saving the passenger by driving through the crowd? That seems like an entirely imaginary scenario. In almost any real world scenario the solution is going to be to apply the brakes and come to a stop.
- bumby 5y ago
  It is an imaginary scenario. But so is the scenario where they can always apply the brakes and come to a stop. What if there is a cement truck tailgaiting and they can't hard brake? What if there are jersey barriers constraining the maneuverability? Either of those would risk the passenger more than running into pedestrians. Performing failure-mode-effects-analysis on software is largely an exercise in creating imaginary scenarios and then effectively mitigating the risks to within an acceptable level. I agree, in most real-world scenarios the tough problems can be mitigated largely by "slow down and re-assess". But that's not what safety-critical software risk mitigation is always about. You have to mitigate low-probability events as well, especially in a domain with governmental regulation. According to the NTSB report, the Uber accident that killed a pedestrian applied the "slow down and re-assess" scenario that attributed to the death. The software was misclassifying enough that an artificial delay was put in to avoid (I assume) nuisance braking. So in the real-world, they didn't want to always apply the "just brake if in trouble" strategy because of trade-offs to the passengers and it didn't work out well.