4 ms·

> for example, swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump. This is just ridiculou

> for example, swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump. This is just ridiculous. If you believe that humans are making rational decisions in split seconds then you are delusional. A swerving, breakless human driven car will hit whatever happens to be where physics takes it. The scared monkey descendant holding on to the controls will as likely do the wrong thing as doing the right thing. Maybe a fighter jet pilot or a rally driver can do better but i wouldn’t count on it. And besides how did that AV end up swerving with no brake? This is the reason why autonomous vehicles are set up with redundant brake actuation. If I would have doubts about our ability to stop the car I would much sooner implement a third independent brake system than to try to solve whatever philosophical runaway trolley problem you are concoting here.

anyfoo 4y ago
Runaway trucks are a very real problem, we even have emergency escape ramps in places where they are most common. There is usually no split second decision here: You have a comparable eternity to decide where you want your non-braking truck to end up. However you construe my particular example to be "ridiculous" under the additional constraints that you imposed on it yourself, you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. In fact, a human, the garbage truck driver, had to react. They were not a "jet pilot" or "rally driver", and yet they were perfectly able to resolve the situation that the AV had gotten themselves into.
- mlyle 4y ago
  > you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. And remote assistance/teleoperation was reacting at the same time, for the vehicle. Humans will screw up similar cases: end up in the emergency vehicle's way and panic and do the wrong thing. In fact, I saw a fire truck impeded Wednesday because of some judgment mistakes by a human driver that an AV would be unlikely to make. So, frequency of "doing the wrong thing", and severity (here, probably measured in seconds) are completely reasonable questions to ask-- even if the circumstances where each tends to mess up are different. I don't think it's reasonable to ask that the autonomous vehicle be superior than a typical human in every axis of performance, just overall superior.
- krisoft 4y ago
  > Runaway trucks are a very real problem Yes. And autonomous trucks are an engineering problem, not an abstract philosophical question. You can ask the question: “how is the autonomous truck going to know if it should slam into the petrol station or into the fruit stand?” And there is no good answer to that. Or you can ask: “How do we engineer autonomous trucks so the probability of a runaway incident is lower than epsilon?” And then suddenly it turns out this is a solveable problem with our existing tools. (With redundant brakes, and with built-in brake health checks.) > under the additional constraints that you imposed on it yourself I assume you mean comparing to what a competent human would do? It is implicit in the whole discussion. Human drivers are the current best practice. You are asking about the “fruit stand vs petrol station” question presumably because a human would know to choose the fruit stand. Nobody is asking this alternate question, because they would immediately feel it is ridiculous: A young boy is crossing the road in front of an AV. In 30 years he will become a politician, will instigate a violent sectarian war which will result in the death of millions of innocents. Should the AV run him over thus preventing all that suffering? Just to state the obvious: no, the AV should not run the boy over. But why is nobody asking this question? Because it is obviously silly. We as a human can’t look at a young boy and know them as a future mass murderer, therefore we don’t expect this from an autonomous car either. > you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. Oh yes. And it is a fascinating one. I was reacting to your “fruit stand vs petrol pump” hypothetical not to the article directly.
  - bcrl 4y ago
    In the real world someone has to program the self driving system to make the decision about how to react. That is, there is a software team somewhere that is going to have to decide what behaviour to program into the system for the trolley problem. So, your statement that it is not an abstract philosophical question is patently false. Obligatory link to The Good Place making the trolley problem real here: https://www.youtube.com/watch?v=DtRhrfhP5b4 https://www.youtube.com/watch?v=DtRhrfhP5b4 It is hopelessly naive to think that these problems can simply be engineered away. In the real world failures happen in redundant systems. Air brakes are supposed to "fail safe", but in reality a host of factors contribute to accidents: how well a truck or trailer's brakes are maintained, engine state, speed, loading, temperature and grade all combine to make them fail. Trains have multiple braking systems, yet sometimes all 3 fail and a spectacular accident occurs. In addition to all the traditional mechanical issues, self driving vehicles have tonnes of software failure modes that traditional cars do not. More importantly, those software issues are not well understood at this point in time. If you want to better understand why software can't be trusted to Do The Right Thing, go back and read investigations analyzing failures of systems that have come before. The Therac-25 is good over here: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 No system a human can build can be completely intrinsically safe. Mistakes by designers occur. Safety is a process that takes time and effort, and it will take decades for self driving cars to work out all the bugs.
rimunroe 4y ago
> If you believe that humans are making rational decisions in split seconds then you are delusional. A swerving, breakless human driven car will hit whatever happens to be where physics takes it. I was in a car accident a few years ago where someone left a stop sign without realizing that there was traffic (me) in a lane they couldn't see. I wouldn't describe having felt like time slowed down, but that isn't an absurd way to describe it. I had a weird sense of clarity for the few moments before impact. I was able to slam the brakes, but I was way too close to avoid hitting them. I had a distinct feeling of "the front of the car is heavier and there's a person there". I swerved left instead of right to avoid hitting the front and slammed into the rear passenger side of their car. This spun their car completely around and totaled both cars, but both of us were able to walk away without any injuries.
- calciphus 4y ago
  Some really fascinating studies have shown that your perception of time doesn't actually slow down in moments of heightened intensity, however the detail of the memory (when reflected on later) is higher than for a non-traumatic experience. An interesting summary of how they studied this if you're curious: https://www.livescience.com/2117-time-slow-emergencies.html https://www.livescience.com/2117-time-slow-emergencies.html

anyfoo 4y ago

Runaway trucks are a very real problem, we even have emergency escape ramps in places where they are most common. There is usually no split second decision here: You have a comparable eternity to decide where you want your non-braking truck to end up. However you construe my particular example to be "ridiculous" under the additional constraints that you imposed on it yourself, you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. In fact, a human, the garbage truck driver, had to react. They were not a "jet pilot" or "rally driver", and yet they were perfectly able to resolve the situation that the AV had gotten themselves into.

mlyle 4y ago
> you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. And remote assistance/teleoperation was reacting at the same time, for the vehicle. Humans will screw up similar cases: end up in the emergency vehicle's way and panic and do the wrong thing. In fact, I saw a fire truck impeded Wednesday because of some judgment mistakes by a human driver that an AV would be unlikely to make. So, frequency of "doing the wrong thing", and severity (here, probably measured in seconds) are completely reasonable questions to ask-- even if the circumstances where each tends to mess up are different. I don't think it's reasonable to ask that the autonomous vehicle be superior than a typical human in every axis of performance, just overall superior.
krisoft 4y ago
> Runaway trucks are a very real problem Yes. And autonomous trucks are an engineering problem, not an abstract philosophical question. You can ask the question: “how is the autonomous truck going to know if it should slam into the petrol station or into the fruit stand?” And there is no good answer to that. Or you can ask: “How do we engineer autonomous trucks so the probability of a runaway incident is lower than epsilon?” And then suddenly it turns out this is a solveable problem with our existing tools. (With redundant brakes, and with built-in brake health checks.) > under the additional constraints that you imposed on it yourself I assume you mean comparing to what a competent human would do? It is implicit in the whole discussion. Human drivers are the current best practice. You are asking about the “fruit stand vs petrol station” question presumably because a human would know to choose the fruit stand. Nobody is asking this alternate question, because they would immediately feel it is ridiculous: A young boy is crossing the road in front of an AV. In 30 years he will become a politician, will instigate a violent sectarian war which will result in the death of millions of innocents. Should the AV run him over thus preventing all that suffering? Just to state the obvious: no, the AV should not run the boy over. But why is nobody asking this question? Because it is obviously silly. We as a human can’t look at a young boy and know them as a future mass murderer, therefore we don’t expect this from an autonomous car either. > you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. Oh yes. And it is a fascinating one. I was reacting to your “fruit stand vs petrol pump” hypothetical not to the article directly.
- bcrl 4y ago
  In the real world someone has to program the self driving system to make the decision about how to react. That is, there is a software team somewhere that is going to have to decide what behaviour to program into the system for the trolley problem. So, your statement that it is not an abstract philosophical question is patently false. Obligatory link to The Good Place making the trolley problem real here: https://www.youtube.com/watch?v=DtRhrfhP5b4 https://www.youtube.com/watch?v=DtRhrfhP5b4 It is hopelessly naive to think that these problems can simply be engineered away. In the real world failures happen in redundant systems. Air brakes are supposed to "fail safe", but in reality a host of factors contribute to accidents: how well a truck or trailer's brakes are maintained, engine state, speed, loading, temperature and grade all combine to make them fail. Trains have multiple braking systems, yet sometimes all 3 fail and a spectacular accident occurs. In addition to all the traditional mechanical issues, self driving vehicles have tonnes of software failure modes that traditional cars do not. More importantly, those software issues are not well understood at this point in time. If you want to better understand why software can't be trusted to Do The Right Thing, go back and read investigations analyzing failures of systems that have come before. The Therac-25 is good over here: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 No system a human can build can be completely intrinsically safe. Mistakes by designers occur. Safety is a process that takes time and effort, and it will take decades for self driving cars to work out all the bugs.

rimunroe 4y ago

> If you believe that humans are making rational decisions in split seconds then you are delusional. A swerving, breakless human driven car will hit whatever happens to be where physics takes it. I was in a car accident a few years ago where someone left a stop sign without realizing that there was traffic (me) in a lane they couldn't see. I wouldn't describe having felt like time slowed down, but that isn't an absurd way to describe it. I had a weird sense of clarity for the few moments before impact. I was able to slam the brakes, but I was way too close to avoid hitting them. I had a distinct feeling of "the front of the car is heavier and there's a person there". I swerved left instead of right to avoid hitting the front and slammed into the rear passenger side of their car. This spun their car completely around and totaled both cars, but both of us were able to walk away without any injuries.

calciphus 4y ago
Some really fascinating studies have shown that your perception of time doesn't actually slow down in moments of heightened intensity, however the detail of the memory (when reflected on later) is higher than for a non-traumatic experience. An interesting summary of how they studied this if you're curious: https://www.livescience.com/2117-time-slow-emergencies.html https://www.livescience.com/2117-time-slow-emergencies.html