6 ms·

The problem is that there are literally uncountably many situations that a human with "general intelligence" will understand and react to accordingly. Sometimes

by anyfoo 4y ago

The problem is that there are literally uncountably many situations that a human with "general intelligence" will understand and react to accordingly. Sometimes smoothly, sometimes less so. But a non-conscious automatic entity needs to have the required behavior programmed in explicitly. So yes, you might argue that for this particular situation, you "just" need to put in the proper programming and AI/ML training and then "maybe the car will notice more often than a human" as long as the situation is within very specific bounds. At least now that somebody made an article about it. But it does not change that the autonomous machine does not understand the complex world it is driving in in the slightest, and that, for example, swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump.

robbrown451 4y ago
"But it does not change that the autonomous machine does not understand the complex world it is driving in in the slightest, and that, for example, swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump." It seems it would have a way of prioritizing such things. That doesn't seem particularly complicated, to be honest... weighted decision making is certainly within its capacity. E.g. shopping cart has fewer "avoid hitting" points than stroller. It's not like every scenario has to be explicitly programmed in, nor does the program need to run some analysis on a detailed backstory to justify that a baby is more valuable than groceries. In effect, somebody -- probably not a programmer either -- just needs to enter some numbers into a spreadsheet. (yes there is complex programming that allows that to be manifested in the car's decisions, but the idea that programmers are themselves constantly making "moral calls" in the code, rather than the control data, is fiction) And if it does have such prioritization in its logic, I'd say yeah, it "understands" the world in that respect. Unless you have defined the word "understand" in some mystical way that precludes non-biological machines by definition.
- anyfoo 4y ago
  > It seems it would have a way of prioritizing such things. You are putting the cart before the horse. The problem is not in prioritization, the problem is in having the correct ontology to even get to the "prioritization" stage. Does the car know what a fruit stand is? Does it know what a gas pump is? Does it know how the fruit stand relates to the gas pump in "expected outcome when being hit by a car"? If you say "we can program that in", read my post again.
  - robbrown451 4y ago
    I would imagine the car has ways of identifying things that are specific hazards, such as a gas pump. Fruit stand is probably categorized as "other." "the problem is in having the correct ontology to even get to the "prioritization" stage." That part isn't done by the program, it is done by whoever enters the prioritization numbers. That is, someone, possibly a committee, can dial up the "avoid gas pumps" weighting relative to the "avoid baby stroller" weighting if they are concerned that cars might swerve so widely to avoid coming near a stroller than they are risking hitting a different hazard. Or they can dial up the weight of grocery carts relative to dogs, since children might be in a grocery cart. Etc. Those are humans, who can do whatever ontological analysis they need when deciding on the the settings. The car doesn't need to access any of that, it just needs a general look up table that can help make optimal decisions based on the human-entered value system.
    - dogecoinbase 4y ago
      Sure, we'll just write down all the things there are in the world in order of how bad it is for a car to hit them.
      - omgwtfbyobbq 4y ago
        We're really lucky we never see more than 1 thing at a time on the road.
      - robbrown451 4y ago
        You don't think self driving cars categorize and prioritize things?
      - dane-pgp 4y ago
        I mean, you're right; someone making that list might not think to include "centaur", and maybe one Halloween a child is dressed up as one, and the computer vision system interprets the "centaur" as a horse instead of a child, and it makes the wrong decision, but how many centaur-related accidents do you think self-driving cars are going to be involved in each year? It's completely feasible to imagine writing a list of the top, say, 100 things in the world that a car needs to make morally-significant decisions about, and then deal with every other accident or near-miss after the fact. Interactions with unrecognised objects should be rare enough to be a rounding error when comparing accident rates between autonomous and human-driven miles.
- deleted 4y ago
  [deleted]
- 2fast4you 4y ago
  You're asking the car to make moral decisions. Given the choice between hitting a child or and old woman, which will it choose? A bicyclist vs a pedestrian? A lemonade vendor vs a hotdog vendor? This is all assuming it can distinguish between all of these objects, and that a real person could assign relative moral values to hitting one over the other.
acchow 4y ago
On the other hand, I’ve seen an enormous number of human-driven vehicles be absolutely assholes to emergency vehicles with lights blinking
feoren 4y ago
> needs to have the required behavior programmed in explicitly This is missing the entire point of ML. ML is literally defined as not having to explicitly program responses in for every situation. Needing to pull over because a fire truck has told you it's coming your way in 2 minutes is pretty easy compared with some of those other "uncountably many" situations these cars need to deal with. > the autonomous machine does not understand the complex world it is driving in Daniel Dennett would like to have a word with you. It's perfectly possible for systems to "understand" things for any useful definition of the word "understand". A calculator absolutely "understands" arithmetic.
- hgomersall 4y ago
  Yeah, you generally have to hope it effectively interpolates between stuff it has been trained on in ways that make sense.
- bowsamic 4y ago
  Do driverless cars use ML in the decision making procedure? > A calculator absolutely "understands" arithmetic. That is absurd and obviously false
  - feoren 4y ago
    > That is absurd and obviously false It is actually "obviously true" unless you believe that human brains have a special metaphysical magic that makes them "more than just a system". That's literally the only alternative: human brains are magic and only humans are ever capable of "understanding". It's a vacuous definition of the word. Systems can understand things, which is good, because the human brain is nothing more. See: Daniel Dennett's response to Searle's Chinese Room.
    - anyfoo 4y ago
      I did not claim it's about "metaphysical magic" that machines lack. But I do believe that humans usually have a very rich and multifaceted life outside of driving on a road, compared to cars, and are therefore able to integrate "training data" that cars are not. Unless your plan is to make them an active member of society.
ithkuil 4y ago
While I do agree that even a below average human intelligence can better cope with this kind of situations (also because it has access to many more opportunities to understand, for example we can understand what's going on by seeing at the face and the hand waving and/or yelling if the driver in the other vehicle; humans are quite good at understanding other humans), I don't think it follows that conscience is necessary to achieve that.
gondolgames 4y ago
> [...] swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump. That's a great example. Not to mention triggering any Rube Goldberg-machine-like chain reaction (even with just a few steps) where a series of events would need to be predicted.
gojomo 4y ago
There are not uncountably many – we live in a finite world governed by understandable physical and civic laws. There's an actually measurable rate of humans blocking other traffic, and for how long before resolution, and an actually measurable rate of autonomous vehicles blocking other traffic, and for how long before resolved. If the rate for autonomous vehicles is already below that of humans, or rapidly headed there, that's far more important to note than to theorize about other corner-cases. (Also, as a San Francisco driver, I have serious doubts about the "general intelligence" of my fellow drivers. I don't see any reasons to hold autonomous cars to a higher standard – perhaps a much higher standard? – than other cars.)
- routerl 4y ago
  > There are not uncountably many OP was obviously not using that term mathematically (i.e. the cardinality of the power set of natural numbers), and obviously meant something in the neighborhood of "effectively not countable". And, again, in all but the formal mathematical meaning of the word "countable", many things are not countable (e.g. no one will live long enough to count all the natural numbers). In the real world, assigning an ordinal number to an object/event/thing has a nonzero time cost. And accounting for every situation in software has a much greater cost. > If the rate for autonomous vehicles is already below that of humans, or rapidly headed there, that's far more important to note than to theorize about other corner-cases. Okay, but dangerous human drivers get systematically removed from the streets. Are we doing the same for self-driving cars? In this context, does every Tesla count as the same "driver"? Maybe all Teslas should have their autonomous driving centrally disabled every time one causes an accident, or breaks a law, until that specific issue is fixed. But it would be impossible to run a car company that way, so of course that takes priority over, say, keeping innocents alive, right? /S
  - DennisP 4y ago
    > Are we doing the same for self-driving cars? Effectively we are, as long as their developers fix the problems that led to the crash. Change the software and you have a new driver.
markdown 4y ago
HOWTO: Make a Tesla pull over with a fake siren.
- munk-a 4y ago
  We've solved that as a society. People can affix fake sirens to their cars and if they blare them at you you should pull over... after the incident society then throws an incredibly harsh penalty at the offender. To make sure things run smoothly it benefits society to never doubt or question whether people who say they're police officers actually them - they might have something incredibly important to say (Like, hey, there's an active shootout ahead - please don't keep driving and get yourself injured). If you as an individual create some funny gag to fake out autonomous vehicles into thinking that you're a cop whether you cheekily do it with "TOTALLY NOT A COP CAR" written on the side to get a laugh or not... you're almost certainly going to be charged with a felony crime.
DennisP 4y ago
Autonomous cars will crash in situations where humans wouldn't, but the opposite is also true. Autonomous cars don't fall asleep, drive drunk, or get distracted on their phones. Personally I try to stay alert while I drive, so I feel I'm safer driving myself than letting the machine do it. But I'm less confident that the best autonomous cars will have more accidents than humans in general, and they're improving all the time.
krisoft 4y ago
> for example, swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump. This is just ridiculous. If you believe that humans are making rational decisions in split seconds then you are delusional. A swerving, breakless human driven car will hit whatever happens to be where physics takes it. The scared monkey descendant holding on to the controls will as likely do the wrong thing as doing the right thing. Maybe a fighter jet pilot or a rally driver can do better but i wouldn’t count on it. And besides how did that AV end up swerving with no brake? This is the reason why autonomous vehicles are set up with redundant brake actuation. If I would have doubts about our ability to stop the car I would much sooner implement a third independent brake system than to try to solve whatever philosophical runaway trolley problem you are concoting here.
- anyfoo 4y ago
  Runaway trucks are a very real problem, we even have emergency escape ramps in places where they are most common. There is usually no split second decision here: You have a comparable eternity to decide where you want your non-braking truck to end up. However you construe my particular example to be "ridiculous" under the additional constraints that you imposed on it yourself, you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. In fact, a human, the garbage truck driver, had to react. They were not a "jet pilot" or "rally driver", and yet they were perfectly able to resolve the situation that the AV had gotten themselves into.
  - mlyle 4y ago
    > you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. And remote assistance/teleoperation was reacting at the same time, for the vehicle. Humans will screw up similar cases: end up in the emergency vehicle's way and panic and do the wrong thing. In fact, I saw a fire truck impeded Wednesday because of some judgment mistakes by a human driver that an AV would be unlikely to make. So, frequency of "doing the wrong thing", and severity (here, probably measured in seconds) are completely reasonable questions to ask-- even if the circumstances where each tends to mess up are different. I don't think it's reasonable to ask that the autonomous vehicle be superior than a typical human in every axis of performance, just overall superior.
mr_toad 4y ago
> But it does not change that the autonomous machine does not understand the complex world it is driving in in the slightest It only needs to understand when to ask for help: the driverless car had correctly yielded to the oncoming fire truck in the opposing lane and contacted the company’s remote assistance workers, who are able to operate vehicles in trouble from afar. According to Cruise, which collects camera and sensor data from its testing vehicles, the fire truck was able to move forward approximately 25 seconds after it first encountered the autonomous vehicle I don’t understand why people think that driverless cars need to deal with every one in a million scenario, it makes no sense.
- anyfoo 4y ago
  > I don’t understand why people think that driverless cars need to deal with every one in a million scenario, it makes no sense. That a particular situation is rare, does not mean that you won't encounter multiple rare situations in a given time frame. (By the way, the article stated in the beginning that the garbage truck had to move? Either way, it required manual intervention and a human's situational awareness. How does the car ask for the proper help at freeway speeds within seconds--not even split seconds?)
- mbreese 4y ago
  > It only needs This phrase is doing a lot of work here. It's one of my least favorite phrases (in a close race with "just do this") and is often associated with unrealistic feature requests. I'm sure there is a disagreement between SFFD and Cruise as to exactly what happened, but the article implies that the Cruise vehicle isn't the one that moved to fix the problem. > The fire truck only passed the blockage when the garbage truck driver ran from their work to move their vehicle. Even if the Cruise vehicle was able to call for help, the car not only needs to call for help, but also wait for a response (at 4am), and give a remote human enough information to control a car remotely in a safe manner. None of these things are easy... not impossible, but not "only needs" easy. > driverless cars need to deal with every one in a million scenario Of course driverless cars need to deal with one in a million scenarios. Human drivers deal with one in a million scenarios every day. Nothing is ever the same when driving, so there are always subtle changes. But even if there is an unusual situation, there must be some kind of response. Even if that response is to move to the side of the road and put on hazard lights to indicate that it doesn't know what to do (which it may have done)... that would have been a better response than to do nothing and sit to wait for a human. There should be a default "unknown input" failure mode. The disagreement here is that SFFD didn't like how the Cruise vehicle failed. Maybe there is a better approach. We are expecting these vehicles to move us around 24/7. That's a lot of trips. At this rate, one in a million scenarios will happen every day. That's the problem with large numbers -- even rare events are to be expected when N is high enough.
InvertedRhodium 4y ago
There are also uncountable ways the human can be erratic, antagonistic, etc.

by anyfoo 4y ago

robbrown451 4y ago

"But it does not change that the autonomous machine does not understand the complex world it is driving in in the slightest, and that, for example, swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump." It seems it would have a way of prioritizing such things. That doesn't seem particularly complicated, to be honest... weighted decision making is certainly within its capacity. E.g. shopping cart has fewer "avoid hitting" points than stroller. It's not like every scenario has to be explicitly programmed in, nor does the program need to run some analysis on a detailed backstory to justify that a baby is more valuable than groceries. In effect, somebody -- probably not a programmer either -- just needs to enter some numbers into a spreadsheet. (yes there is complex programming that allows that to be manifested in the car's decisions, but the idea that programmers are themselves constantly making "moral calls" in the code, rather than the control data, is fiction) And if it does have such prioritization in its logic, I'd say yeah, it "understands" the world in that respect. Unless you have defined the word "understand" in some mystical way that precludes non-biological machines by definition.

anyfoo 4y ago
> It seems it would have a way of prioritizing such things. You are putting the cart before the horse. The problem is not in prioritization, the problem is in having the correct ontology to even get to the "prioritization" stage. Does the car know what a fruit stand is? Does it know what a gas pump is? Does it know how the fruit stand relates to the gas pump in "expected outcome when being hit by a car"? If you say "we can program that in", read my post again.
- robbrown451 4y ago
  I would imagine the car has ways of identifying things that are specific hazards, such as a gas pump. Fruit stand is probably categorized as "other." "the problem is in having the correct ontology to even get to the "prioritization" stage." That part isn't done by the program, it is done by whoever enters the prioritization numbers. That is, someone, possibly a committee, can dial up the "avoid gas pumps" weighting relative to the "avoid baby stroller" weighting if they are concerned that cars might swerve so widely to avoid coming near a stroller than they are risking hitting a different hazard. Or they can dial up the weight of grocery carts relative to dogs, since children might be in a grocery cart. Etc. Those are humans, who can do whatever ontological analysis they need when deciding on the the settings. The car doesn't need to access any of that, it just needs a general look up table that can help make optimal decisions based on the human-entered value system.
  - dogecoinbase 4y ago
    Sure, we'll just write down all the things there are in the world in order of how bad it is for a car to hit them.
    - omgwtfbyobbq 4y ago
      We're really lucky we never see more than 1 thing at a time on the road.
    - robbrown451 4y ago
      You don't think self driving cars categorize and prioritize things?
    - dane-pgp 4y ago
      I mean, you're right; someone making that list might not think to include "centaur", and maybe one Halloween a child is dressed up as one, and the computer vision system interprets the "centaur" as a horse instead of a child, and it makes the wrong decision, but how many centaur-related accidents do you think self-driving cars are going to be involved in each year? It's completely feasible to imagine writing a list of the top, say, 100 things in the world that a car needs to make morally-significant decisions about, and then deal with every other accident or near-miss after the fact. Interactions with unrecognised objects should be rare enough to be a rounding error when comparing accident rates between autonomous and human-driven miles.
deleted 4y ago
[deleted]
2fast4you 4y ago
You're asking the car to make moral decisions. Given the choice between hitting a child or and old woman, which will it choose? A bicyclist vs a pedestrian? A lemonade vendor vs a hotdog vendor? This is all assuming it can distinguish between all of these objects, and that a real person could assign relative moral values to hitting one over the other.

acchow 4y ago

On the other hand, I’ve seen an enormous number of human-driven vehicles be absolutely assholes to emergency vehicles with lights blinking

feoren 4y ago

> needs to have the required behavior programmed in explicitly This is missing the entire point of ML. ML is literally defined as not having to explicitly program responses in for every situation. Needing to pull over because a fire truck has told you it's coming your way in 2 minutes is pretty easy compared with some of those other "uncountably many" situations these cars need to deal with. > the autonomous machine does not understand the complex world it is driving in Daniel Dennett would like to have a word with you. It's perfectly possible for systems to "understand" things for any useful definition of the word "understand". A calculator absolutely "understands" arithmetic.

hgomersall 4y ago
Yeah, you generally have to hope it effectively interpolates between stuff it has been trained on in ways that make sense.
bowsamic 4y ago
Do driverless cars use ML in the decision making procedure? > A calculator absolutely "understands" arithmetic. That is absurd and obviously false
- feoren 4y ago
  > That is absurd and obviously false It is actually "obviously true" unless you believe that human brains have a special metaphysical magic that makes them "more than just a system". That's literally the only alternative: human brains are magic and only humans are ever capable of "understanding". It's a vacuous definition of the word. Systems can understand things, which is good, because the human brain is nothing more. See: Daniel Dennett's response to Searle's Chinese Room.
  - anyfoo 4y ago
    I did not claim it's about "metaphysical magic" that machines lack. But I do believe that humans usually have a very rich and multifaceted life outside of driving on a road, compared to cars, and are therefore able to integrate "training data" that cars are not. Unless your plan is to make them an active member of society.

ithkuil 4y ago

While I do agree that even a below average human intelligence can better cope with this kind of situations (also because it has access to many more opportunities to understand, for example we can understand what's going on by seeing at the face and the hand waving and/or yelling if the driver in the other vehicle; humans are quite good at understanding other humans), I don't think it follows that conscience is necessary to achieve that.

gondolgames 4y ago

> [...] swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump. That's a great example. Not to mention triggering any Rube Goldberg-machine-like chain reaction (even with just a few steps) where a series of events would need to be predicted.

gojomo 4y ago

There are not uncountably many – we live in a finite world governed by understandable physical and civic laws. There's an actually measurable rate of humans blocking other traffic, and for how long before resolution, and an actually measurable rate of autonomous vehicles blocking other traffic, and for how long before resolved. If the rate for autonomous vehicles is already below that of humans, or rapidly headed there, that's far more important to note than to theorize about other corner-cases. (Also, as a San Francisco driver, I have serious doubts about the "general intelligence" of my fellow drivers. I don't see any reasons to hold autonomous cars to a higher standard – perhaps a much higher standard? – than other cars.)

routerl 4y ago
> There are not uncountably many OP was obviously not using that term mathematically (i.e. the cardinality of the power set of natural numbers), and obviously meant something in the neighborhood of "effectively not countable". And, again, in all but the formal mathematical meaning of the word "countable", many things are not countable (e.g. no one will live long enough to count all the natural numbers). In the real world, assigning an ordinal number to an object/event/thing has a nonzero time cost. And accounting for every situation in software has a much greater cost. > If the rate for autonomous vehicles is already below that of humans, or rapidly headed there, that's far more important to note than to theorize about other corner-cases. Okay, but dangerous human drivers get systematically removed from the streets. Are we doing the same for self-driving cars? In this context, does every Tesla count as the same "driver"? Maybe all Teslas should have their autonomous driving centrally disabled every time one causes an accident, or breaks a law, until that specific issue is fixed. But it would be impossible to run a car company that way, so of course that takes priority over, say, keeping innocents alive, right? /S
- DennisP 4y ago
  > Are we doing the same for self-driving cars? Effectively we are, as long as their developers fix the problems that led to the crash. Change the software and you have a new driver.

markdown 4y ago

HOWTO: Make a Tesla pull over with a fake siren.

munk-a 4y ago
We've solved that as a society. People can affix fake sirens to their cars and if they blare them at you you should pull over... after the incident society then throws an incredibly harsh penalty at the offender. To make sure things run smoothly it benefits society to never doubt or question whether people who say they're police officers actually them - they might have something incredibly important to say (Like, hey, there's an active shootout ahead - please don't keep driving and get yourself injured). If you as an individual create some funny gag to fake out autonomous vehicles into thinking that you're a cop whether you cheekily do it with "TOTALLY NOT A COP CAR" written on the side to get a laugh or not... you're almost certainly going to be charged with a felony crime.

DennisP 4y ago

Autonomous cars will crash in situations where humans wouldn't, but the opposite is also true. Autonomous cars don't fall asleep, drive drunk, or get distracted on their phones. Personally I try to stay alert while I drive, so I feel I'm safer driving myself than letting the machine do it. But I'm less confident that the best autonomous cars will have more accidents than humans in general, and they're improving all the time.

krisoft 4y ago

> for example, swerving into an unmanned fruit stand without being able to brake is much better than swerving into an unmanned gas pump. This is just ridiculous. If you believe that humans are making rational decisions in split seconds then you are delusional. A swerving, breakless human driven car will hit whatever happens to be where physics takes it. The scared monkey descendant holding on to the controls will as likely do the wrong thing as doing the right thing. Maybe a fighter jet pilot or a rally driver can do better but i wouldn’t count on it. And besides how did that AV end up swerving with no brake? This is the reason why autonomous vehicles are set up with redundant brake actuation. If I would have doubts about our ability to stop the car I would much sooner implement a third independent brake system than to try to solve whatever philosophical runaway trolley problem you are concoting here.

anyfoo 4y ago
Runaway trucks are a very real problem, we even have emergency escape ramps in places where they are most common. There is usually no split second decision here: You have a comparable eternity to decide where you want your non-braking truck to end up. However you construe my particular example to be "ridiculous" under the additional constraints that you imposed on it yourself, you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. In fact, a human, the garbage truck driver, had to react. They were not a "jet pilot" or "rally driver", and yet they were perfectly able to resolve the situation that the AV had gotten themselves into.
- mlyle 4y ago
  > you are looking at an instance where an autonomous vehicle failed to make a reasonable decision that a human would right in this article. And remote assistance/teleoperation was reacting at the same time, for the vehicle. Humans will screw up similar cases: end up in the emergency vehicle's way and panic and do the wrong thing. In fact, I saw a fire truck impeded Wednesday because of some judgment mistakes by a human driver that an AV would be unlikely to make. So, frequency of "doing the wrong thing", and severity (here, probably measured in seconds) are completely reasonable questions to ask-- even if the circumstances where each tends to mess up are different. I don't think it's reasonable to ask that the autonomous vehicle be superior than a typical human in every axis of performance, just overall superior.

mr_toad 4y ago

> But it does not change that the autonomous machine does not understand the complex world it is driving in in the slightest It only needs to understand when to ask for help: the driverless car had correctly yielded to the oncoming fire truck in the opposing lane and contacted the company’s remote assistance workers, who are able to operate vehicles in trouble from afar. According to Cruise, which collects camera and sensor data from its testing vehicles, the fire truck was able to move forward approximately 25 seconds after it first encountered the autonomous vehicle I don’t understand why people think that driverless cars need to deal with every one in a million scenario, it makes no sense.

anyfoo 4y ago
> I don’t understand why people think that driverless cars need to deal with every one in a million scenario, it makes no sense. That a particular situation is rare, does not mean that you won't encounter multiple rare situations in a given time frame. (By the way, the article stated in the beginning that the garbage truck had to move? Either way, it required manual intervention and a human's situational awareness. How does the car ask for the proper help at freeway speeds within seconds--not even split seconds?)
mbreese 4y ago
> It only needs This phrase is doing a lot of work here. It's one of my least favorite phrases (in a close race with "just do this") and is often associated with unrealistic feature requests. I'm sure there is a disagreement between SFFD and Cruise as to exactly what happened, but the article implies that the Cruise vehicle isn't the one that moved to fix the problem. > The fire truck only passed the blockage when the garbage truck driver ran from their work to move their vehicle. Even if the Cruise vehicle was able to call for help, the car not only needs to call for help, but also wait for a response (at 4am), and give a remote human enough information to control a car remotely in a safe manner. None of these things are easy... not impossible, but not "only needs" easy. > driverless cars need to deal with every one in a million scenario Of course driverless cars need to deal with one in a million scenarios. Human drivers deal with one in a million scenarios every day. Nothing is ever the same when driving, so there are always subtle changes. But even if there is an unusual situation, there must be some kind of response. Even if that response is to move to the side of the road and put on hazard lights to indicate that it doesn't know what to do (which it may have done)... that would have been a better response than to do nothing and sit to wait for a human. There should be a default "unknown input" failure mode. The disagreement here is that SFFD didn't like how the Cruise vehicle failed. Maybe there is a better approach. We are expecting these vehicles to move us around 24/7. That's a lot of trips. At this rate, one in a million scenarios will happen every day. That's the problem with large numbers -- even rare events are to be expected when N is high enough.

InvertedRhodium 4y ago

There are also uncountable ways the human can be erratic, antagonistic, etc.