6 ms·

>these adverse weather conditions can lead to large numbers of collisions at the same time. Cars sliding out of control on a snowy road is not a systemic failu

>these adverse weather conditions can lead to large numbers of collisions at the same time. Cars sliding out of control on a snowy road is not a systemic failure. If you have an entire fleet of self-driving cars using the same (or similar) technology, the exact same condition could cause all of them to misbehave in the exact the same way. This could happen simultaneously with a large number of cars. Or could be something that propagates in a cascading manner. (A car breaks, turns and spins out of control. The next car detects that, breaks, turns and spins out of control, etc.) If you have no clear idea how your driving software works, how are you going to make sure this doesn't happen? People have common sense, variability and adaptive learning to prevent things like that. Moreover, our traffic system, our cars and our driver education have been co-evolving prioritizing safety for the last 100 years. (Heck, some of it started with horses and carriages.) The complexity and size of the system have been gradually scaled up for just as long, and it was designed and refined to be used by humans. It's astounding how many of AI enthusiasts fail to see these issues and differences. Traffic system isn't a board game.

johnwyles 8y ago
> If you have no clear idea how your driving software works, how are you going to make sure this doesn't happen? A lot of people don't know how typing "www.google.com" into their browser works but DNS "just works" for them. There is software in nearly every part of our lives that could systemically go wrong in catastrophic ways - we do our best to plan and prevent them. > People have common sense, variability and adaptive learning to prevent things like that. Moreover, our traffic system, our cars and our driver education have been co-evolving for safety for the last 100 years. (Heck, some of it started with horses and carriages.) The complexity and size of the system have been gradually scaled up for just as long, and it's designed and refined to be used by humans. Isn't that incredible? The complexity goes up: we adapt. As self-driving cars are introduced we adapt and evolve and incorporate these changes in technology, the driving "environment", security, safety measures, and laws and regulations. > Traffic system isn't a board game. I didn't see any efforts at self-driving vehicles just say "ahhhh! this driving stuff is so easy!" Of course it comes with huge difficulties, an enormous amount of complexity, a baffling amount of data, a ton of edge cases, and now a wealth of tribal knowledge on the subject. When considering these things think of the systemic benefits as well. There was a story on the Google self-driving cars in Mountain View not taking a turn when the passengers of the car expected it would. Turns out that the IR sensors on the vehicle saw that a jogger behind a wall of hedges was about to enter the intersection to cross while jogging. Systemically implementing that change across a fleet of cars would then save untold numbers of injuries or lives. We adapt.
- laythea 8y ago
  How about: I like driving so don't take it away. or perhaps: Just because we can, doesn't mean we should.
- apta 8y ago
  > Turns out that the IR sensors on the vehicle saw that a jogger behind a wall of hedges was about to enter the intersection to cross while jogging. Some cars today already have IR sensors (night vision systems) installed in them, and include pedestrian detection monitors as well. Cars don't have to be full on self-driving to have a large increase in safety and accident prevention.
- YeGoblynQueenne 8y ago
  >> A lot of people don't know how typing "www.google.com" into their browser works but DNS "just works" for them. On the other hand, noone was ever run over by a mistyped URL.
SiVal 8y ago
To guard against the risk of a bug/design flaw affecting all vehicles simultaneously, I would include an independent "dumb" override that was too dumb to drive but could stop all vehicles as soon as any of several TBD conditions occurred, regardless of what the primary "smart" system was telling the vehicle to do. It wouldn't be "smart" AI, but just something listening to both the AI's smart sensors and it's own, independent, simple sensors, and if ANY of them reported a listed condition, it would override the smart system and stop traffic. Human intervention would be required to enable the vehicles to move again. After some real-world experience, I would want to give the emergency stop system the ability to release the hold on one layer of vehicles at a time, allowing the smart AI to drive again but at a speed limited by the "dumb" system to nothing faster than human walking speed, enabling them to clear the area. If they cleared without further incident, the next layer would be released until the whole stoppage either cleared itself or was restopped because something was still wrong.
darawk 8y ago
Self driving cars do not represent a correlated risk. The situation you describe is completely fanciful. There is no plausible circumstance in which a large number of cars would simultaneously crash.
- neolefty 8y ago
  A 0-day vulnerability in their OS, exploited ...
- jcranmer 8y ago
  Here's a link to a 100-car pileup: https://www.washingtonpost.com/news/dr-gridlock/wp/2018/02/05/stunning-drone-footage-shows-massive-100-vehicle-pileup-on-snow-covered-highway/?noredirect=on&utm_term=.cf2a0b91ae70 https://www.washingtonpost.com/news/dr-gridlock/wp/2018/02/0... These kinds of crashes generally have the some root cause: everybody is traveling too fast for conditions, and when one thing goes wrong, the end result is that you get a chain reaction of everyone accreting into a single crash. If self-driving cars are capable of making the same misjudgements, then these kinds of crashes absolutely can happen.
  - darawk 8y ago
    Sure, but you can't criticize SDC's by saying that they have the same failure modes as humans. The comment I was referring to was suggesting that they'd have some sort of mass failure mode, in which some substantial percentage of all operating SDC's in an area would crash simultaneously. That is clearly not going to happen.

johnwyles 8y ago

> If you have no clear idea how your driving software works, how are you going to make sure this doesn't happen? A lot of people don't know how typing "www.google.com" into their browser works but DNS "just works" for them. There is software in nearly every part of our lives that could systemically go wrong in catastrophic ways - we do our best to plan and prevent them. > People have common sense, variability and adaptive learning to prevent things like that. Moreover, our traffic system, our cars and our driver education have been co-evolving for safety for the last 100 years. (Heck, some of it started with horses and carriages.) The complexity and size of the system have been gradually scaled up for just as long, and it's designed and refined to be used by humans. Isn't that incredible? The complexity goes up: we adapt. As self-driving cars are introduced we adapt and evolve and incorporate these changes in technology, the driving "environment", security, safety measures, and laws and regulations. > Traffic system isn't a board game. I didn't see any efforts at self-driving vehicles just say "ahhhh! this driving stuff is so easy!" Of course it comes with huge difficulties, an enormous amount of complexity, a baffling amount of data, a ton of edge cases, and now a wealth of tribal knowledge on the subject. When considering these things think of the systemic benefits as well. There was a story on the Google self-driving cars in Mountain View not taking a turn when the passengers of the car expected it would. Turns out that the IR sensors on the vehicle saw that a jogger behind a wall of hedges was about to enter the intersection to cross while jogging. Systemically implementing that change across a fleet of cars would then save untold numbers of injuries or lives. We adapt.

laythea 8y ago
How about: I like driving so don't take it away. or perhaps: Just because we can, doesn't mean we should.
apta 8y ago
> Turns out that the IR sensors on the vehicle saw that a jogger behind a wall of hedges was about to enter the intersection to cross while jogging. Some cars today already have IR sensors (night vision systems) installed in them, and include pedestrian detection monitors as well. Cars don't have to be full on self-driving to have a large increase in safety and accident prevention.
YeGoblynQueenne 8y ago
>> A lot of people don't know how typing "www.google.com" into their browser works but DNS "just works" for them. On the other hand, noone was ever run over by a mistyped URL.

SiVal 8y ago

To guard against the risk of a bug/design flaw affecting all vehicles simultaneously, I would include an independent "dumb" override that was too dumb to drive but could stop all vehicles as soon as any of several TBD conditions occurred, regardless of what the primary "smart" system was telling the vehicle to do. It wouldn't be "smart" AI, but just something listening to both the AI's smart sensors and it's own, independent, simple sensors, and if ANY of them reported a listed condition, it would override the smart system and stop traffic. Human intervention would be required to enable the vehicles to move again. After some real-world experience, I would want to give the emergency stop system the ability to release the hold on one layer of vehicles at a time, allowing the smart AI to drive again but at a speed limited by the "dumb" system to nothing faster than human walking speed, enabling them to clear the area. If they cleared without further incident, the next layer would be released until the whole stoppage either cleared itself or was restopped because something was still wrong.

darawk 8y ago

Self driving cars do not represent a correlated risk. The situation you describe is completely fanciful. There is no plausible circumstance in which a large number of cars would simultaneously crash.

neolefty 8y ago
A 0-day vulnerability in their OS, exploited ...
jcranmer 8y ago
Here's a link to a 100-car pileup: https://www.washingtonpost.com/news/dr-gridlock/wp/2018/02/05/stunning-drone-footage-shows-massive-100-vehicle-pileup-on-snow-covered-highway/?noredirect=on&utm_term=.cf2a0b91ae70 https://www.washingtonpost.com/news/dr-gridlock/wp/2018/02/0... These kinds of crashes generally have the some root cause: everybody is traveling too fast for conditions, and when one thing goes wrong, the end result is that you get a chain reaction of everyone accreting into a single crash. If self-driving cars are capable of making the same misjudgements, then these kinds of crashes absolutely can happen.
- darawk 8y ago
  Sure, but you can't criticize SDC's by saying that they have the same failure modes as humans. The comment I was referring to was suggesting that they'd have some sort of mass failure mode, in which some substantial percentage of all operating SDC's in an area would crash simultaneously. That is clearly not going to happen.