8 ms·

This isn't just about Uber -- all semi-autonomous cars are going to have this problem. The people behind the wheel will become distracted, increasingly so as th

This isn't just about Uber -- all semi-autonomous cars are going to have this problem. The people behind the wheel will become distracted, increasingly so as the technology becomes better and they need to intervene less and less often. Even people who are trained to pay attention while testing self-driving cars become distracted, so we can assume the general public will become distracted too. The only viable autonomous cars are ones that will never require human intervention, because humans aren't able to intervene effectively and may make situations worse as they snap out of a distracted state. But we can't get to full autonomy without first testing the technology, and no artificial test conditions could replicate the complexity of the real world. Society will either learn to accept some avoidable deaths during the testing phase or will ban the technology. I suspect a few more incidents will result in a ban, and we will not have fully autonomous cars for a very long time.

mc32 9y ago
While I agree with much of what you say, I think we can have autonomous vehicles without having to be 100% independent of human intervention. It will require an effective "early warning system" which foresees issues it may not be able to process --however, this will require more inter-automobile communication and extravehicular communication (GPS, other inputs). Relying on only what the vehicle has on-board will probably not be good enough, for now. These supporting systems would communicate "trouble ahead" (ambiguous information, incomplete information, lack of information, unknown information, etc.)
- snaily 9y ago
  I think you're describing SAE level 4, while the parent is arguing for level 3 being unsafe. Level 3 means the user has to take over from an active driving situation within a pre-specified time period, whereas level 4 means the car will always be able to hand over in a "safe" state (an thus with unbounded response time).
agumonkey 9y ago
As other people sometime say, there can be self driving slow vehicles. For elders, for tourism, for near-home moving of things or just parking/unparking. In the meantime level 5 auto automation can mature in test areas.
- akira2501 9y ago
  > there can be self driving slow vehicles. This is not a safe option, particularly on certain roadways. Accidents increase when you have a higher levels of relative speed difference between lanes of traffic. Having "turtle mode AI cars" isn't going to help us. Speaking of the elderly.. they often end up in fatal accidents even at low speeds. Their frail bodies combined with strong safety restraints are not a good combination; particularly in the rear seats where their level of risk increases to nearly 2x the front seats.
  - agumonkey 9y ago
    I meant reaaally slow. Think not-road-accepted slow. Golf cart speeds at 'worst'. Maybe these aren't cars. But you can still transport some stuff and people in specific places and usage. Otherwise I agree with you.
    - nradov 9y ago
      You're referring to Neighborhood Electric Vehicles. Those operate under special rules in many areas. https://en.wikipedia.org/wiki/Neighborhood_Electric_Vehicle https://en.wikipedia.org/wiki/Neighborhood_Electric_Vehicle
- ramphastidae 9y ago
  Great comment. Uber's behavior shows it is interested in securing a monopoly of local transportation markets, not self-driving technology. Self-driving technology is a means to that end. If they cared about self-driving technology itself they would be starting with much smaller applications and working their way up rather than forcing themselves on to our streets as quickly as possible before their funding runs out.
snaily 9y ago
Also known as the "Level 3 is inherently unsafe" argument. Waymo noticed this back in 2012: https://driverless.wonderhowto.com/news/waymo-was-right-why-every-car-maker-should-skip-level-3-0178497/ https://driverless.wonderhowto.com/news/waymo-was-right-why-...
- digi_owl 9y ago
  In case others are wondering about the details about the levels: https://en.wikipedia.org/wiki/Autonomous_car#Classification https://en.wikipedia.org/wiki/Autonomous_car#Classification
- stale2002 9y ago
  Well, so level 3 can be safe under certain conditions and still be world changing. For example, imagine a self driving truck that works perfectly, but only during the day, on highways, when it is not raining. Such a system would still put millions out of work, as firing half of the truck drivers is possible if something works when it is sunny, on the highway.
  - rco8786 9y ago
    I’ve been pretty convinced for a while that this is effectively how it will play out. Effectively the “iRobot” model of designated areas/highways that are autonomous and highly controlled/protected and surface streets will remain human controlled for a long time.
  - tjoff 9y ago
    At least level 4 is required for that.
  - twblalock 9y ago
    Truck drivers do a lot more than just drive trucks: http://marginalrevolution.com/marginalrevolution/2018/02/will-truckers-automated-comments.html http://marginalrevolution.com/marginalrevolution/2018/02/wil...
    - 9y ago
digi_owl 9y ago
I wonder if we can gain any insights from experiences with airplane autopilots.
- greenyoda 9y ago
  Airplane autopilots operate in a completely different environment, in which planes' flight paths are set and monitored by Air Traffic Control. A plane on autopilot would never have to dodge a pedestrian, and would be separated from other planes by thousands of feet.
  - akira2501 9y ago
    > in which planes' flight paths are set They're 'set' in the sense that if communications is lost between the pilot and ATC, there's a pre-determined and more importantly pre-authorized path for the aircraft to fly through. > A plane on autopilot would never have to dodge a pedestrian It happens quite a bit, which is why we invented TCAS. [1] A common misconception is that _all_ traffic is required to file a flight plan, and that's only true for IFR (instrument) traffic and not for VFR (visual) traffic. So, an IFR flight can, and will, encouter VFR flights that interfere with their flight. > and would be separated from other planes by thousands of feet. In certain situations, in transit regions and during clearance/delivery, this is not true. 1: https://en.wikipedia.org/wiki/Traffic_collision_avoidance_system https://en.wikipedia.org/wiki/Traffic_collision_avoidance_sy...
    - shanghaiaway 9y ago
      An airplane has never had to dodge a pedestrian.
      - jlgaddis 9y ago
        Since you said "never", I'll just mention that there are videos on Youtube from skydivers who would disagree with you.
        9y ago
jay_kyburz 9y ago
> But we can't get to full autonomy without first testing the technology, and no artificial test conditions could replicate the complexity of the real world. Actually, I think a test environment could be a lot more complex. It's mind blowing to me that there doesn't seem to be some big track somewhere with automated crash test dummies riding their bikes across the street in the dark. Automated cars running red lights and stop signs. Automated accidents in front of self driving cars. Automated kids chasing a football across the road. It's mind blowing to me that governments do not require self driving cars do well on all these test before they allow the cars on the road. Governments should prepare these test, and car technology should complete the tests before they are allowed on the road. And when a car fails any real world situation, we can step back and create new tests. Really: We can test to any complexity we want. A plan lading on the road? A flood? A heard of sheep or flock of birds? A dust storm or other weather than might prevent the cars sensors working.
- TAForObvReasons 9y ago
  Proper development requires a conservative and methodical approach. Uber is racing to be the first, rushing to launch before properly testing cases like those you enumerated. As they would say, "Move Fast and Break Things"
  - twodave 9y ago
    People
- as2587 9y ago
  fyi -- people are working on this: https://mcity.umich.edu/ https://mcity.umich.edu/
- inverse_pi 9y ago
  There are test tracks. Both Waymo and Uber have publicly said that they have fake cities to test the code https://www.theatlantic.com/technology/archive/2017/08/inside-waymos-secret-testing-and-simulation-facilities/537648/ https://www.theatlantic.com/technology/archive/2017/08/insid... http://www.businessinsider.com/ubers-fake-city-pittsburgh-self-driving-cars-2017-10 http://www.businessinsider.com/ubers-fake-city-pittsburgh-se...
nradov 9y ago
Most autonomous vehicle companies are approaching the problem backwards. They're trying to build a vehicle where the computer is driving until it gets confused, and then asks a human (either in the vehicle or remote operator) to take over with little notice in an emergency. But we could get greater safety benefits sooner by having the human driver always drive with the computer overriding control inputs when it detects an unsafe situation. The basic concept is already proven with stability control systems and, more recently, front collision avoidance (automatic braking) systems. We should focus on expanding and extending those systems to cover more situations and the full range of vehicle control axes.
- newnewpdro 9y ago
  There's more money and power to be had by taking control away from people. Fundamentally that is what's going on here. This is a massive power grab. This is convincing people to give up control of their transportation under a facade of improved safety and convenience. Imagine streets full of powerful multi-ton wheeled drones relatively few people have control over the software operating. Ignoring conspiracy scenarios, just imagine how 0-day hacks of that system manifest. None of these situations are possible until you convince everyone to take ownership of one of these robots, or at least accept and utilize a ridesharing fleet of them.
- 05 9y ago
  That's impossible in reality, because computer doesn't have God's view of the environment - it can only predict things with a degree of confidence. It can't say for sure whether you get T-boned at an intersection you enter at 35MPH the moment the light turns green, so it won't reliably save you from a collision. A fully self driving system, on the other hand, won't enter the intersection without waiting 1-2 seconds, and would lower its speed below the speed limit when visibility is limited. A system that constantly keeps braking 'for no reason' won't get very far in the real world, because people would be very frustrated with it.
  - nradov 9y ago
    It's not at all impossible. It's already being done. Existing vehicles with forward collision avoidance systems are working well today. We'll never reach zero risk and you can always contrive some scenarios where automated systems wouldn't help. But it's still worth pursuing incremental improvements. For example, street sign recognition is getting pretty good. In a few years it should be possible to build a system that detects when the driver is about to run through a stop light or sign, and automatically brake.
deleted 9y ago
[deleted]
BurningFrog 9y ago
Surely this is a solved problem in several other contexts/industries?
joshuak 9y ago
Human attention vs low probably events is a known and solved problem. I've heard the TSA does this. You simply introduce fake events into the system at a frequency judged to cause human operators to stay alert, and responsive.
WalterSear 9y ago
They don't >have< to have this problem. The problem is that the robot should be watching the person, not vice-versa. This however, can't be sold as 'almost self-driving' to a gullible public.
icc97 9y ago
I think however the other way round does work. That is computers do stay alert and are able to take over with very short notice. This is essentially human augmentation. The regular collision detection in many cars now is a good example. It's a relatively simple system that should stop a distracted driver from crashing. Musk said something similar for the Tesla trucks, the auto-pilot can take over if the driver falls asleep or becomes unwell. Again this is augmentation. The only way to keep the driver alert is to have them driving all the time.

mc32 9y ago

While I agree with much of what you say, I think we can have autonomous vehicles without having to be 100% independent of human intervention. It will require an effective "early warning system" which foresees issues it may not be able to process --however, this will require more inter-automobile communication and extravehicular communication (GPS, other inputs). Relying on only what the vehicle has on-board will probably not be good enough, for now. These supporting systems would communicate "trouble ahead" (ambiguous information, incomplete information, lack of information, unknown information, etc.)

snaily 9y ago
I think you're describing SAE level 4, while the parent is arguing for level 3 being unsafe. Level 3 means the user has to take over from an active driving situation within a pre-specified time period, whereas level 4 means the car will always be able to hand over in a "safe" state (an thus with unbounded response time).

agumonkey 9y ago

As other people sometime say, there can be self driving slow vehicles. For elders, for tourism, for near-home moving of things or just parking/unparking. In the meantime level 5 auto automation can mature in test areas.

akira2501 9y ago
> there can be self driving slow vehicles. This is not a safe option, particularly on certain roadways. Accidents increase when you have a higher levels of relative speed difference between lanes of traffic. Having "turtle mode AI cars" isn't going to help us. Speaking of the elderly.. they often end up in fatal accidents even at low speeds. Their frail bodies combined with strong safety restraints are not a good combination; particularly in the rear seats where their level of risk increases to nearly 2x the front seats.
- agumonkey 9y ago
  I meant reaaally slow. Think not-road-accepted slow. Golf cart speeds at 'worst'. Maybe these aren't cars. But you can still transport some stuff and people in specific places and usage. Otherwise I agree with you.
  - nradov 9y ago
    You're referring to Neighborhood Electric Vehicles. Those operate under special rules in many areas. https://en.wikipedia.org/wiki/Neighborhood_Electric_Vehicle https://en.wikipedia.org/wiki/Neighborhood_Electric_Vehicle
ramphastidae 9y ago
Great comment. Uber's behavior shows it is interested in securing a monopoly of local transportation markets, not self-driving technology. Self-driving technology is a means to that end. If they cared about self-driving technology itself they would be starting with much smaller applications and working their way up rather than forcing themselves on to our streets as quickly as possible before their funding runs out.

snaily 9y ago

Also known as the "Level 3 is inherently unsafe" argument. Waymo noticed this back in 2012: https://driverless.wonderhowto.com/news/waymo-was-right-why-every-car-maker-should-skip-level-3-0178497/ https://driverless.wonderhowto.com/news/waymo-was-right-why-...

digi_owl 9y ago
In case others are wondering about the details about the levels: https://en.wikipedia.org/wiki/Autonomous_car#Classification https://en.wikipedia.org/wiki/Autonomous_car#Classification
stale2002 9y ago
Well, so level 3 can be safe under certain conditions and still be world changing. For example, imagine a self driving truck that works perfectly, but only during the day, on highways, when it is not raining. Such a system would still put millions out of work, as firing half of the truck drivers is possible if something works when it is sunny, on the highway.
- rco8786 9y ago
  I’ve been pretty convinced for a while that this is effectively how it will play out. Effectively the “iRobot” model of designated areas/highways that are autonomous and highly controlled/protected and surface streets will remain human controlled for a long time.
- tjoff 9y ago
  At least level 4 is required for that.
- twblalock 9y ago
  Truck drivers do a lot more than just drive trucks: http://marginalrevolution.com/marginalrevolution/2018/02/will-truckers-automated-comments.html http://marginalrevolution.com/marginalrevolution/2018/02/wil...
  - 9y ago

digi_owl 9y ago

I wonder if we can gain any insights from experiences with airplane autopilots.

greenyoda 9y ago
Airplane autopilots operate in a completely different environment, in which planes' flight paths are set and monitored by Air Traffic Control. A plane on autopilot would never have to dodge a pedestrian, and would be separated from other planes by thousands of feet.
- akira2501 9y ago
  > in which planes' flight paths are set They're 'set' in the sense that if communications is lost between the pilot and ATC, there's a pre-determined and more importantly pre-authorized path for the aircraft to fly through. > A plane on autopilot would never have to dodge a pedestrian It happens quite a bit, which is why we invented TCAS. [1] A common misconception is that _all_ traffic is required to file a flight plan, and that's only true for IFR (instrument) traffic and not for VFR (visual) traffic. So, an IFR flight can, and will, encouter VFR flights that interfere with their flight. > and would be separated from other planes by thousands of feet. In certain situations, in transit regions and during clearance/delivery, this is not true. 1: https://en.wikipedia.org/wiki/Traffic_collision_avoidance_system https://en.wikipedia.org/wiki/Traffic_collision_avoidance_sy...
  - shanghaiaway 9y ago
    An airplane has never had to dodge a pedestrian.
    - jlgaddis 9y ago
      Since you said "never", I'll just mention that there are videos on Youtube from skydivers who would disagree with you.
      - 9y ago

jay_kyburz 9y ago

> But we can't get to full autonomy without first testing the technology, and no artificial test conditions could replicate the complexity of the real world. Actually, I think a test environment could be a lot more complex. It's mind blowing to me that there doesn't seem to be some big track somewhere with automated crash test dummies riding their bikes across the street in the dark. Automated cars running red lights and stop signs. Automated accidents in front of self driving cars. Automated kids chasing a football across the road. It's mind blowing to me that governments do not require self driving cars do well on all these test before they allow the cars on the road. Governments should prepare these test, and car technology should complete the tests before they are allowed on the road. And when a car fails any real world situation, we can step back and create new tests. Really: We can test to any complexity we want. A plan lading on the road? A flood? A heard of sheep or flock of birds? A dust storm or other weather than might prevent the cars sensors working.

TAForObvReasons 9y ago
Proper development requires a conservative and methodical approach. Uber is racing to be the first, rushing to launch before properly testing cases like those you enumerated. As they would say, "Move Fast and Break Things"
- twodave 9y ago
  People
as2587 9y ago
fyi -- people are working on this: https://mcity.umich.edu/ https://mcity.umich.edu/
inverse_pi 9y ago
There are test tracks. Both Waymo and Uber have publicly said that they have fake cities to test the code https://www.theatlantic.com/technology/archive/2017/08/inside-waymos-secret-testing-and-simulation-facilities/537648/ https://www.theatlantic.com/technology/archive/2017/08/insid... http://www.businessinsider.com/ubers-fake-city-pittsburgh-self-driving-cars-2017-10 http://www.businessinsider.com/ubers-fake-city-pittsburgh-se...

nradov 9y ago

Most autonomous vehicle companies are approaching the problem backwards. They're trying to build a vehicle where the computer is driving until it gets confused, and then asks a human (either in the vehicle or remote operator) to take over with little notice in an emergency. But we could get greater safety benefits sooner by having the human driver always drive with the computer overriding control inputs when it detects an unsafe situation. The basic concept is already proven with stability control systems and, more recently, front collision avoidance (automatic braking) systems. We should focus on expanding and extending those systems to cover more situations and the full range of vehicle control axes.

newnewpdro 9y ago
There's more money and power to be had by taking control away from people. Fundamentally that is what's going on here. This is a massive power grab. This is convincing people to give up control of their transportation under a facade of improved safety and convenience. Imagine streets full of powerful multi-ton wheeled drones relatively few people have control over the software operating. Ignoring conspiracy scenarios, just imagine how 0-day hacks of that system manifest. None of these situations are possible until you convince everyone to take ownership of one of these robots, or at least accept and utilize a ridesharing fleet of them.
05 9y ago
That's impossible in reality, because computer doesn't have God's view of the environment - it can only predict things with a degree of confidence. It can't say for sure whether you get T-boned at an intersection you enter at 35MPH the moment the light turns green, so it won't reliably save you from a collision. A fully self driving system, on the other hand, won't enter the intersection without waiting 1-2 seconds, and would lower its speed below the speed limit when visibility is limited. A system that constantly keeps braking 'for no reason' won't get very far in the real world, because people would be very frustrated with it.
- nradov 9y ago
  It's not at all impossible. It's already being done. Existing vehicles with forward collision avoidance systems are working well today. We'll never reach zero risk and you can always contrive some scenarios where automated systems wouldn't help. But it's still worth pursuing incremental improvements. For example, street sign recognition is getting pretty good. In a few years it should be possible to build a system that detects when the driver is about to run through a stop light or sign, and automatically brake.

deleted 9y ago

[deleted]

BurningFrog 9y ago

Surely this is a solved problem in several other contexts/industries?

joshuak 9y ago

Human attention vs low probably events is a known and solved problem. I've heard the TSA does this. You simply introduce fake events into the system at a frequency judged to cause human operators to stay alert, and responsive.

WalterSear 9y ago

They don't >have< to have this problem. The problem is that the robot should be watching the person, not vice-versa. This however, can't be sold as 'almost self-driving' to a gullible public.

icc97 9y ago

I think however the other way round does work. That is computers do stay alert and are able to take over with very short notice. This is essentially human augmentation. The regular collision detection in many cars now is a good example. It's a relatively simple system that should stop a distracted driver from crashing. Musk said something similar for the Tesla trucks, the auto-pilot can take over if the driver falls asleep or becomes unwell. Again this is augmentation. The only way to keep the driver alert is to have them driving all the time.