5 ms·

That's around 7.71 crashes/million miles driven by Waymo. By comparison, US national statistics from 2018 [1] are around 2.08 crashes/million miles driven. Se

That's around 7.71 crashes/million miles driven by Waymo. By comparison, US national statistics from 2018 [1] are around 2.08 crashes/million miles driven. Seems like they're far from human performance. There are ton of ways to slice this data, though. I used national stats for accidents where the police showed up; maybe Waymo's data includes no-police accidents. It would also be interesting to compare injury statistics. And of course this is Phoenix-specific data from Waymo, which is a place with good road conditions. [1] https://cdan.nhtsa.gov/tsftables/National%20Statistics.pdf https://cdan.nhtsa.gov/tsftables/National%20Statistics.pdf (edited to include simulated crashes, which were only averted through human safety driver intervention.)

brixon 6y ago
Did you count all incidents or only the ones that happened in real life and not the simulations?
- maliker 6y ago
  I omitted the simulations in the Waymo count. Good idea to add them in?
  - IshKebab 6y ago
    Yeah you should include them because they were only prevented by a safety driver who won't exist in the long term.
    - maliker 6y ago
      Thanks! I missed that definition in the article. Updated the ratios. So it's actually 3.7x more accidents not 1.5x. Definitely changed my conclusion.
- danielvf 6y ago
  Simulations counts were times when a safety driver took control of the vechile, and post-incident simulations with the same data show that the AI would have crashed the car without the safety driver. So fair to include them.
beambot 6y ago
You also need to account for confounding variables, such as geography. Does Phoenix mirror the nation as a whole?
bufferoverflow 6y ago
Phonenix doesn't get much snow / ice / crazy fog / crazy rain. The roads are wide. The amount of traffic is low compared to, say, NYC or LA. So yeah, they are not near human level yet.
- maxerickson 6y ago
  I think there's a solid chance that a 'near human capable' driving system will handle slippery roads better than the human average. The system will be in a modern vehicle with traction control and likely make conservative judgments about road condition.
  - adrianmonk 6y ago
    There might also be differences in how accurately a human or machine can model which maneuvers can/can't be done without losing traction. I could see this going either way. Maybe a human will be able to identify spots (like puddles) where traction will be especially bad, but a machine can do a detailed physics simulation of a maneuver before trying it. In whatever way a machine does adjust its behavior for slippery roads, it will probably be more consistent at it than a human. Humans have to deal with force of habit, so for example they may momentarily forget that they can't brake hard or they shouldn't take that turn at a familiar intersection at the same speed as yesterday.
  - notatoad 6y ago
    also poor visibility. humans need to rely on their eyes, a car with a bunch of lidar sensors is absolutely going to outperform us in fog, blowing snow, or heavy rain.
- bryanlarsen 6y ago
  I suspect that Waymo will do better compared to a human outside of Phoenix than inside it. The hardest thing that Waymo will have to do is to interact with humans behaving oddly. Phoenix has those, just not as many as NYC/LA/SF. But one thing computers do exceedingly well is scale. Dealing with one strange human is very hard for a computer, but dealing with two or more of them is very similar. Snow / ice / rain on the roads means changing underlying models, behaving more cautiously. Computers can switch modalities much quicker/easier than humans. Fog and other visibility problems are sensor issues. Humans are stuck with the old Mark 1 eyeball, but self driving cars have a mix of sensors. These sensors may currently be worse than eyeballs in fog, but may not be in the future. Even given those advantages, you still start in an easy location. Going from 0 to 1 is really hard, give yourself all the advantages you can. But going from 1 to N is a lot easier.
IshKebab 6y ago
But is it 2.08 crashes/million miles of driving around extremely easy and empty suburbs? There's a reason they started where they have - it's the easiest driving in the world. That's very sensible but it does mean you can't just compare statistics like that.
mchusma 6y ago
I thought your Your 2.08 crashes/million miles number had to be wrong, since if that were true the average person would have about 1 accident every 50 years or so (which doesn't match my experience of everyone I know having accidents). Your number was "police reported accidents" which was apparently 6.7M. Anecdotally, I have been involved in 10 accidents or so as a passenger or driver, and none were reported to the police. So I can't find good numbers on what the real number is but I think its conservative to estimate it at 1/2 are reported to the police. In addition, "Crashes" mostly involve 2 cars, so the "police reported cars that crashed" is roughly 2x, or >12M. Basically, those changes makes it roughly 8 crashes per million miles for normal driving, or Waymo being basically the same. I still think that accident rate is too low though. This estimate is 6 crashes every million miles based on allstate data https://mashable.com/2012/08/07/google-driverless-cars-safer-than-you/ https://mashable.com/2012/08/07/google-driverless-cars-safer.... If that was the case, then humans would be at about 12 vehicle crashes/million miles, almost 2x as dangerous as Waymo. So about 2x better than human performance?
- maliker 6y ago
  Great analysis and new data!
  - mchusma 6y ago
    Thank you for getting it started. I'm not 100% sure I understand it all. It's obviously important to figure out if it is safer or riskier.
- YeGoblynQueenne 6y ago
  >> So about 2x better than human performance? If you really want the comparison to be on an even footing, you should consider only the crashes that happen during miles driven by humans in Phoenix. Not all over the US. In Phoenix only- because Waymo would have many more crashes outside Phoenix (e.g. New York or LA, from what I'm told of the traffic conditions there) and because so would humans.
  - rootusrootus 6y ago
ehsankia 6y ago
Maybe a stupid question, I don't know your exact calculations, but wouldn't you be half-counting the number of collisions, since a collision is almost always between two cars, so you're counting the miles driven by both cars, but only counting the collision between them as 1. So you'd be off by a factor of 2 if you compare that to a distance driver by a single car divided by its collisions.
- maliker 6y ago
  Yes, thank you, definitely correct. So it's more like 4 accident/million miles from this dataset, although another poster made a good case that Waymo's data includes minor accidents that aren't in the data I looked at, which would make 12 accidents/million miles a better benchmark.
SomewhatLikely 6y ago
They are also counting incidents where the relative speed was 1mph, or a person walked into the car. Perhaps fatalities per billion miles would be a more comparable number.
scoopertrooper 6y ago
* Seems like they're far from human performance. * On highways while under the supervision of a trained driver and autonomously within a select area.

brixon 6y ago

Did you count all incidents or only the ones that happened in real life and not the simulations?

maliker 6y ago
I omitted the simulations in the Waymo count. Good idea to add them in?
- IshKebab 6y ago
  Yeah you should include them because they were only prevented by a safety driver who won't exist in the long term.
  - maliker 6y ago
    Thanks! I missed that definition in the article. Updated the ratios. So it's actually 3.7x more accidents not 1.5x. Definitely changed my conclusion.
danielvf 6y ago
Simulations counts were times when a safety driver took control of the vechile, and post-incident simulations with the same data show that the AI would have crashed the car without the safety driver. So fair to include them.

beambot 6y ago

You also need to account for confounding variables, such as geography. Does Phoenix mirror the nation as a whole?

bufferoverflow 6y ago

Phonenix doesn't get much snow / ice / crazy fog / crazy rain. The roads are wide. The amount of traffic is low compared to, say, NYC or LA. So yeah, they are not near human level yet.

maxerickson 6y ago
I think there's a solid chance that a 'near human capable' driving system will handle slippery roads better than the human average. The system will be in a modern vehicle with traction control and likely make conservative judgments about road condition.
- adrianmonk 6y ago
  There might also be differences in how accurately a human or machine can model which maneuvers can/can't be done without losing traction. I could see this going either way. Maybe a human will be able to identify spots (like puddles) where traction will be especially bad, but a machine can do a detailed physics simulation of a maneuver before trying it. In whatever way a machine does adjust its behavior for slippery roads, it will probably be more consistent at it than a human. Humans have to deal with force of habit, so for example they may momentarily forget that they can't brake hard or they shouldn't take that turn at a familiar intersection at the same speed as yesterday.
- notatoad 6y ago
  also poor visibility. humans need to rely on their eyes, a car with a bunch of lidar sensors is absolutely going to outperform us in fog, blowing snow, or heavy rain.
bryanlarsen 6y ago
I suspect that Waymo will do better compared to a human outside of Phoenix than inside it. The hardest thing that Waymo will have to do is to interact with humans behaving oddly. Phoenix has those, just not as many as NYC/LA/SF. But one thing computers do exceedingly well is scale. Dealing with one strange human is very hard for a computer, but dealing with two or more of them is very similar. Snow / ice / rain on the roads means changing underlying models, behaving more cautiously. Computers can switch modalities much quicker/easier than humans. Fog and other visibility problems are sensor issues. Humans are stuck with the old Mark 1 eyeball, but self driving cars have a mix of sensors. These sensors may currently be worse than eyeballs in fog, but may not be in the future. Even given those advantages, you still start in an easy location. Going from 0 to 1 is really hard, give yourself all the advantages you can. But going from 1 to N is a lot easier.

IshKebab 6y ago

But is it 2.08 crashes/million miles of driving around extremely easy and empty suburbs? There's a reason they started where they have - it's the easiest driving in the world. That's very sensible but it does mean you can't just compare statistics like that.

mchusma 6y ago

I thought your Your 2.08 crashes/million miles number had to be wrong, since if that were true the average person would have about 1 accident every 50 years or so (which doesn't match my experience of everyone I know having accidents). Your number was "police reported accidents" which was apparently 6.7M. Anecdotally, I have been involved in 10 accidents or so as a passenger or driver, and none were reported to the police. So I can't find good numbers on what the real number is but I think its conservative to estimate it at 1/2 are reported to the police. In addition, "Crashes" mostly involve 2 cars, so the "police reported cars that crashed" is roughly 2x, or >12M. Basically, those changes makes it roughly 8 crashes per million miles for normal driving, or Waymo being basically the same. I still think that accident rate is too low though. This estimate is 6 crashes every million miles based on allstate data https://mashable.com/2012/08/07/google-driverless-cars-safer-than-you/ https://mashable.com/2012/08/07/google-driverless-cars-safer.... If that was the case, then humans would be at about 12 vehicle crashes/million miles, almost 2x as dangerous as Waymo. So about 2x better than human performance?

maliker 6y ago
Great analysis and new data!
- mchusma 6y ago
  Thank you for getting it started. I'm not 100% sure I understand it all. It's obviously important to figure out if it is safer or riskier.
YeGoblynQueenne 6y ago
>> So about 2x better than human performance? If you really want the comparison to be on an even footing, you should consider only the crashes that happen during miles driven by humans in Phoenix. Not all over the US. In Phoenix only- because Waymo would have many more crashes outside Phoenix (e.g. New York or LA, from what I'm told of the traffic conditions there) and because so would humans.
- rootusrootus 6y ago

ehsankia 6y ago

Maybe a stupid question, I don't know your exact calculations, but wouldn't you be half-counting the number of collisions, since a collision is almost always between two cars, so you're counting the miles driven by both cars, but only counting the collision between them as 1. So you'd be off by a factor of 2 if you compare that to a distance driver by a single car divided by its collisions.

maliker 6y ago
Yes, thank you, definitely correct. So it's more like 4 accident/million miles from this dataset, although another poster made a good case that Waymo's data includes minor accidents that aren't in the data I looked at, which would make 12 accidents/million miles a better benchmark.

SomewhatLikely 6y ago

They are also counting incidents where the relative speed was 1mph, or a person walked into the car. Perhaps fatalities per billion miles would be a more comparable number.

scoopertrooper 6y ago

* Seems like they're far from human performance. * On highways while under the supervision of a trained driver and autonomously within a select area.