6 ms·

It seems like they compared apples to apples. Which make that an exciting result! “The benchmark was calibrated using both mileage (driving exposure) and resid

by zurfer 2y ago

It seems like they compared apples to apples. Which make that an exciting result! “The benchmark was calibrated using both mileage (driving exposure) and residence zip-code (geographic region). Specifically, only the claims associated with vehicles registered to addresses (i.e., where the insured resides) within Waymo’s operating zip codes in San Francisco, the Phoenix metropolitan region, Los Angeles, and Austin were included.” Page 11 “Waymo also almost exclusively operated on surface streets (non-access-controlled freeways) with a unique distribution of driving that is representative of a ride-hailing fleet. In contrast, the benchmark represents the privately insured driver population that resides in these geographic regions. The associated benchmark mileage has more freeway driving than the Waymo ADS.” Page 26 “Including freeway driving makes this benchmark crash rate artificially lower, so, by including freeways in this study’s benchmark, the benchmark crash rate underestimates the true driving risk of where the Waymo ADS operates.” Page 26 “Four claim frequencies were independently calculated for vehicles within Waymo’s operating zip codes in San Francisco, Phoenix, Los Angeles, and Austin. Thereafter, these frequencies were proportionally weighted according to Waymo’s mileage distribution across these four regions. This resulted in the definition of a benchmark driving population which had driven the same driving distribution across geographic regions as the Waymo Driver.” Page 12 “Waymo also almost exclusively operated on surface streets with a unique distribution of driving that is representative of a ride-hailing fleet… Due to all three of these limitations being expected to artificially suppress the benchmark crash rate (underestimation), the benchmarking results in this study are considered to be conservative.” Page 26

amelius 2y ago
> It seems like they compared apples to apples. Which make that an exciting result! I'd want to compare to our best drivers (e.g. ambulance drivers), not average drivers.
- mulmen 2y ago
  > I'd want to compare to our best drivers (e.g. ambulance drivers), not average drivers. Why? As long as it is better than the median driver the roads are safer.
  - notahacker 2y ago
    > Why? As long as it is better than the median driver the roads are safer. The intent is to increase miles driven, and the drivers a commercial service can be expected to replace first are professionals, not "median drivers". In practice accident stats are also benchmarked against mean drivers in a world in which serious accidents disproportionately involve inadequate drivers (or conditions Waymo does not deploy in)
    - mulmen 2y ago
      Then shouldn’t the comparison be to Uber instead of an ambulance?
      - notahacker 2y ago
        Ambulance drivers might be a step too far unless autonomous systems are expected to drive ambulances at high speed and run red lights. But my point is that they really do need to be a lot better than the median driver given their real world operational expectations, and especially significantly better than the mean driver they're being benchmarked against here, who probably loses their license for the bodily injuries they cause...
        deleted 2y ago
        [deleted]
        mulmen 2y ago
        Well Uber drivers don’t have any special qualifications. That’s kinda the point of Uber. Is there any reason to expect them to be safer than the average driver?
        notahacker 2y ago
        Not being super nervous about being behind the wheel, not showing off their speed to their friends and probably being sober is a starting point
- jpgvm 2y ago
  Why? We aren't looking at introducing autonomous ambulances any time soon. They should be compared to taxi/Uber/Lyft drivers that is what is being replaced by this and as such where the net harm increase or reduction will occur.
  - amelius 2y ago
    Why -> because if someone caused an accident I'd want to look them in the eyes and call them names; as opposed to calling a support number. If these vehicles were among the best drivers, at least I could live with it maybe. Emotions are real, you cannot always rationalize them away. Average drivers sometimes end up in jail for their mistakes. If Waymo made those same mistakes, guess what will happen.
    - jpgvm 2y ago
      I'm not sure it's possible to solve for that with additional prowess. If you are wronged by a machine you are always going to be emotional about it, i.e having your email account unceremoniously terminated by Google without recourse by an essentially automated process. The thing to solve for is the lack of recourse. Very very difficult in American society given how skewed the relationship is between corporations and people (or the government for that matter). If instead when a Waymo demonstratively caused an accident the payout was 10-100x higher than if a human was involved I suspect many peoples qualms would go away. While I think just the "knowledge" (assuming Americans would accept facts...) that the driver is X% better than the best drivers on the roads would provide little comfort to most. I think in societies outside of America these may be more successful actually. i.e most parts of Asia.
- ben_w 2y ago
  Both measures are valid. When the AI is better than the best humans, that's when the best humans finally change career. There's nothing that requires delaying until that point, the time in which courts order drunk drivers to only be allowed to own self-driving models.
- tempestn 2y ago
  You just need to know the injury claims of those drivers vs the average to compare. But I agree, if I were to trust such a system, I'd want it to be a better driver than me, not than average. (And like everyone, I believe I'm significantly above average...)
bertil 2y ago
> privately insured driver population that resides in these geographic regions If I were to hide something fishy, I’d define the shared coverage to include zip codes with fewer human accidents: * zip codes are relatively small, so you have a lot of variance per code due to randomness of rare events; * Waymo mostly covers urban areas and nearby suburban areas, so you can claim suburban zones were only recently covered by Waymo or that the insurance had too few members in certain zip codes to include them. I don’t think that study is wrong, but it is in Waymo’s interest to claim they have few accidents and in insurers’ interest to claim there have many, or rather more, accidents to raise their premiums.
aithrowawaycomm 2y ago
I don't think this is apples-to-apples. You need to somehow filter to people who are not speeding or otherwise breaking traffic laws. My problem with studies like these is "human-operated" is an enormous catch-all that blends together too many causes for accidents. 1) Not filtering out drunk drivers is downright misleading. I am less interested in "is mandatory self-driving safer than human drivers?" than "is mandatory self-driving safer than mandatory breathalyzer ignition?" There are some uniquely human downsides which are fair comparisons to AI - distractions, slowness, anger -but drunk drivers should be removed from this. 2) The bigger problem is people rationally speeding or running red lights. Waymo is strictly legal, but Tesla self-driving historically encouraged speeding, rolling stop signs, etc. Ubers and Lyft are preferred if you're in a hurry precisely because human drivers bend the law. Human speeding is not caused by a lack of intelligence and needing an AI to figure it out that it's dangerous: humans do it because they are selfish and reckless, and someone like Tesla will make a driving AI that fulfills their selfish demands. There is absolutely no reason to think that a speeding Waymo would be safer than a speeding human. Studies like this makes me worry about a medium-term where driving is more dangerous for everyone: lawful drivers get in more accidents because they're using sub-human AI, whereas nothing changes for unlawful drivers because they are using dangerous AI, turning off the AI and driving drunk, etc.
- viraptor 2y ago
  It depends what you want to know. "Are autonomous cars better than good drivers?", or "am I safer in Waymo than a random human driven taxi?". Both are very valid questions.
  - aithrowawaycomm 2y ago
    But "random human-driven taxi" is arbitrary since random humans are not driving taxis. This study doesn't at all show that Waymos are safer than Ubers driven by Uber drivers (though I suspect that is the case).
- nmca 2y ago
  You’re trying to to characterise personal risk and they were tying to characterise societal risk; both are valid.
YeGoblynQueenne 2y ago
>> “Including freeway driving makes this benchmark crash rate artificially lower, so, by including freeways in this study’s benchmark, the benchmark crash rate underestimates the true driving risk of where the Waymo ADS operates.” Page 26 Except of course if Waymo would crash a lot more in freeways, which we can't assume won't happen just because human drivers don't. Far from this making human drivers look better it makes the comparison unequal and for that reason, uninformative. Half of those apples are orange and smell like citrus fruits.

by zurfer 2y ago

amelius 2y ago

> It seems like they compared apples to apples. Which make that an exciting result! I'd want to compare to our best drivers (e.g. ambulance drivers), not average drivers.

mulmen 2y ago
> I'd want to compare to our best drivers (e.g. ambulance drivers), not average drivers. Why? As long as it is better than the median driver the roads are safer.
- notahacker 2y ago
  > Why? As long as it is better than the median driver the roads are safer. The intent is to increase miles driven, and the drivers a commercial service can be expected to replace first are professionals, not "median drivers". In practice accident stats are also benchmarked against mean drivers in a world in which serious accidents disproportionately involve inadequate drivers (or conditions Waymo does not deploy in)
  - mulmen 2y ago
    Then shouldn’t the comparison be to Uber instead of an ambulance?
    - notahacker 2y ago
      Ambulance drivers might be a step too far unless autonomous systems are expected to drive ambulances at high speed and run red lights. But my point is that they really do need to be a lot better than the median driver given their real world operational expectations, and especially significantly better than the mean driver they're being benchmarked against here, who probably loses their license for the bodily injuries they cause...
      - deleted 2y ago
        [deleted]
      - mulmen 2y ago
        Well Uber drivers don’t have any special qualifications. That’s kinda the point of Uber. Is there any reason to expect them to be safer than the average driver?
        notahacker 2y ago
        Not being super nervous about being behind the wheel, not showing off their speed to their friends and probably being sober is a starting point
jpgvm 2y ago
Why? We aren't looking at introducing autonomous ambulances any time soon. They should be compared to taxi/Uber/Lyft drivers that is what is being replaced by this and as such where the net harm increase or reduction will occur.
- amelius 2y ago
  Why -> because if someone caused an accident I'd want to look them in the eyes and call them names; as opposed to calling a support number. If these vehicles were among the best drivers, at least I could live with it maybe. Emotions are real, you cannot always rationalize them away. Average drivers sometimes end up in jail for their mistakes. If Waymo made those same mistakes, guess what will happen.
  - jpgvm 2y ago
    I'm not sure it's possible to solve for that with additional prowess. If you are wronged by a machine you are always going to be emotional about it, i.e having your email account unceremoniously terminated by Google without recourse by an essentially automated process. The thing to solve for is the lack of recourse. Very very difficult in American society given how skewed the relationship is between corporations and people (or the government for that matter). If instead when a Waymo demonstratively caused an accident the payout was 10-100x higher than if a human was involved I suspect many peoples qualms would go away. While I think just the "knowledge" (assuming Americans would accept facts...) that the driver is X% better than the best drivers on the roads would provide little comfort to most. I think in societies outside of America these may be more successful actually. i.e most parts of Asia.
ben_w 2y ago
Both measures are valid. When the AI is better than the best humans, that's when the best humans finally change career. There's nothing that requires delaying until that point, the time in which courts order drunk drivers to only be allowed to own self-driving models.
tempestn 2y ago
You just need to know the injury claims of those drivers vs the average to compare. But I agree, if I were to trust such a system, I'd want it to be a better driver than me, not than average. (And like everyone, I believe I'm significantly above average...)

bertil 2y ago

> privately insured driver population that resides in these geographic regions If I were to hide something fishy, I’d define the shared coverage to include zip codes with fewer human accidents: * zip codes are relatively small, so you have a lot of variance per code due to randomness of rare events; * Waymo mostly covers urban areas and nearby suburban areas, so you can claim suburban zones were only recently covered by Waymo or that the insurance had too few members in certain zip codes to include them. I don’t think that study is wrong, but it is in Waymo’s interest to claim they have few accidents and in insurers’ interest to claim there have many, or rather more, accidents to raise their premiums.

aithrowawaycomm 2y ago

I don't think this is apples-to-apples. You need to somehow filter to people who are not speeding or otherwise breaking traffic laws. My problem with studies like these is "human-operated" is an enormous catch-all that blends together too many causes for accidents. 1) Not filtering out drunk drivers is downright misleading. I am less interested in "is mandatory self-driving safer than human drivers?" than "is mandatory self-driving safer than mandatory breathalyzer ignition?" There are some uniquely human downsides which are fair comparisons to AI - distractions, slowness, anger -but drunk drivers should be removed from this. 2) The bigger problem is people rationally speeding or running red lights. Waymo is strictly legal, but Tesla self-driving historically encouraged speeding, rolling stop signs, etc. Ubers and Lyft are preferred if you're in a hurry precisely because human drivers bend the law. Human speeding is not caused by a lack of intelligence and needing an AI to figure it out that it's dangerous: humans do it because they are selfish and reckless, and someone like Tesla will make a driving AI that fulfills their selfish demands. There is absolutely no reason to think that a speeding Waymo would be safer than a speeding human. Studies like this makes me worry about a medium-term where driving is more dangerous for everyone: lawful drivers get in more accidents because they're using sub-human AI, whereas nothing changes for unlawful drivers because they are using dangerous AI, turning off the AI and driving drunk, etc.

viraptor 2y ago
It depends what you want to know. "Are autonomous cars better than good drivers?", or "am I safer in Waymo than a random human driven taxi?". Both are very valid questions.
- aithrowawaycomm 2y ago
  But "random human-driven taxi" is arbitrary since random humans are not driving taxis. This study doesn't at all show that Waymos are safer than Ubers driven by Uber drivers (though I suspect that is the case).
nmca 2y ago
You’re trying to to characterise personal risk and they were tying to characterise societal risk; both are valid.

YeGoblynQueenne 2y ago

>> “Including freeway driving makes this benchmark crash rate artificially lower, so, by including freeways in this study’s benchmark, the benchmark crash rate underestimates the true driving risk of where the Waymo ADS operates.” Page 26 Except of course if Waymo would crash a lot more in freeways, which we can't assume won't happen just because human drivers don't. Far from this making human drivers look better it makes the comparison unequal and for that reason, uninformative. Half of those apples are orange and smell like citrus fruits.