4 ms·

The US drives about 3.2 trillion miles per year. Waymo has 56.7 million miles over several years. Their percentage data is essentially useless. 50% of fatali

The US drives about 3.2 trillion miles per year. Waymo has 56.7 million miles over several years. Their percentage data is essentially useless. 50% of fatalities involve alcohol or drugs and are often single vehicle accidents. 25% involve youth or inexperience. 15% involve motorcycles. 15% involve pedestrians. What I really need to see is a complete breakdown of every accident a Waymo has had. Then I can start to compare their actual performance to the previously known outcomes.

p_j_w 1y ago
> The US drives about 3.2 trillion miles per year. Waymo has 56.7 million miles over several years. Their percentage data is essentially useless. Your third sentence doesn’t follow from your first two. On what grounds do you draw this conclusion?
prasadjoglekar 1y ago
This is the statistic I was hoping to see. It wasn't in the paper as best as I looked.
shermantanktop 1y ago
How would you compare single-incident reports to get to a meaningful conclusion?My guess is that Waymo makes mistakes that a human wouldn't, and vice versa. At that point the overall safety record, which normalizes those differences, seems the most relevant.
crazygringo 1y ago
> The US drives about 3.2 trillion miles per year. Waymo has 56.7 million miles over several years. Their percentage data is essentially useless. No, that's not how statistics works. The percentage data's accuracy depends mainly on the number of incidents recorded (and somewhat on the rate of incidents). But the percentage of the whole is completely irrelevant. If you are basing something on 10 incidents but it's 50% of the total, it's still terrible accuracy. Whereas if you are basing something on 100,000 incidents but it's only 0.1% of the total, it's still going to be quite accurate, assuming the incidents come from the same overall distribution.
- timewizard 1y ago
  > If you are basing something on 10 incidents but it's 50% of the total, it's still terrible accuracy. The ratio of 3.2 trillion to 56.7 million, which is already incredibly generous to Waymo's position, is 5 orders of magnitude in difference. So any calculations from Waymos data are going to be insanely inaccurate and not something you can extrapolate from. The main, and most obvious case, evidenced by this, is Waymo does not operate where snow falls. Human beings do. We're missing so much of the picture I don't think you can say Waymo's are 75% less accident prone, or 80% less likely to hit a pedestrian. Those are just nonsense numbers.
  - achatham 1y ago
    The paper under discussion only considers human accidents in similar environments to where Waymo operates. So it's only making a claim about like-for-like driving. You could still say you care about snow driving and want to see that comparison, but it doesn't mean the claims in this paper are wrong.
- DAGdug 1y ago
  This! (Thank you for the comment). There’s a reason a 1000 random samples is adequate to reasonably estimate what’s common metrics in a population the size of USA or India (or infinitely large).
  - timewizard 1y ago
    Random samples of the _same_ user base. If the user base of "waymo riders" and "everyday drivers" does not match then you're not sampling what you think you are.
    - DAGdug 1y ago
      Yeah, that’s fair, but is implicit since I’m arguing against the “sample size is inadequate” POV, not the “there are distributional biases in data” POV. There are a gazillion ways to adjust for these biases (ex. propensity score matching) going beyond just user-base but also including weather type, road type, location, time of day, day of week, traffic density, pedestrian density … that can be done easily with far less than the sample size waymo has. And I bet they do these adjustments.
D-Coder 1y ago
> 50% of fatalities involve alcohol or drugs and are often single vehicle accidents. This suggests that Waymo is cutting traffic fatalities by 50% (per million miles) right off the top.
- timewizard 1y ago
  If an only if every drunk person decides to take one instead of driving themselves home. Drunk people being known for having exceptionally poor judgement and self awareness. It suggests that they _could_ cut fatalities by that much. Then again, a whole new mode of accident, where the inebriated decide to step out of a moving vehicle and injure themselves that way. This is a dynamic system where human decisions are never fully removed from the loop.
  - standardUser 1y ago
    > Then again, a whole new mode of accident, where the inebriated decide to step out of a moving vehicle and injure themselves that way. If I understand correctly, you believe that the advent of self-driving cars will cause passengers to voluntarily exit a moving vehicle? That sounds like absolute nonsense with no basis in reality.
  - Mawr 1y ago
    > Then again, a whole new mode of accident, where the inebriated decide to step out of a moving vehicle and injure themselves that way. Why would they need to be in a self-driving vehicle to do that?
SpicyLemonZest 1y ago
You'll be happy to know, then, that the NHTSA publishes the data you're looking for at https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting https://www.nhtsa.gov/laws-regulations/standing-general-orde..., in addition to Waymo's own reporting in their safety hub. For me personally, I find summary data to be more informative than a pile of individual reports, but I hope your comparison goes well!
bluGill 1y ago
I do want to know how Waymo compares to middle ages adults who are not on alcohol/drugs. However that they are better than humans overall is still a big deal even if the data is somewhat suspect by not doing that additional breakdown.
- boulos 1y ago
  Our safety research team is interested in this topic, too! In a previous study, they've tried to model it: > Building on that, the Collision Avoidance Benchmarking paper presents a novel methodology to evaluate how well autonomous driving systems avoid crashes. The study, which to our knowledge is the first of its kind, introduces a reference model that represents an ideal human state for driving—the response time and evasive action of a human driver that is non-impaired, with eyes always on the conflict (NIEON). Put simply, unlike an average human driver, NIEON is always attentive and doesn’t get distracted or fatigued¹. The data showed that the Waymo Driver outperformed the NIEON human driver model by avoiding more collisions and mitigating serious injury risk in simulated fatal crash scenarios. (From https://waymo.com/blog/2022/09/benchmarking-av-safety https://waymo.com/blog/2022/09/benchmarking-av-safety) AIUI (I'm not on that team), a major challenge is getting good baseline data. Collision reports may not (reliably) capture that kind of data, and it's clearly subjective or often self-reported outside of cases like DUI charges.

p_j_w 1y ago

> The US drives about 3.2 trillion miles per year. Waymo has 56.7 million miles over several years. Their percentage data is essentially useless. Your third sentence doesn’t follow from your first two. On what grounds do you draw this conclusion?

prasadjoglekar 1y ago

This is the statistic I was hoping to see. It wasn't in the paper as best as I looked.

shermantanktop 1y ago

How would you compare single-incident reports to get to a meaningful conclusion?My guess is that Waymo makes mistakes that a human wouldn't, and vice versa. At that point the overall safety record, which normalizes those differences, seems the most relevant.

crazygringo 1y ago

> The US drives about 3.2 trillion miles per year. Waymo has 56.7 million miles over several years. Their percentage data is essentially useless. No, that's not how statistics works. The percentage data's accuracy depends mainly on the number of incidents recorded (and somewhat on the rate of incidents). But the percentage of the whole is completely irrelevant. If you are basing something on 10 incidents but it's 50% of the total, it's still terrible accuracy. Whereas if you are basing something on 100,000 incidents but it's only 0.1% of the total, it's still going to be quite accurate, assuming the incidents come from the same overall distribution.

timewizard 1y ago
> If you are basing something on 10 incidents but it's 50% of the total, it's still terrible accuracy. The ratio of 3.2 trillion to 56.7 million, which is already incredibly generous to Waymo's position, is 5 orders of magnitude in difference. So any calculations from Waymos data are going to be insanely inaccurate and not something you can extrapolate from. The main, and most obvious case, evidenced by this, is Waymo does not operate where snow falls. Human beings do. We're missing so much of the picture I don't think you can say Waymo's are 75% less accident prone, or 80% less likely to hit a pedestrian. Those are just nonsense numbers.
- achatham 1y ago
  The paper under discussion only considers human accidents in similar environments to where Waymo operates. So it's only making a claim about like-for-like driving. You could still say you care about snow driving and want to see that comparison, but it doesn't mean the claims in this paper are wrong.
DAGdug 1y ago
This! (Thank you for the comment). There’s a reason a 1000 random samples is adequate to reasonably estimate what’s common metrics in a population the size of USA or India (or infinitely large).
- timewizard 1y ago
  Random samples of the _same_ user base. If the user base of "waymo riders" and "everyday drivers" does not match then you're not sampling what you think you are.
  - DAGdug 1y ago
    Yeah, that’s fair, but is implicit since I’m arguing against the “sample size is inadequate” POV, not the “there are distributional biases in data” POV. There are a gazillion ways to adjust for these biases (ex. propensity score matching) going beyond just user-base but also including weather type, road type, location, time of day, day of week, traffic density, pedestrian density … that can be done easily with far less than the sample size waymo has. And I bet they do these adjustments.

D-Coder 1y ago

> 50% of fatalities involve alcohol or drugs and are often single vehicle accidents. This suggests that Waymo is cutting traffic fatalities by 50% (per million miles) right off the top.

timewizard 1y ago
If an only if every drunk person decides to take one instead of driving themselves home. Drunk people being known for having exceptionally poor judgement and self awareness. It suggests that they _could_ cut fatalities by that much. Then again, a whole new mode of accident, where the inebriated decide to step out of a moving vehicle and injure themselves that way. This is a dynamic system where human decisions are never fully removed from the loop.
- standardUser 1y ago
  > Then again, a whole new mode of accident, where the inebriated decide to step out of a moving vehicle and injure themselves that way. If I understand correctly, you believe that the advent of self-driving cars will cause passengers to voluntarily exit a moving vehicle? That sounds like absolute nonsense with no basis in reality.
- Mawr 1y ago
  > Then again, a whole new mode of accident, where the inebriated decide to step out of a moving vehicle and injure themselves that way. Why would they need to be in a self-driving vehicle to do that?

SpicyLemonZest 1y ago

You'll be happy to know, then, that the NHTSA publishes the data you're looking for at https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting https://www.nhtsa.gov/laws-regulations/standing-general-orde..., in addition to Waymo's own reporting in their safety hub. For me personally, I find summary data to be more informative than a pile of individual reports, but I hope your comparison goes well!

bluGill 1y ago

I do want to know how Waymo compares to middle ages adults who are not on alcohol/drugs. However that they are better than humans overall is still a big deal even if the data is somewhat suspect by not doing that additional breakdown.

boulos 1y ago
Our safety research team is interested in this topic, too! In a previous study, they've tried to model it: > Building on that, the Collision Avoidance Benchmarking paper presents a novel methodology to evaluate how well autonomous driving systems avoid crashes. The study, which to our knowledge is the first of its kind, introduces a reference model that represents an ideal human state for driving—the response time and evasive action of a human driver that is non-impaired, with eyes always on the conflict (NIEON). Put simply, unlike an average human driver, NIEON is always attentive and doesn’t get distracted or fatigued¹. The data showed that the Waymo Driver outperformed the NIEON human driver model by avoiding more collisions and mitigating serious injury risk in simulated fatal crash scenarios. (From https://waymo.com/blog/2022/09/benchmarking-av-safety https://waymo.com/blog/2022/09/benchmarking-av-safety) AIUI (I'm not on that team), a major challenge is getting good baseline data. Collision reports may not (reliably) capture that kind of data, and it's clearly subjective or often self-reported outside of cases like DUI charges.