4 ms·

This sounds quite a bit like massaging statistics until you get the numbers you like. A few things that pop to mind: did we exclude drivers who were not eligib

This sounds quite a bit like massaging statistics until you get the numbers you like. A few things that pop to mind: did we exclude drivers who were not eligible to drive (under the influence, no valid driving permit, cars with issues [braking broken]...)? (This mostly matters for evaluating the average performance compared to average non-impaired human — a drunk person might mess up with self-driving vehicle sensors as well just for kicks once they are in suitable numbers and widely available) Did Waymo drive across all the same areas and time periods as the humans did with an equal "proportion"? Could we compare "time behind wheel" instead of miles travelled? (Eg. lots of small crashes happen when traffic is at a standstill) There are so many ways to slice and dice this and come up with favourable results for self-driving systems that I can't but be dubious of any claim. But notably, if we can liberate ourselves from driving through congested traffic, it would certainly reduce the incentive to speed and aim for more efficient driving as we'd be free to do other stuff while we get to our destination — and simply slowing down would buy both human and software drivers time to react and avoid most crashes (nothing really saves you from someone recklessly driving straight into your car).

Reubend 2y ago
> did we exclude drivers who were not eligible to drive One could argue here that it would be unfair to exclude the worst performing humans. Plenty of humans cause accidents for easily preventable reasons, but that's an honest part of our behavior. Or to rephrase, I think excluding the categories of people who drive terribly, but do still drive nonetheless, would be less representative of reality. > Did Waymo drive across all the same areas and time periods as the humans did with an equal "proportion"? No, they didn't. Their methodology didn't adjust for things like highway vs city street miles, although it did adjust for city and state. > Could we compare "time behind wheel" instead of miles travelled? We could, but that would incentivize slow driving rather than safe driving. You need to travel the same minimum distance from a hypothetical "Point A" to "Point B" no matter what, but the number of minutes could be inflated easily. Economic factors discourage inflating the number of miles. > There are so many ways to slice and dice this and come up with favourable results for self-driving systems that I can't but be dubious of any claim. I think it's fair to criticize the fact that they didn't adjust for geographic area with as much granularity as they should have (i.e. for highways vs city streets, or for excluded zones like airports). But other than that, their methodology seems solid to me.
- necovek 2y ago
  I understand that viewpoint from an insurance perspective, but not necessarily from my own safety perspective: I wonder how well Waymo would fare in circumstances I see as a driver. Eg. humans can control whether they drive under the influence or not (eg. if you don't take your car to a bar, you ain't coming back driving it). But most notably, aggregated statistics like these focus on comparisons similar to comparing averages vs better statistical representations (like average vs median or p75 or... eg. it's possible for one driver to cause 20 collisions, and in a set of 10 drivers with no other collisions, we'd be at 2 collisions per driver instead of 0 really). It's also not only about highway/city street miles, it's also about time of day: if Waymo is proportionally less on the streets and areas where there is more risky driving (eg. around bars and around midnight to 3am when there are also drunk pedestrians around) compared to human drivers, that would obviously skew the numbers in SDV's favour. Again, potentially they did since they acknowledge a separate study focusing on those, but later on they only talk about aggregated insurance claims. They also acknowledge they are not accounting for accident severity, not even by using the dollar amount: while they had no fatalities, it's obviously important to weigh accidents by their severity. With all this said, this does demonstrate current insurance liability of an average human driver compared to a Waymo SDV in Phoenix and SFO, but only once we have more comparable data should we make a bigger claim.
  - hnaccount_rng 2y ago
    I get where you are coming from and you are correct. This is, up to a point, a PR piece of Waymo and we should take that into account. But they are pretty transparent about what they are doing. You just have to read the report. And for their metric, which rounds to "all distance travelled in the interesting ZIP codes", they are succeeding. What I'd take away from this is: If you give them a different metric, they can optimise for that too and they will end up being better than humans. That doesn't answer the questions: 1) Is it worth it? (I.e. what does it cost to achieve that) and 2) Is a Waymo better in any situation? I guess they don't have the answer to either yet. I'm pretty sure their cost-per-mile is still dropping like a rock (probably though not fast enough for their accountants) and they haven't reached all situations yet either. But I wouldn't bet on "because it's too hard", but rather "because we didn't get there yet" (and yes, they will have started with the easy part, as they should)
myrmidon 2y ago
Regarding drivers that are drunk or otherwise driving illegally-- why would you want to exclude them from such a study, if you can not exclude them from real traffic? If our current system of checks and incentives is not able to keep them off the road, then you need to take them and the accidents they cause into account when comparing driving safety with an autonomous system. Also note that the margins are too big, since only about a third of accidents involve intoxicated drivers, and the accident reduction rates from the paper are much higher than that. I don't want to come across as harsh or accusing, but the "Methodology" section contains a lot of detail (and a more nuanced discussion could be had without disregarding it entirely :P).
- necovek 2y ago
  Because they don't disappear even with half the non-illegal vehicles being autonomous — some crashes also happen due to avoidance manoeuvres. So the fairer comparison would be to say that if existing SDVs were instead being driven by human drivers, we would have seen ~60 property claims more and ~40 bodily injury claims more (forgot the exact numbers from the paper), which is what, like 0.1% improvement in claims overall (certainly still a great result for the small comparative number of miles travelled, but doesn't jump off the page really). But because the miles traveled are so far apart, nothing is really a "fair" comparison.
  - hnaccount_rng 2y ago
    But.. they would disappear? Not all of course. But some drive under the influence because there is no other convenient way to get home after a night out. And Waymo solves that! And they are highly aware of their "small number statistics", they literally provide uncertainty ranges. For both SDVs and HDVs. And their values are consistent with increasing data. I.e. they ~only get smaller uncertainties but no significant drift. But they are statistically significantly better (which wasn't the case two years ago)
    - necovek 2y ago
      Are you suggesting Waymo would become available where cabs aren't due to economics, or that Waymo-like technology in privately owned vehicles would solve that? Because I wouldn't buy the former, but I can see how the latter would, yet we are far away from that. I am not referring only to "small number statistics", but to obvious miscomparisons. We don't want SDVs to beat "average human driver", because average is worsened by impaired humans — we need them to "beat" at least an average, non-impaired human. Going by the numbers alone, they would achieve that too, but not by the same amount. But more importantly, we need them to compare both in identical driving conditions, which is where most of my complaint really is. Unless Waymo has driven in the same amount (proportionally) of rush-hour traffic, road-works, risky-drivers-around conditions, parking into tight spots and garages... of miles out of their 25M miles as their human counterparts, we could very well be looking into Waymo-in-simpler-situations vs humans-in-all-situations. And the numbers would be meaningless, which to me they seem they are. And it is important that they have achieved similar success with 6x the miles, but they need to account for more challenging conditions. And while some tricks can work while they are in small numbers (eg. avoiding road works by re-routing), it will lead to congestion elsewhere when such technology dominates the traffic. I would also note that it is impressive, regardless of the conditions they operated in, that they were involved in exactly 2 "bodily injury" claims over those 25M miles, and both of those appear not to have been the fault of Waymo.
jbarrow 2y ago
> did we exclude drivers who were not eligible to drive I don't think we can discount people who aren't fit/eligible to drive _if they still drove_. They might not have a _legal_ right to be on the road, but they are still on the road.
- necovek 2y ago
  It depends on how you want to measure your safety when driven in one of these? Do you want them to beat a drunk driver, or an average, non-impaired human? If you got into a cab and cab driver was obviously drunk, would you ride along or get out asap? By "letting" them compare to an "average" (skewed by impaired drivers' incidents), we are letting them appear much better than an average, non-impaired driver, whereas they might be, or might not be (numbers clearly suggest they'd still beat them, but not by this much).
f1shy 2y ago
>>did we exclude drivers who were not eligible to drive (under the influence, no valid driving permit, cars with issues [braking broken]...)? now THAT would be massaging statistics! Just take the best drivers, that had enough sleep, no drink, perfect car conditions... No. You have to compare with the whole world, "as it is"
- necovek 2y ago
  What's the guarantee we'll have only fully up-to-date, safety-checked self-driving vehicles in the future with no messed up sensors or controls (there'll be impaired people doing the checks, right? :))? This is simple with a small, experimental fleet — so we are looking at the best case scenario for these vehicles, but the question is what does it fall down to in a realistic commercial application? That would also be comparing safety, because averages are always skewed by bad apples (i.e. 1 driver with 20 collisions gets an average to 2 for a group of 10 drivers with no other collisions). We at least need to start talking about medians, standard deviations groups and such. And we need autonomous vehicles to beat or match good drivers, otherwise, good drivers are worse off in the streets (and due to how averages are used, this might be more than 50% of drivers). Not sure why that's so controversial?
  - hnaccount_rng 2y ago
    I think you should open the PDF and look into that xD Really they discussed this. I'm still skeptical that they measure what we are interested in. But _your_ points they discuss
    - necovek 2y ago
      I went through the report, and they reference other studies that consider my points, but they end up only measuring incident numbers, don't specifically say how comparable were the driving conditions (iow whether they are representative). If anything, they could have removed incidents by human drivers which did not happen within their operating constraints, but I did not see that mentioned.

Reubend 2y ago

> did we exclude drivers who were not eligible to drive One could argue here that it would be unfair to exclude the worst performing humans. Plenty of humans cause accidents for easily preventable reasons, but that's an honest part of our behavior. Or to rephrase, I think excluding the categories of people who drive terribly, but do still drive nonetheless, would be less representative of reality. > Did Waymo drive across all the same areas and time periods as the humans did with an equal "proportion"? No, they didn't. Their methodology didn't adjust for things like highway vs city street miles, although it did adjust for city and state. > Could we compare "time behind wheel" instead of miles travelled? We could, but that would incentivize slow driving rather than safe driving. You need to travel the same minimum distance from a hypothetical "Point A" to "Point B" no matter what, but the number of minutes could be inflated easily. Economic factors discourage inflating the number of miles. > There are so many ways to slice and dice this and come up with favourable results for self-driving systems that I can't but be dubious of any claim. I think it's fair to criticize the fact that they didn't adjust for geographic area with as much granularity as they should have (i.e. for highways vs city streets, or for excluded zones like airports). But other than that, their methodology seems solid to me.

necovek 2y ago
I understand that viewpoint from an insurance perspective, but not necessarily from my own safety perspective: I wonder how well Waymo would fare in circumstances I see as a driver. Eg. humans can control whether they drive under the influence or not (eg. if you don't take your car to a bar, you ain't coming back driving it). But most notably, aggregated statistics like these focus on comparisons similar to comparing averages vs better statistical representations (like average vs median or p75 or... eg. it's possible for one driver to cause 20 collisions, and in a set of 10 drivers with no other collisions, we'd be at 2 collisions per driver instead of 0 really). It's also not only about highway/city street miles, it's also about time of day: if Waymo is proportionally less on the streets and areas where there is more risky driving (eg. around bars and around midnight to 3am when there are also drunk pedestrians around) compared to human drivers, that would obviously skew the numbers in SDV's favour. Again, potentially they did since they acknowledge a separate study focusing on those, but later on they only talk about aggregated insurance claims. They also acknowledge they are not accounting for accident severity, not even by using the dollar amount: while they had no fatalities, it's obviously important to weigh accidents by their severity. With all this said, this does demonstrate current insurance liability of an average human driver compared to a Waymo SDV in Phoenix and SFO, but only once we have more comparable data should we make a bigger claim.
- hnaccount_rng 2y ago
  I get where you are coming from and you are correct. This is, up to a point, a PR piece of Waymo and we should take that into account. But they are pretty transparent about what they are doing. You just have to read the report. And for their metric, which rounds to "all distance travelled in the interesting ZIP codes", they are succeeding. What I'd take away from this is: If you give them a different metric, they can optimise for that too and they will end up being better than humans. That doesn't answer the questions: 1) Is it worth it? (I.e. what does it cost to achieve that) and 2) Is a Waymo better in any situation? I guess they don't have the answer to either yet. I'm pretty sure their cost-per-mile is still dropping like a rock (probably though not fast enough for their accountants) and they haven't reached all situations yet either. But I wouldn't bet on "because it's too hard", but rather "because we didn't get there yet" (and yes, they will have started with the easy part, as they should)

myrmidon 2y ago

Regarding drivers that are drunk or otherwise driving illegally-- why would you want to exclude them from such a study, if you can not exclude them from real traffic? If our current system of checks and incentives is not able to keep them off the road, then you need to take them and the accidents they cause into account when comparing driving safety with an autonomous system. Also note that the margins are too big, since only about a third of accidents involve intoxicated drivers, and the accident reduction rates from the paper are much higher than that. I don't want to come across as harsh or accusing, but the "Methodology" section contains a lot of detail (and a more nuanced discussion could be had without disregarding it entirely :P).

necovek 2y ago
Because they don't disappear even with half the non-illegal vehicles being autonomous — some crashes also happen due to avoidance manoeuvres. So the fairer comparison would be to say that if existing SDVs were instead being driven by human drivers, we would have seen ~60 property claims more and ~40 bodily injury claims more (forgot the exact numbers from the paper), which is what, like 0.1% improvement in claims overall (certainly still a great result for the small comparative number of miles travelled, but doesn't jump off the page really). But because the miles traveled are so far apart, nothing is really a "fair" comparison.
- hnaccount_rng 2y ago
  But.. they would disappear? Not all of course. But some drive under the influence because there is no other convenient way to get home after a night out. And Waymo solves that! And they are highly aware of their "small number statistics", they literally provide uncertainty ranges. For both SDVs and HDVs. And their values are consistent with increasing data. I.e. they ~only get smaller uncertainties but no significant drift. But they are statistically significantly better (which wasn't the case two years ago)
  - necovek 2y ago
    Are you suggesting Waymo would become available where cabs aren't due to economics, or that Waymo-like technology in privately owned vehicles would solve that? Because I wouldn't buy the former, but I can see how the latter would, yet we are far away from that. I am not referring only to "small number statistics", but to obvious miscomparisons. We don't want SDVs to beat "average human driver", because average is worsened by impaired humans — we need them to "beat" at least an average, non-impaired human. Going by the numbers alone, they would achieve that too, but not by the same amount. But more importantly, we need them to compare both in identical driving conditions, which is where most of my complaint really is. Unless Waymo has driven in the same amount (proportionally) of rush-hour traffic, road-works, risky-drivers-around conditions, parking into tight spots and garages... of miles out of their 25M miles as their human counterparts, we could very well be looking into Waymo-in-simpler-situations vs humans-in-all-situations. And the numbers would be meaningless, which to me they seem they are. And it is important that they have achieved similar success with 6x the miles, but they need to account for more challenging conditions. And while some tricks can work while they are in small numbers (eg. avoiding road works by re-routing), it will lead to congestion elsewhere when such technology dominates the traffic. I would also note that it is impressive, regardless of the conditions they operated in, that they were involved in exactly 2 "bodily injury" claims over those 25M miles, and both of those appear not to have been the fault of Waymo.

jbarrow 2y ago

> did we exclude drivers who were not eligible to drive I don't think we can discount people who aren't fit/eligible to drive _if they still drove_. They might not have a _legal_ right to be on the road, but they are still on the road.

necovek 2y ago
It depends on how you want to measure your safety when driven in one of these? Do you want them to beat a drunk driver, or an average, non-impaired human? If you got into a cab and cab driver was obviously drunk, would you ride along or get out asap? By "letting" them compare to an "average" (skewed by impaired drivers' incidents), we are letting them appear much better than an average, non-impaired driver, whereas they might be, or might not be (numbers clearly suggest they'd still beat them, but not by this much).

f1shy 2y ago

>>did we exclude drivers who were not eligible to drive (under the influence, no valid driving permit, cars with issues [braking broken]...)? now THAT would be massaging statistics! Just take the best drivers, that had enough sleep, no drink, perfect car conditions... No. You have to compare with the whole world, "as it is"

necovek 2y ago
What's the guarantee we'll have only fully up-to-date, safety-checked self-driving vehicles in the future with no messed up sensors or controls (there'll be impaired people doing the checks, right? :))? This is simple with a small, experimental fleet — so we are looking at the best case scenario for these vehicles, but the question is what does it fall down to in a realistic commercial application? That would also be comparing safety, because averages are always skewed by bad apples (i.e. 1 driver with 20 collisions gets an average to 2 for a group of 10 drivers with no other collisions). We at least need to start talking about medians, standard deviations groups and such. And we need autonomous vehicles to beat or match good drivers, otherwise, good drivers are worse off in the streets (and due to how averages are used, this might be more than 50% of drivers). Not sure why that's so controversial?
- hnaccount_rng 2y ago
  I think you should open the PDF and look into that xD Really they discussed this. I'm still skeptical that they measure what we are interested in. But _your_ points they discuss
  - necovek 2y ago
    I went through the report, and they reference other studies that consider my points, but they end up only measuring incident numbers, don't specifically say how comparable were the driving conditions (iow whether they are representative). If anything, they could have removed incidents by human drivers which did not happen within their operating constraints, but I did not see that mentioned.