3 ms·

>>did we exclude drivers who were not eligible to drive (under the influence, no valid driving permit, cars with issues [braking broken]...)? now THAT would be

by f1shy 2y ago

>>did we exclude drivers who were not eligible to drive (under the influence, no valid driving permit, cars with issues [braking broken]...)? now THAT would be massaging statistics! Just take the best drivers, that had enough sleep, no drink, perfect car conditions... No. You have to compare with the whole world, "as it is"

necovek 2y ago
What's the guarantee we'll have only fully up-to-date, safety-checked self-driving vehicles in the future with no messed up sensors or controls (there'll be impaired people doing the checks, right? :))? This is simple with a small, experimental fleet — so we are looking at the best case scenario for these vehicles, but the question is what does it fall down to in a realistic commercial application? That would also be comparing safety, because averages are always skewed by bad apples (i.e. 1 driver with 20 collisions gets an average to 2 for a group of 10 drivers with no other collisions). We at least need to start talking about medians, standard deviations groups and such. And we need autonomous vehicles to beat or match good drivers, otherwise, good drivers are worse off in the streets (and due to how averages are used, this might be more than 50% of drivers). Not sure why that's so controversial?
- hnaccount_rng 2y ago
  I think you should open the PDF and look into that xD Really they discussed this. I'm still skeptical that they measure what we are interested in. But _your_ points they discuss
  - necovek 2y ago
    I went through the report, and they reference other studies that consider my points, but they end up only measuring incident numbers, don't specifically say how comparable were the driving conditions (iow whether they are representative). If anything, they could have removed incidents by human drivers which did not happen within their operating constraints, but I did not see that mentioned.
    - hnaccount_rng 2y ago
      But that data isn’t available to them. They are (iiuc) simply looking at the full set of insurance data and their own set. There simply is no driver-level data which would allow statistically relevant conclusions. Up to now their data for their own cars is still not able to strongly constrain the results. You wouldn’t be able to tell effects of sensor degradation. But let me ask this instead: What kind of analysis would you like to see?
      - necovek 2y ago
        The report specifically mentions a number of other studies and claims it improves on their methodology, and seems to want to have their methodology become the "gold" standard of measuring SDV vs HDV performance: "This methodology establishes a foundation for future research into the safety impact potential of ADS and offers a framework for assessment as these systems continue to scale and develop." Or: "By analyzing an unprecedented volume of autonomous miles, introducing a novel benchmark, and utilizing third-party liability insurance claims, we aim to set a new standard for ADS safety evaluation." Which leads me to worry that they are pushing for this to be considered comparative, and they want policy to be influenced by their claims. And they do: "Our findings have the potential to significantly impact policy decisions, insurance practices, and public acceptance of ADS technology, contributing to the broader societal dialogue on the future of autonomous transportation." So I would like for them to compare to median non-impaired human driver with a car in good order (both in time-to-destination and safety), aggregate those over time of day, week and year (basically ensure identical driving conditions between comparison miles), adding claim dollar amount as weight for how severe the incident was. I believe that would be a great start. With them being Google and already tracking most Android phones as they go through traffic (they use this for traffic conditions), they do have a very rich data set they could compare to as well, but excluding that, they could exclude insurance claims which have happened in conditions they don't operate in (claims will always have time of event and location).
- fragmede 2y ago
  If you've used Google Chromecast or Google Chrome, a Tesla, or an Xbox, keeping software up to date isn't the chore it once was. Or you have to update to go online/have the car drive. The "guarantee" is that the other players who were too aggressive and did bad had to exit the industry. Uber killed a woman and exited the industry. Cruise dragged a woman 20 feet and lied about having footage of it, and exited the industry. So you ask what's the guarantee, and the guarantee is the same as when Google came out with Gmail, and everyone said that's stupid because how are you going to read your email when that website is down, so then they made it so that website doesn't go down.
  - necovek 2y ago
    How do you upgrade brake pads over the air? Or ensure tyre and wheel conditions? Or any of the other mechanical parts of the car (suspension, axles, transmission in-so-much-it's-still used...) Or ensure sensors are not interfered with? As soon as you get Waymo into private hands (or competitors need to cut corners), you'll start having issues like these. I don't think hoping for self-driving cars is stupid, and I would certainly love to have one I can trust to take me and my family around safely. But to get to that trust, I want better reporting than what this study provides and proposes :)
- f1shy 2y ago
  The study is about today. I would bot ever think to extrapolate that to “forever in the future” also is only one vendor. That is what the statistic shows and I think is fair comparison against human drivers currently on the street.
  - necovek 2y ago
    The study is about today and certainly has plenty of factual data in there, and their safety record is amazing. But they are purporting to make this the standard for comparing SDVs and HDVs, and explicitly want this report to influence policy. They say as much in the report.

by f1shy 2y ago

necovek 2y ago

What's the guarantee we'll have only fully up-to-date, safety-checked self-driving vehicles in the future with no messed up sensors or controls (there'll be impaired people doing the checks, right? :))? This is simple with a small, experimental fleet — so we are looking at the best case scenario for these vehicles, but the question is what does it fall down to in a realistic commercial application? That would also be comparing safety, because averages are always skewed by bad apples (i.e. 1 driver with 20 collisions gets an average to 2 for a group of 10 drivers with no other collisions). We at least need to start talking about medians, standard deviations groups and such. And we need autonomous vehicles to beat or match good drivers, otherwise, good drivers are worse off in the streets (and due to how averages are used, this might be more than 50% of drivers). Not sure why that's so controversial?

hnaccount_rng 2y ago
I think you should open the PDF and look into that xD Really they discussed this. I'm still skeptical that they measure what we are interested in. But _your_ points they discuss
- necovek 2y ago
  I went through the report, and they reference other studies that consider my points, but they end up only measuring incident numbers, don't specifically say how comparable were the driving conditions (iow whether they are representative). If anything, they could have removed incidents by human drivers which did not happen within their operating constraints, but I did not see that mentioned.
  - hnaccount_rng 2y ago
    But that data isn’t available to them. They are (iiuc) simply looking at the full set of insurance data and their own set. There simply is no driver-level data which would allow statistically relevant conclusions. Up to now their data for their own cars is still not able to strongly constrain the results. You wouldn’t be able to tell effects of sensor degradation. But let me ask this instead: What kind of analysis would you like to see?
    - necovek 2y ago
      The report specifically mentions a number of other studies and claims it improves on their methodology, and seems to want to have their methodology become the "gold" standard of measuring SDV vs HDV performance: "This methodology establishes a foundation for future research into the safety impact potential of ADS and offers a framework for assessment as these systems continue to scale and develop." Or: "By analyzing an unprecedented volume of autonomous miles, introducing a novel benchmark, and utilizing third-party liability insurance claims, we aim to set a new standard for ADS safety evaluation." Which leads me to worry that they are pushing for this to be considered comparative, and they want policy to be influenced by their claims. And they do: "Our findings have the potential to significantly impact policy decisions, insurance practices, and public acceptance of ADS technology, contributing to the broader societal dialogue on the future of autonomous transportation." So I would like for them to compare to median non-impaired human driver with a car in good order (both in time-to-destination and safety), aggregate those over time of day, week and year (basically ensure identical driving conditions between comparison miles), adding claim dollar amount as weight for how severe the incident was. I believe that would be a great start. With them being Google and already tracking most Android phones as they go through traffic (they use this for traffic conditions), they do have a very rich data set they could compare to as well, but excluding that, they could exclude insurance claims which have happened in conditions they don't operate in (claims will always have time of event and location).
fragmede 2y ago
If you've used Google Chromecast or Google Chrome, a Tesla, or an Xbox, keeping software up to date isn't the chore it once was. Or you have to update to go online/have the car drive. The "guarantee" is that the other players who were too aggressive and did bad had to exit the industry. Uber killed a woman and exited the industry. Cruise dragged a woman 20 feet and lied about having footage of it, and exited the industry. So you ask what's the guarantee, and the guarantee is the same as when Google came out with Gmail, and everyone said that's stupid because how are you going to read your email when that website is down, so then they made it so that website doesn't go down.
- necovek 2y ago
  How do you upgrade brake pads over the air? Or ensure tyre and wheel conditions? Or any of the other mechanical parts of the car (suspension, axles, transmission in-so-much-it's-still used...) Or ensure sensors are not interfered with? As soon as you get Waymo into private hands (or competitors need to cut corners), you'll start having issues like these. I don't think hoping for self-driving cars is stupid, and I would certainly love to have one I can trust to take me and my family around safely. But to get to that trust, I want better reporting than what this study provides and proposes :)
f1shy 2y ago
The study is about today. I would bot ever think to extrapolate that to “forever in the future” also is only one vendor. That is what the statistic shows and I think is fair comparison against human drivers currently on the street.
- necovek 2y ago
  The study is about today and certainly has plenty of factual data in there, and their safety record is amazing. But they are purporting to make this the standard for comparing SDVs and HDVs, and explicitly want this report to influence policy. They say as much in the report.