4 ms·

Another interesting article popped up with the release of the CA disengagement reports: https://jalopnik.com/californias-autonomous-car-reports-are-the-best-in-

by kyrra 9y ago

Another interesting article popped up with the release of the CA disengagement reports: https://jalopnik.com/californias-autonomous-car-reports-are-the-best-in-the-1822606953 https://jalopnik.com/californias-autonomous-car-reports-are-... TLDR: Cruise had, what some would consider, to be a disengagement incident but didn't report it because it does not fall into the categories defined by California. The core issue here is that the disengagement reports should not be the end-all-be-all of where companies are. As the ieee article calls out, we have no clue what kind of testing the companies are doing (how hard they are pushing their systems). There is also some room for companies to avoid reporting certain types of incidents. Waymo did 2 million miles in 2017, but only 330k were in CA. We are only seeing a slice of their data. If we (the public) want a better picture of where companies are on this, the rules around what is reported need to be changed. They possibly need to also be done at the federal level so no company can hide their public testing in states that don't require reporting.

deegles 9y ago
My biggest issue with the "miles driven" metric is that it doesn't account for the software version. Maybe updating from v1.0 to v1.1 won't warrant resetting the counter, but I believe that over time each historical mile driven becomes materially less relevant to currently deployed version.
- ocdtrekkie 9y ago
  I would argue the biggest issue with the "miles driven" metric is that it isn't contiguous. Your test drivers can handle all the rough stuff, and your mile counter ticks up on the straight shot highways. (Arguably, you could be super dishonest and handle every single intersection manually, and still have a massive miles driven statistic since most distance is accumulated on straight-ways.) Disengagements is a useful number, and the disengagement numbers say so much more about the real state of self-driving cars: That they fail as often as a non-synthetic oil change. And if humans had driving system failures that often, we would not be considered good at driving. The longest number a company should be marketing is how many miles they've driven between disengagements. Like the "days since an accident" sign at a workplace: Fail, and you go back to zero. The numbers will be smaller, but they'll be a lot more meaningful.
  - saltyworker 9y ago
    what about hours, like a pilot?
    - Zigurd 9y ago
      Just like miles, which can look rather good if they are all highway miles, aircraft incidents are concentrated around takeoff and landing. Risk also skews results: Pilots taking off and landing on beaches in Alaska aren't comparable to pilots taking off from airports on the basis of incidents per 1000 hours.
    - ocdtrekkie 9y ago
      Highlight point is: A pilot's hours are in almost all cases "without an incident". The problem is Google's accounting is "total miles". Distance or time isn't the problem, it's the fact that it includes "incidents" that, without a test driver, would likely be "accidents".
    - praptak 9y ago
      All simple metrics can be gamed.
      - dredmorbius 9y ago
        s/simple//
  - notatoad 9y ago
    >Arguably, you could be super dishonest and handle every single intersection manually, and still have a massive miles driven statistic since most distance is accumulated on straight-ways. i'm not sure what benefit there would be to gaming the disengagement stats. If anything, i think the incentive would be the other way around - inflate your number of disengagements so your competition thinks you're further behind than you actually are. The point of testing is to bring a product to market, and if your product sucks people are going to figure that out relatively quickly. hiding your disengagements from the california DMV isn't going to do anything for you except build hype your product can't live up to.
- deleted 9y ago
  [deleted]
closeparen 9y ago
>If we (the public) want a better picture of where companies are on this Why? I'm as curious as anyone, but what is the argument for mandatory disclosure of disengagement numbers?
jacquesm 9y ago
The 'miles driven' number is a bit like years of programmer experience. It all depends on the content of the miles. And without transparency about that it sounds impressive but could very well be 95%+ that doesn't contribute much and where your average toddler without a driving license could have handled the car.

by kyrra 9y ago

deegles 9y ago

My biggest issue with the "miles driven" metric is that it doesn't account for the software version. Maybe updating from v1.0 to v1.1 won't warrant resetting the counter, but I believe that over time each historical mile driven becomes materially less relevant to currently deployed version.

ocdtrekkie 9y ago
I would argue the biggest issue with the "miles driven" metric is that it isn't contiguous. Your test drivers can handle all the rough stuff, and your mile counter ticks up on the straight shot highways. (Arguably, you could be super dishonest and handle every single intersection manually, and still have a massive miles driven statistic since most distance is accumulated on straight-ways.) Disengagements is a useful number, and the disengagement numbers say so much more about the real state of self-driving cars: That they fail as often as a non-synthetic oil change. And if humans had driving system failures that often, we would not be considered good at driving. The longest number a company should be marketing is how many miles they've driven between disengagements. Like the "days since an accident" sign at a workplace: Fail, and you go back to zero. The numbers will be smaller, but they'll be a lot more meaningful.
- saltyworker 9y ago
  what about hours, like a pilot?
  - Zigurd 9y ago
    Just like miles, which can look rather good if they are all highway miles, aircraft incidents are concentrated around takeoff and landing. Risk also skews results: Pilots taking off and landing on beaches in Alaska aren't comparable to pilots taking off from airports on the basis of incidents per 1000 hours.
  - ocdtrekkie 9y ago
    Highlight point is: A pilot's hours are in almost all cases "without an incident". The problem is Google's accounting is "total miles". Distance or time isn't the problem, it's the fact that it includes "incidents" that, without a test driver, would likely be "accidents".
  - praptak 9y ago
    All simple metrics can be gamed.
    - dredmorbius 9y ago
      s/simple//
- notatoad 9y ago
  >Arguably, you could be super dishonest and handle every single intersection manually, and still have a massive miles driven statistic since most distance is accumulated on straight-ways. i'm not sure what benefit there would be to gaming the disengagement stats. If anything, i think the incentive would be the other way around - inflate your number of disengagements so your competition thinks you're further behind than you actually are. The point of testing is to bring a product to market, and if your product sucks people are going to figure that out relatively quickly. hiding your disengagements from the california DMV isn't going to do anything for you except build hype your product can't live up to.
deleted 9y ago
[deleted]

closeparen 9y ago

>If we (the public) want a better picture of where companies are on this Why? I'm as curious as anyone, but what is the argument for mandatory disclosure of disengagement numbers?

jacquesm 9y ago

The 'miles driven' number is a bit like years of programmer experience. It all depends on the content of the miles. And without transparency about that it sounds impressive but could very well be 95%+ that doesn't contribute much and where your average toddler without a driving license could have handled the car.