5 ms·

My biggest issue with the "miles driven" metric is that it doesn't account for the software version. Maybe updating from v1.0 to v1.1 won't warrant resetting th

My biggest issue with the "miles driven" metric is that it doesn't account for the software version. Maybe updating from v1.0 to v1.1 won't warrant resetting the counter, but I believe that over time each historical mile driven becomes materially less relevant to currently deployed version.

ocdtrekkie 9y ago
I would argue the biggest issue with the "miles driven" metric is that it isn't contiguous. Your test drivers can handle all the rough stuff, and your mile counter ticks up on the straight shot highways. (Arguably, you could be super dishonest and handle every single intersection manually, and still have a massive miles driven statistic since most distance is accumulated on straight-ways.) Disengagements is a useful number, and the disengagement numbers say so much more about the real state of self-driving cars: That they fail as often as a non-synthetic oil change. And if humans had driving system failures that often, we would not be considered good at driving. The longest number a company should be marketing is how many miles they've driven between disengagements. Like the "days since an accident" sign at a workplace: Fail, and you go back to zero. The numbers will be smaller, but they'll be a lot more meaningful.
- saltyworker 9y ago
  what about hours, like a pilot?
  - Zigurd 9y ago
    Just like miles, which can look rather good if they are all highway miles, aircraft incidents are concentrated around takeoff and landing. Risk also skews results: Pilots taking off and landing on beaches in Alaska aren't comparable to pilots taking off from airports on the basis of incidents per 1000 hours.
  - ocdtrekkie 9y ago
    Highlight point is: A pilot's hours are in almost all cases "without an incident". The problem is Google's accounting is "total miles". Distance or time isn't the problem, it's the fact that it includes "incidents" that, without a test driver, would likely be "accidents".
  - praptak 9y ago
    All simple metrics can be gamed.
    - dredmorbius 9y ago
      s/simple//
- notatoad 9y ago
  >Arguably, you could be super dishonest and handle every single intersection manually, and still have a massive miles driven statistic since most distance is accumulated on straight-ways. i'm not sure what benefit there would be to gaming the disengagement stats. If anything, i think the incentive would be the other way around - inflate your number of disengagements so your competition thinks you're further behind than you actually are. The point of testing is to bring a product to market, and if your product sucks people are going to figure that out relatively quickly. hiding your disengagements from the california DMV isn't going to do anything for you except build hype your product can't live up to.
  - analognoise 9y ago
    Not if you're trying to get investment(s) or keep your stock price stable.
deleted 9y ago
[deleted]

ocdtrekkie 9y ago

I would argue the biggest issue with the "miles driven" metric is that it isn't contiguous. Your test drivers can handle all the rough stuff, and your mile counter ticks up on the straight shot highways. (Arguably, you could be super dishonest and handle every single intersection manually, and still have a massive miles driven statistic since most distance is accumulated on straight-ways.) Disengagements is a useful number, and the disengagement numbers say so much more about the real state of self-driving cars: That they fail as often as a non-synthetic oil change. And if humans had driving system failures that often, we would not be considered good at driving. The longest number a company should be marketing is how many miles they've driven between disengagements. Like the "days since an accident" sign at a workplace: Fail, and you go back to zero. The numbers will be smaller, but they'll be a lot more meaningful.

saltyworker 9y ago
what about hours, like a pilot?
- Zigurd 9y ago
  Just like miles, which can look rather good if they are all highway miles, aircraft incidents are concentrated around takeoff and landing. Risk also skews results: Pilots taking off and landing on beaches in Alaska aren't comparable to pilots taking off from airports on the basis of incidents per 1000 hours.
- ocdtrekkie 9y ago
  Highlight point is: A pilot's hours are in almost all cases "without an incident". The problem is Google's accounting is "total miles". Distance or time isn't the problem, it's the fact that it includes "incidents" that, without a test driver, would likely be "accidents".
- praptak 9y ago
  All simple metrics can be gamed.
  - dredmorbius 9y ago
    s/simple//
notatoad 9y ago
>Arguably, you could be super dishonest and handle every single intersection manually, and still have a massive miles driven statistic since most distance is accumulated on straight-ways. i'm not sure what benefit there would be to gaming the disengagement stats. If anything, i think the incentive would be the other way around - inflate your number of disengagements so your competition thinks you're further behind than you actually are. The point of testing is to bring a product to market, and if your product sucks people are going to figure that out relatively quickly. hiding your disengagements from the california DMV isn't going to do anything for you except build hype your product can't live up to.
- analognoise 9y ago
  Not if you're trying to get investment(s) or keep your stock price stable.

deleted 9y ago

[deleted]