5 ms·

Waymo disengagement report

phaemon 9y ago
There isn't much to this blog post. The actual report is at: https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115-a72a-97f6f24b23cc/Waymofull.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115...
phsource 9y ago
It seems like Waymo's hitting a point of diminishing returns. The past 4 years' worth of disengagements per 1,000 miles driven were: - 2014 (last 3 months): 1.27 [1] - 2015: 0.80 [2] - 2016: 0.20 [2] - 2017: 0.18 [3] There are two possible causes off the top of my mind: 1. Waymo's cars are being tested in more demanding environments. 2. There's a floor to the number of disengagements due to technology and road conditions: situations that are different/unique enough that no model could accurately detect/respond to it. Case 2 would be definitely be a problem for the full-autonomous (i.e. zero user-input) type of car. I'd love to hear other hypotheses/feedback from those closer to this kind of work! Also, another interesting note: total miles driven on public roads is actually down from 2016 (352k vs. 636k earlier). Perhaps Waymo's finding the simulated/private-road tests more fruitful for winnowing down the final edge cases. [1] https://www.dmv.ca.gov/portal/wcm/connect/dff67186-70dd-4042-bc8c-d7b2a9904665/google_disengagement_report.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/dff67186-70dd-4042... [2] https://www.dmv.ca.gov/portal/wcm/connect/946b3502-c959-4e3b-b119-91319c27788f/GoogleAutoWaymo_disengage_report_2016.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/946b3502-c959-4e3b... [3] https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115-a72a-97f6f24b23cc/Waymofull.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115...
- Animats 9y ago
  Waymo is doing a lot more testing outside California now. They've done Mountain View so thoroughly they were seeing the same situations over and over. Last year, they were rear-ended twice at the same intersection. One disengagement every 5000 miles is a problem. It's too infrequent for humans to remain attentive, and too frequent to ignore.
  - toomuchtodo 9y ago
    The “Autopilot Valley”.
- kthejoker2 9y ago
  I think the devil's in the details a bit here. For example, for the first 251k miles driven in 2017 (71% of the total) they had 1 disengagement due to a "software discrepancy." Then over the next 43k miles they had 8 disengagements due to software. And then none since October. Looks like a regression bug to me. Similarly with "perception discrepancies" they had 14 disengagements for the first 175k miles driven (50%) and then 2 over the second half. Also only 24 of the 63 incidents were "failure detections" where the vehicle immediately handed over control of the vehicle; the rest were handed over "safely" (I assume this means the driver took over control voluntarily.) Also they don't say what the criteria is, but I suspect given what we've heard about the cars' conservative natures that "disengage for unwanted maneuver of the vehicle" is code for "the car is going too slow / being too passive at an intersection, etc." So all in all I see positive trends above and beyond the overall numbers. But I'm bullishly biased about self-driving cars, so get out your salt grains.
Animats 9y ago
Here are all the CA DMV disengagement reports, not just Waymo's.[1] Cruise is doing a lot of autonomous driving in SF. [1] https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2017 https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disen...

There isn't much to this blog post. The actual report is at: https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115-a72a-97f6f24b23cc/Waymofull.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115...

phsource 9y ago

It seems like Waymo's hitting a point of diminishing returns. The past 4 years' worth of disengagements per 1,000 miles driven were: - 2014 (last 3 months): 1.27 [1] - 2015: 0.80 [2] - 2016: 0.20 [2] - 2017: 0.18 [3] There are two possible causes off the top of my mind: 1. Waymo's cars are being tested in more demanding environments. 2. There's a floor to the number of disengagements due to technology and road conditions: situations that are different/unique enough that no model could accurately detect/respond to it. Case 2 would be definitely be a problem for the full-autonomous (i.e. zero user-input) type of car. I'd love to hear other hypotheses/feedback from those closer to this kind of work! Also, another interesting note: total miles driven on public roads is actually down from 2016 (352k vs. 636k earlier). Perhaps Waymo's finding the simulated/private-road tests more fruitful for winnowing down the final edge cases. [1] https://www.dmv.ca.gov/portal/wcm/connect/dff67186-70dd-4042-bc8c-d7b2a9904665/google_disengagement_report.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/dff67186-70dd-4042... [2] https://www.dmv.ca.gov/portal/wcm/connect/946b3502-c959-4e3b-b119-91319c27788f/GoogleAutoWaymo_disengage_report_2016.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/946b3502-c959-4e3b... [3] https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115-a72a-97f6f24b23cc/Waymofull.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115...

Animats 9y ago
Waymo is doing a lot more testing outside California now. They've done Mountain View so thoroughly they were seeing the same situations over and over. Last year, they were rear-ended twice at the same intersection. One disengagement every 5000 miles is a problem. It's too infrequent for humans to remain attentive, and too frequent to ignore.
- toomuchtodo 9y ago
  The “Autopilot Valley”.
kthejoker2 9y ago
I think the devil's in the details a bit here. For example, for the first 251k miles driven in 2017 (71% of the total) they had 1 disengagement due to a "software discrepancy." Then over the next 43k miles they had 8 disengagements due to software. And then none since October. Looks like a regression bug to me. Similarly with "perception discrepancies" they had 14 disengagements for the first 175k miles driven (50%) and then 2 over the second half. Also only 24 of the 63 incidents were "failure detections" where the vehicle immediately handed over control of the vehicle; the rest were handed over "safely" (I assume this means the driver took over control voluntarily.) Also they don't say what the criteria is, but I suspect given what we've heard about the cars' conservative natures that "disengage for unwanted maneuver of the vehicle" is code for "the car is going too slow / being too passive at an intersection, etc." So all in all I see positive trends above and beyond the overall numbers. But I'm bullishly biased about self-driving cars, so get out your salt grains.

Animats 9y ago

Here are all the CA DMV disengagement reports, not just Waymo's.[1] Cruise is doing a lot of autonomous driving in SF. [1] https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2017 https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disen...