5 ms·

Tesla runs their new releases for millions of miles on the wide fleet in shadow mode before they release to limited beta. This is not an untested release in th

Tesla runs their new releases for millions of miles on the wide fleet in shadow mode before they release to limited beta. This is not an untested release in the way the article makes it seem.

6gvONxR4sf7o 6y ago
That's not as useful as it might sound. If a particular action the driver didn't take would lead to results that didn't happen, then shadow mode will have a lot of trouble knowing that. In general it's a problem called "off policy evaluation" and it's really hard. Tesla has to estimate what would have happened counterfactually, had the bot been driving, instead of seeing what happens with the bot driving. A million miles of this shadow mode evaluation isn't worth nearly as many miles of actual self driving.
- optimiz3 6y ago
  > If a particular action the driver didn't take would lead to results that didn't happen I'm having a hard time parsing your double-negative, but when shadow mode is running, it's choices are back-tested in real-time with the driver's choices, so I'm not sure how this fails to replicate what a real-driver would have done in such scenarios.
  - egsmi 6y ago
    Here an example. I’m on the freeway in my Tesla and instead of staying behind a car, at a safe following distance as predicted, I change lanes and speed past. Was it because, 1) the car was doing something I perceived as dangerous and I moved to a safer position, 2) I’m in a freaking Tesla and I can
  - matthewdgreen 6y ago
    Tesla's shadow mode is collecting real data up until the point where the car's decisionmaking process and the human driver disagree. Everything after that is simulated fantasy. And since human drivers aren't necessarily going to do "the right thing" in all circumstances (there isn't necessarily a single 'right thing') you can't treat 'car-disagrees-with-human' as an error case.
  - snovv_crash 6y ago
    Other cars respond to your actions, so the moment your actions deviate from the bot the reactions of the other cars are no longer valid to test against.
  - 6gvONxR4sf7o 6y ago
    Backtesting a driving model, it’s very hard to tell what would have happened, had the car deviated at all from what the driver did. e.g. You see the driver swerve for no apparent reason, did the driver simply drive more recklessly than the car would, or did they avoid an obstacle the model didn’t catch?
FartyMcFarter 6y ago
And yet there are several recent videos of Tesla cars doing crazy stuff on the roads with this system.
- optimiz3 6y ago
  Wait till you see the videos of humans doing crazy stuff on roads in cars!
  - bathtub365 6y ago
    Who is legally responsible for the car doing something crazy? Will engineers or executives go to jail?
  - sidibe 6y ago
    If humans did the crazy stuff on the road at the same rate as we've seen from the few select hours of FSD driving uploaded by Tesla owners (who are fans and more likely trying to show it at it's best) this week, we'd all be dead.
    - hobby-coder-guy 6y ago
      No
  - threeseed 6y ago
    Do people randomly brake at high speeds for no reason ? Tesla cars do: https://m.youtube.com/watch?v=xj73Q3lWvFM https://m.youtube.com/watch?v=xj73Q3lWvFM
    - optimiz3 6y ago
      All the time. Witness the effects of cellphones/children/food/spilled food/makeup/jarring news/exciting music/significant others/attractive people/gawking at random bits of interest. Also, brake-checking just cause' and for insurance fraud.
threeseed 6y ago
How does that explain the persistent issue with phantom breaking ? https://m.youtube.com/watch?v=xj73Q3lWvFM https://m.youtube.com/watch?v=xj73Q3lWvFM It's been a problem for years now.
- matthewdgreen 6y ago
  And it's scary as hell. Had it happen to me twice in a 600-mile highway roundtrip this summer. Was very happy nobody was following me closely, because the combination of regen breaks and disc breaks really stops the car quickly.

Tesla runs their new releases for millions of miles on the wide fleet in shadow mode before they release to limited beta. This is not an untested release in the way the article makes it seem.

6gvONxR4sf7o 6y ago

That's not as useful as it might sound. If a particular action the driver didn't take would lead to results that didn't happen, then shadow mode will have a lot of trouble knowing that. In general it's a problem called "off policy evaluation" and it's really hard. Tesla has to estimate what would have happened counterfactually, had the bot been driving, instead of seeing what happens with the bot driving. A million miles of this shadow mode evaluation isn't worth nearly as many miles of actual self driving.

optimiz3 6y ago
> If a particular action the driver didn't take would lead to results that didn't happen I'm having a hard time parsing your double-negative, but when shadow mode is running, it's choices are back-tested in real-time with the driver's choices, so I'm not sure how this fails to replicate what a real-driver would have done in such scenarios.
- egsmi 6y ago
  Here an example. I’m on the freeway in my Tesla and instead of staying behind a car, at a safe following distance as predicted, I change lanes and speed past. Was it because, 1) the car was doing something I perceived as dangerous and I moved to a safer position, 2) I’m in a freaking Tesla and I can
- matthewdgreen 6y ago
  Tesla's shadow mode is collecting real data up until the point where the car's decisionmaking process and the human driver disagree. Everything after that is simulated fantasy. And since human drivers aren't necessarily going to do "the right thing" in all circumstances (there isn't necessarily a single 'right thing') you can't treat 'car-disagrees-with-human' as an error case.
- snovv_crash 6y ago
  Other cars respond to your actions, so the moment your actions deviate from the bot the reactions of the other cars are no longer valid to test against.
- 6gvONxR4sf7o 6y ago
  Backtesting a driving model, it’s very hard to tell what would have happened, had the car deviated at all from what the driver did. e.g. You see the driver swerve for no apparent reason, did the driver simply drive more recklessly than the car would, or did they avoid an obstacle the model didn’t catch?

FartyMcFarter 6y ago

And yet there are several recent videos of Tesla cars doing crazy stuff on the roads with this system.

optimiz3 6y ago
Wait till you see the videos of humans doing crazy stuff on roads in cars!
- bathtub365 6y ago
  Who is legally responsible for the car doing something crazy? Will engineers or executives go to jail?
- sidibe 6y ago
  If humans did the crazy stuff on the road at the same rate as we've seen from the few select hours of FSD driving uploaded by Tesla owners (who are fans and more likely trying to show it at it's best) this week, we'd all be dead.
  - hobby-coder-guy 6y ago
    No
- threeseed 6y ago
  Do people randomly brake at high speeds for no reason ? Tesla cars do: https://m.youtube.com/watch?v=xj73Q3lWvFM https://m.youtube.com/watch?v=xj73Q3lWvFM
  - optimiz3 6y ago
    All the time. Witness the effects of cellphones/children/food/spilled food/makeup/jarring news/exciting music/significant others/attractive people/gawking at random bits of interest. Also, brake-checking just cause' and for insurance fraud.

threeseed 6y ago

How does that explain the persistent issue with phantom breaking ? https://m.youtube.com/watch?v=xj73Q3lWvFM https://m.youtube.com/watch?v=xj73Q3lWvFM It's been a problem for years now.

matthewdgreen 6y ago
And it's scary as hell. Had it happen to me twice in a 600-mile highway roundtrip this summer. Was very happy nobody was following me closely, because the combination of regen breaks and disc breaks really stops the car quickly.