3 ms·

Is FSD still operating on a frame-by-frame basis? I remember it was discussed on Autonomy day that the ideal implementation would operate on video feeds, and no

by manmal 5y ago

Is FSD still operating on a frame-by-frame basis? I remember it was discussed on Autonomy day that the ideal implementation would operate on video feeds, and not just the last frame, to improve accuracy. When you look at the dashboard visualizations of the cars‘ surroundings, the model that is built up looks quirky and inconsistent. Other cars flicker into view for one frame and disappear again; lane markings come and go. I saw a video where a car in front of the Tesla indicated, and the traffic light in the visualization (wrongly) started switching from red to green and back, in sync with the indicator blinking. How could a car behave correctly as long as its surroundings model is so flawed? As long as the dashboard viz isn’t a perfect mirror of what’s outside, this simply cannot work.

cptskippy 5y ago
I would think the flickering objects in the UI is a result of objects hovering around the confidence threshold of the model. But... I have a Model 3 and the flickering happens when stationary and nothing around you is moving.
joakleaf 5y ago
There also seem to be general problems with objects disappearing when they are obscured by other objects, and then reappear later, when no longer obscured. It is ridiculous, that the model doesn’t keep track of objects, and assume they continue with current velocity when they become obscured. It seems like a relatively simple thing to add depending on how they represent objects. You could even determine when the objects are obscured by other objects. In 8.x videos I noticed cars shifting and rotating a lot over fractions of a second, so it seemed like they needed a Kalman filter for objects and roads. Objects in 9.0 look more stable, but I still see lanes, curbs, and entire intersections shifting noticeably from frame to frame. So if they added time (multiple frames) to the model, it is still not working that well.
EForEndeavour 5y ago
You've nicely articulated what was bothering me about the jittery dashboard visualizations. Why on earth is everything flickering in and out of existence, and why is the car's own planned trajectory also flickering with discontinuities?? It seems like they aren't modeling the dimension of time, thus throwing away crucial information about speed and needlessly re-fitting the model to static snapshots of dynamic scenes. It's like the ML system needs its inferences constrained by rules like "objects don't teleport" or "acceleration is never infinite."

by manmal 5y ago

cptskippy 5y ago

I would think the flickering objects in the UI is a result of objects hovering around the confidence threshold of the model. But... I have a Model 3 and the flickering happens when stationary and nothing around you is moving.

joakleaf 5y ago

There also seem to be general problems with objects disappearing when they are obscured by other objects, and then reappear later, when no longer obscured. It is ridiculous, that the model doesn’t keep track of objects, and assume they continue with current velocity when they become obscured. It seems like a relatively simple thing to add depending on how they represent objects. You could even determine when the objects are obscured by other objects. In 8.x videos I noticed cars shifting and rotating a lot over fractions of a second, so it seemed like they needed a Kalman filter for objects and roads. Objects in 9.0 look more stable, but I still see lanes, curbs, and entire intersections shifting noticeably from frame to frame. So if they added time (multiple frames) to the model, it is still not working that well.

EForEndeavour 5y ago

You've nicely articulated what was bothering me about the jittery dashboard visualizations. Why on earth is everything flickering in and out of existence, and why is the car's own planned trajectory also flickering with discontinuities?? It seems like they aren't modeling the dimension of time, thus throwing away crucial information about speed and needlessly re-fitting the model to static snapshots of dynamic scenes. It's like the ML system needs its inferences constrained by rules like "objects don't teleport" or "acceleration is never infinite."