Let's assume ego and other vehicle are on a straight lane and move with constant speed along the same line. All trajectories where the other vehicle is not fast

Let's assume ego and other vehicle are on a straight lane and move with constant speed along the same line. All trajectories where the other vehicle is not faster than ego are eventually colliding. But for each velocity that other vehicle projects to different positions, doesn't it? You can accumulate for one realtive motion only, but there are many dangerous ones. Taking ego speed and steering will accumulate standstill objects on the road plane (if the homography between image and road is accounter for properly). Clearly standstill objects in your path are on a collision course but the original problem was about moving and especially decelerating ones. Also it takes a large amount of frames to really average out. Pre-ML We used to implement multi-hypothesis Kalman filters and took the one with the least invention as best prediction.