>Yes the problem is very hard. Mainly because high DOF generalization is very difficult. >We have self driving cars because what are the control inputs? Pedal,

>Yes the problem is very hard. Mainly because high DOF generalization is very difficult. >We have self driving cars because what are the control inputs? Pedal, brake, steering wheel. This already took many many years. Its actually amazing to me that this hasn't been solved yet. Its really not that hard of a problem. Modern robotics, including self driving, are famously all about end-to-end training. We are trying to replicate what humans do through muscle memory. But muscle memory is not what makes us good at operating in the physical world. The thing that matters the most is our ability to simulate the world around us in a compressed form into the future, which lets us predict how our inputs will affect the world. A similar system in a self driving car should be able to drive perfectly without self inflicted accidents 100% of the time, especially with basic lidar to serve as an error correction mechanism to the camera 3d scene reconstruction.