Indeed, "why not just"; I can't tell if you're being sarcastic or not. I'll assume the latter: Even if you were able to perfectly maintain an accurate 3D model

Indeed, "why not just"; I can't tell if you're being sarcastic or not. I'll assume the latter: Even if you were able to perfectly maintain an accurate 3D model of the surrounding geometry (and associated trajectory of all moving objects), it doesn't mean you have enough information to make 100% accurate predictions. For example, a frozen over road often has very nearly exactly the same physical geometry and color, but extremely different behavior to cars trying to drive on it. Other weather effects like fog or snow bring their own challenges. You're right though, that precise and accurate 3D models do do a lot to make driving easier and more correct, the problem is that the hardware necessary to produce those models is still very expensive (LIDAR). Most (all?) of the advanced self-driving prototype cars use LIDAR, while Tesla has decided that they can do without for their consumer models, sticking with just the simpler and less expensive RADAR and cameras. In theory, Tesla's right, since humans get by just fine with only two "cameras", but we're clearly a long way from automating that level of performance.