The main objection I have is that a differentiable renderer is giving you the rope to hang yourself, it helps you work in the wrong space. There is some specia

The main objection I have is that a differentiable renderer is giving you the rope to hang yourself, it helps you work in the wrong space. There is some special structure to the rendering problem, that is probably tackled better. One inherent difficulty for efficient solving is working with multiple hypothesis at the same time (which differentiable renderer makes you miss). Also there is some special structure of the probability space which is "once you know the position of the camera it is easy to render, once you know the position of the objects it is easy to locate the camera". This means algorithm used for SLAM are a lot more able to tackle the problem. Techniques like Rao-Blackwellized Particle Filters, for example are more appropriate, It doesn't need a differentiable renderer for them and you can combine them with neural networks for example for pose estimation https://arxiv.org/abs/1905.09304 https://arxiv.org/abs/1905.09304 . What I try to say is that there is not a lot of useful extra information in a differentiable solver, and the information doesn't flow well. These gradients are on the detail scale, when what matters in on the large scale. Sure they can be used to stitch model together but that's an unreliable stitch, and you won't know if your agent is not driving because he can't take the right decision or your agent is not driving because he can't process the scene right. The gradients will also flow a lot better if you use a RL-model based approach where you ask the agent model to render what it think it sees and train it to match with the non-differentiable renderer. These gradient will be a lot smoother and on a larger scale.