3 ms·

This is less a problem with differentiable rendering than a problem of defining image distance, which I agree is hard (though there are plenty of interesting ap

This is less a problem with differentiable rendering than a problem of defining image distance, which I agree is hard (though there are plenty of interesting approaches, like losses defined by conv nets). More interesting though is when the renderer is part of a larger pipeline, and the loss is defined some other way (such as how well the agent is driving, in our DuckieTown simulator). Backprop through the renderer lets us do BPTT, which is very powerful, but we don't ever have to compare images, avoiding that non-differentiability issue. Another example would be the autoencoder-like setups you can use to bootstrap CV models. It remains to be seen how effective these things are in practice, of course, but there are plenty of interesting directions.

GistNoesis 7y ago
The main objection I have is that a differentiable renderer is giving you the rope to hang yourself, it helps you work in the wrong space. There is some special structure to the rendering problem, that is probably tackled better. One inherent difficulty for efficient solving is working with multiple hypothesis at the same time (which differentiable renderer makes you miss). Also there is some special structure of the probability space which is "once you know the position of the camera it is easy to render, once you know the position of the objects it is easy to locate the camera". This means algorithm used for SLAM are a lot more able to tackle the problem. Techniques like Rao-Blackwellized Particle Filters, for example are more appropriate, It doesn't need a differentiable renderer for them and you can combine them with neural networks for example for pose estimation https://arxiv.org/abs/1905.09304 https://arxiv.org/abs/1905.09304 . What I try to say is that there is not a lot of useful extra information in a differentiable solver, and the information doesn't flow well. These gradients are on the detail scale, when what matters in on the large scale. Sure they can be used to stitch model together but that's an unreliable stitch, and you won't know if your agent is not driving because he can't take the right decision or your agent is not driving because he can't process the scene right. The gradients will also flow a lot better if you use a RL-model based approach where you ask the agent model to render what it think it sees and train it to match with the non-differentiable renderer. These gradient will be a lot smoother and on a larger scale.

GistNoesis 7y ago

The main objection I have is that a differentiable renderer is giving you the rope to hang yourself, it helps you work in the wrong space. There is some special structure to the rendering problem, that is probably tackled better. One inherent difficulty for efficient solving is working with multiple hypothesis at the same time (which differentiable renderer makes you miss). Also there is some special structure of the probability space which is "once you know the position of the camera it is easy to render, once you know the position of the objects it is easy to locate the camera". This means algorithm used for SLAM are a lot more able to tackle the problem. Techniques like Rao-Blackwellized Particle Filters, for example are more appropriate, It doesn't need a differentiable renderer for them and you can combine them with neural networks for example for pose estimation https://arxiv.org/abs/1905.09304 https://arxiv.org/abs/1905.09304 . What I try to say is that there is not a lot of useful extra information in a differentiable solver, and the information doesn't flow well. These gradients are on the detail scale, when what matters in on the large scale. Sure they can be used to stitch model together but that's an unreliable stitch, and you won't know if your agent is not driving because he can't take the right decision or your agent is not driving because he can't process the scene right. The gradients will also flow a lot better if you use a RL-model based approach where you ask the agent model to render what it think it sees and train it to match with the non-differentiable renderer. These gradient will be a lot smoother and on a larger scale.