4 ms·
Yeah, you can do that as an inference step, like you described, and that has its uses without any learning required. But you can also make the output of the ren
by dimatura 7y ago
Yeah, you can do that as an inference step, like you described, and that has its uses without any learning required. But you can also make the output of the renderer depend (differentiably) on parameters apart from whatever you have as input - and those parameters can be learned via gradient descent. For example, you could make a conventional CNN take an image of say, a cube, as an input, and predict its 3D pose wrt to the camera, feed that prediction to the renderer, and have the renderer render the cube. Then, as training, e.g., a pixelwise error can be computed and backpropagated to the CNN. At the end of the process, ideally you would have a network capable of predicting the pose of a cube (and rendering a close match) in one shot, without iterative parameter tweaking.
- mov 7y agoSo, in that case, we could learn without the need of a differentiable renderer neither graphics operators, right? Maybe the throughput of communicating with external renderer is too much when compared with iterative parameter tweaking happening inside of loss functions on CNN?
- PeterisP 7y ago> we could learn without the need of a differentiable renderer neither graphics operators, right? No. The sentence of the parent post "a pixelwise error can be computed and backpropagated to the CNN" is possible only if the renderer is differentiable.
- mov 7y agoGot it. So let's suppose we have an external renderer, then we could learn parameters to tweak rendered scene and then get rendered pixels, so we can calculate pixelwise errors from it and some target image we're trying to optimize for. In this way, do we still need differentiable renderer in your opinion? Update: It would require more training cycles and would not be as "atomic" as iterative tweaks but seems possible. At same time, I wonder about making loss function talk with some external renderer would make it possible to mix both approaches.
- PeterisP 7y agoHow would you learn parameters to tweak the rendered scene if the renderer is not differentiable, and you can't backpropagate through the renderer to calculate the appropriate parameter adjustments from the pixelwise errors? I suppose you theoretically could do it with some trial and error method or grid search or something like that, but it's going to be absolutely computationally unfeasible in the general case; the pixelwise errors only become practically useful if you have an uninterrupted differentiable/'backpropagatable' path from your parameters to the pixels.
- mov 7y agoYes, throughput would be larger and we would loose the backprop path like you said, but it seems practical in some ways and actually guiding approaches like https://nv-tlabs.github.io/meta-sim/ https://nv-tlabs.github.io/meta-sim/
- antpls 7y agoIt sounds like the description of an auto-encoder. You encode the image of a cube to a representation of 3D points, then there is a decoder that transforms it back to an image of a cube. Am I correct ? Edit : nevermind, the article does state that it is similar to an autoencoder