3 ms·

This looks very cool, but can someone with insight into the topic explain why the graphics part needs to be differentiable? Couldn't we just automatically gene

by rsp1984 7y ago

This looks very cool, but can someone with insight into the topic explain why the graphics part needs to be differentiable? Couldn't we just automatically generate lots of graphics renderings (that are by definition already perfectly labelled) and use them to train a ML model? That wouldn't require any differentiability, would it? So how is this approach different?

PeterisP 7y ago
Differentiable graphics rendering allows you to transform the difference (what you got vs what you wanted) of the resulting image back to a difference of the underlying data which you rendered. It allows you to get from "these 1000 output pixels were wrong" to "the truck in the scene model actually was two inches to the left compared to what I expected/predicted" or "the texture of that apple should be changed this way to match reality". You wouldn't use it to tweak labels for image classification tasks, but to learn better underlying models of physical reality and behavior.
- rsp1984 7y ago
  I think I need a higher level explanation. Is the goal to tweak the rendering parameters until it matches a given input image? If yes that would be inference, not learning. What am I missing?
  - PeterisP 7y ago
    That's one use case - quoting the article, "[...] analysis by synthesis where the vision system extracts the scene parameters and the graphics system renders back an image based on them. If the rendering matches the original image, the vision system has accurately extracted the scene parameters." - so this allows you to learn a image-analysis system from unlabeled data in an iterative manner, rendering what you "understood", looking for differences, and adjusting your understanding. There are other use cases, for example, rendering "what-if" scenarios for reinforcement learning - e.g. you may have a model that predicts that if an agent does action sequence ABC then it will result in a world state X, and it can render an image Y which it expects to perceive. When it actually performs these actions, it actually obtains a different image Y' ... and needs to update its prediction, so it'd need differentiation/backpropagation through that rendering (and beyond) to figure out how it could have predicted the correct world state; so this feature is needed to allow learning the predictive model.
  - dimatura 7y ago
    Yeah, you can do that as an inference step, like you described, and that has its uses without any learning required. But you can also make the output of the renderer depend (differentiably) on parameters apart from whatever you have as input - and those parameters can be learned via gradient descent. For example, you could make a conventional CNN take an image of say, a cube, as an input, and predict its 3D pose wrt to the camera, feed that prediction to the renderer, and have the renderer render the cube. Then, as training, e.g., a pixelwise error can be computed and backpropagated to the CNN. At the end of the process, ideally you would have a network capable of predicting the pose of a cube (and rendering a close match) in one shot, without iterative parameter tweaking.
    - mov 7y ago
      So, in that case, we could learn without the need of a differentiable renderer neither graphics operators, right? Maybe the throughput of communicating with external renderer is too much when compared with iterative parameter tweaking happening inside of loss functions on CNN?
      - PeterisP 7y ago
        > we could learn without the need of a differentiable renderer neither graphics operators, right? No. The sentence of the parent post "a pixelwise error can be computed and backpropagated to the CNN" is possible only if the renderer is differentiable.
        mov 7y ago
        Got it. So let's suppose we have an external renderer, then we could learn parameters to tweak rendered scene and then get rendered pixels, so we can calculate pixelwise errors from it and some target image we're trying to optimize for. In this way, do we still need differentiable renderer in your opinion? Update: It would require more training cycles and would not be as "atomic" as iterative tweaks but seems possible. At same time, I wonder about making loss function talk with some external renderer would make it possible to mix both approaches.

PeterisP 7y ago

Differentiable graphics rendering allows you to transform the difference (what you got vs what you wanted) of the resulting image back to a difference of the underlying data which you rendered. It allows you to get from "these 1000 output pixels were wrong" to "the truck in the scene model actually was two inches to the left compared to what I expected/predicted" or "the texture of that apple should be changed this way to match reality". You wouldn't use it to tweak labels for image classification tasks, but to learn better underlying models of physical reality and behavior.

rsp1984 7y ago
I think I need a higher level explanation. Is the goal to tweak the rendering parameters until it matches a given input image? If yes that would be inference, not learning. What am I missing?
- PeterisP 7y ago
  That's one use case - quoting the article, "[...] analysis by synthesis where the vision system extracts the scene parameters and the graphics system renders back an image based on them. If the rendering matches the original image, the vision system has accurately extracted the scene parameters." - so this allows you to learn a image-analysis system from unlabeled data in an iterative manner, rendering what you "understood", looking for differences, and adjusting your understanding. There are other use cases, for example, rendering "what-if" scenarios for reinforcement learning - e.g. you may have a model that predicts that if an agent does action sequence ABC then it will result in a world state X, and it can render an image Y which it expects to perceive. When it actually performs these actions, it actually obtains a different image Y' ... and needs to update its prediction, so it'd need differentiation/backpropagation through that rendering (and beyond) to figure out how it could have predicted the correct world state; so this feature is needed to allow learning the predictive model.
- dimatura 7y ago
  Yeah, you can do that as an inference step, like you described, and that has its uses without any learning required. But you can also make the output of the renderer depend (differentiably) on parameters apart from whatever you have as input - and those parameters can be learned via gradient descent. For example, you could make a conventional CNN take an image of say, a cube, as an input, and predict its 3D pose wrt to the camera, feed that prediction to the renderer, and have the renderer render the cube. Then, as training, e.g., a pixelwise error can be computed and backpropagated to the CNN. At the end of the process, ideally you would have a network capable of predicting the pose of a cube (and rendering a close match) in one shot, without iterative parameter tweaking.
  - mov 7y ago
    So, in that case, we could learn without the need of a differentiable renderer neither graphics operators, right? Maybe the throughput of communicating with external renderer is too much when compared with iterative parameter tweaking happening inside of loss functions on CNN?
    - PeterisP 7y ago
      > we could learn without the need of a differentiable renderer neither graphics operators, right? No. The sentence of the parent post "a pixelwise error can be computed and backpropagated to the CNN" is possible only if the renderer is differentiable.
      - mov 7y ago
        Got it. So let's suppose we have an external renderer, then we could learn parameters to tweak rendered scene and then get rendered pixels, so we can calculate pixelwise errors from it and some target image we're trying to optimize for. In this way, do we still need differentiable renderer in your opinion? Update: It would require more training cycles and would not be as "atomic" as iterative tweaks but seems possible. At same time, I wonder about making loss function talk with some external renderer would make it possible to mix both approaches.