4 ms·

ELI5 please

saagarjha 8y ago
I think it's doing this: https://en.wikipedia.org/wiki/Gradient-domain_image_processing https://en.wikipedia.org/wiki/Gradient-domain_image_processi...?
- remcob 8y ago
  Don't think so. This solves problems in the 2D domain, but the paper solves problems in the 3D domain.
- research_agency 8y ago
  It's not really about image processing, rather scene processing. A scene is kind of like a parameterizable hyper-image that depends on real world parameters (such as angle, light diffusivity, etc.). OTOH not all images come from scenes.
remcob 8y ago
Rendering a scene is a function that takes lights, camera, geometry, textures as inputs and outputs an image. This function is very complex and hard to invert. But, and this is the interesting thing I got from the article, it is similar enough to deep learning functions that the same strategy works: We can take gradients of it and optimize the inputs to fit the target output. So, while we can't compute the 3D model that would generate the target picture directly. We can ask it "How do we nudge the model to make it more similar to what we want?". If we keep nudging it, eventually it will be very similar. The tool will find the exact position of the lights, the textures, model, etc.
- spuz 8y ago
  What is the purpose of doing this though? Why would you want to find the original parameters of a scene that was ray traced? Wouldn't those parameters already be available?
  - macromaniac 8y ago
    No,you cant tell how pixels will change when you nudge a camera normally without recalculating everything. If environment is differentiable you can cheat and see for instance that you needed to move your arm a little to the left last bowl to get a strike (moving your arm effects your hand which shifts the ball which knocks down more pins which increases your score)
  - Scene_Cast2 8y ago
    The potential uses I see are: 1) Given a finished 3D render and a first pass at recreating the scene, nudge your scene closer to the render 2) Given a photograph and a first pass at recreating the photograph, make your render more photorealistic.
  - JulianMorrison 8y ago
    Given an image from a physical camera, convert it into a 3D map with comparable properties.
modeless 8y ago
A differentiable renderer can efficiently answer the question "If I tweaked this input parameter (e.g. camera position, object color, light direction) then which output pixels on screen would change, and by how much?" With this information you can run an optimization algorithm to repeatedly tweak the input parameters in the right direction to make the output pixels look more like a target that you want, e.g. a photograph. Even better, using the chain rule you can connect the differentiable renderer to other differentiable things, like a neural network controller for a simulated robot. Then you can ask e.g. "If I tweaked the robot's arm position, which camera pixels would change?" This would allow you to, given an image of a target configuration (e.g. two stacked blocks), optimize the neural network to produce that image by controlling the robot (e.g. by stacking the blocks).

ELI5 please

I think it's doing this: https://en.wikipedia.org/wiki/Gradient-domain_image_processing https://en.wikipedia.org/wiki/Gradient-domain_image_processi...?

remcob 8y ago
Don't think so. This solves problems in the 2D domain, but the paper solves problems in the 3D domain.
research_agency 8y ago
It's not really about image processing, rather scene processing. A scene is kind of like a parameterizable hyper-image that depends on real world parameters (such as angle, light diffusivity, etc.). OTOH not all images come from scenes.

remcob 8y ago

Rendering a scene is a function that takes lights, camera, geometry, textures as inputs and outputs an image. This function is very complex and hard to invert. But, and this is the interesting thing I got from the article, it is similar enough to deep learning functions that the same strategy works: We can take gradients of it and optimize the inputs to fit the target output. So, while we can't compute the 3D model that would generate the target picture directly. We can ask it "How do we nudge the model to make it more similar to what we want?". If we keep nudging it, eventually it will be very similar. The tool will find the exact position of the lights, the textures, model, etc.

spuz 8y ago
What is the purpose of doing this though? Why would you want to find the original parameters of a scene that was ray traced? Wouldn't those parameters already be available?
- macromaniac 8y ago
  No,you cant tell how pixels will change when you nudge a camera normally without recalculating everything. If environment is differentiable you can cheat and see for instance that you needed to move your arm a little to the left last bowl to get a strike (moving your arm effects your hand which shifts the ball which knocks down more pins which increases your score)
- Scene_Cast2 8y ago
  The potential uses I see are: 1) Given a finished 3D render and a first pass at recreating the scene, nudge your scene closer to the render 2) Given a photograph and a first pass at recreating the photograph, make your render more photorealistic.
- JulianMorrison 8y ago
  Given an image from a physical camera, convert it into a 3D map with comparable properties.

modeless 8y ago

A differentiable renderer can efficiently answer the question "If I tweaked this input parameter (e.g. camera position, object color, light direction) then which output pixels on screen would change, and by how much?" With this information you can run an optimization algorithm to repeatedly tweak the input parameters in the right direction to make the output pixels look more like a target that you want, e.g. a photograph. Even better, using the chain rule you can connect the differentiable renderer to other differentiable things, like a neural network controller for a simulated robot. Then you can ask e.g. "If I tweaked the robot's arm position, which camera pixels would change?" This would allow you to, given an image of a target configuration (e.g. two stacked blocks), optimize the neural network to produce that image by controlling the robot (e.g. by stacking the blocks).