2 ms·
It doesn't appear that this project uses it, but a 3D model would give you all the other information like depth maps/normal maps that you would need to light th
by astrange 3y ago
It doesn't appear that this project uses it, but a 3D model would give you all the other information like depth maps/normal maps that you would need to light the object itself properly. i.e. change the pixels of the object and not just draw a background around it.
- neilxm 3y agoThat’s the idea for next steps. Basically if you generate a backdrop and then estimate light direction you can inverse render that onto the 3d model given all the depth information you get for free from the model
- astrange 3y agoYou should be able to do relighting with ControlNet. Basically render the model to all the maps you'd use for PBR (fullbright color/depth/reflectivity/etc), train ControlNets that hold all those constant and do img2img but let it make up the background. Though I'm not aware of anyone already doing this, since I think research has moved on to NeRF models that act on 3D scenes directly.