4 ms·

In somewhat related topics, I think we can just use stable diffusion to help convert single photos to 3D NERFs. 1. find the prompt that best generates the imag

In somewhat related topics, I think we can just use stable diffusion to help convert single photos to 3D NERFs. 1. find the prompt that best generates the image 2. generate a (crude) NERF from your starting image and render views from other angles 3. use stable diffusion with the views from other angles as seed images, refine them using the prompt from 1 combined with(add descriptions to generate "view from back", "view from top", etc 4. feed the refined views back to the NERF generator, keeping the initial photo view constant 5. Generate new views from the NERF, which should now be much more realistic. Run the above steps 2-5 in a loop indefinitely. Eventually you should end up with a highly accurate, realistic NERF which is full 3d from any angle, all from a single photo. Similar techniques could be used to extend the scene in all directions.

eutectic 4y ago
I have my doubts that this will converge to anything meaningful.
- aliqot 4y ago
  In the short term you may be right, but in the long run it's a certainty you won't.
rsp1984 4y ago
The problem with such an approach would be that NERFs require a set of input images with their exact poses and exact poses are only available if the underlying geometry is static. However if you use SD to generate new views it's only an approximation and you wouldn't be able to get the exact poses. Not all hope is lost though. I'm pretty sure in a few years (perhaps sooner) we'll be able to generate entire 3D scenes directly without going through 2D images as an intermediate step.
- eutectic 4y ago
  You can optimize the poses as part of the model.
- londons_explore 4y ago
  I don't see it as a blocker... Especially if you alternate NeRF and SD iterations - ie. don't generate a whole image each time, instead just choose a random angle, do NeRF reconstruction, then run a single SD iteration on that NeRF, and do another training step of the NeRF. That way, you know the exact pose for each image, because you chose it when rendering the NeRF.
- WhiteNoiz3 4y ago
  This 100%. Most images in SD are paintings and have a certain compostion (for instance people usually looking at the camera, posed a certain way, etc). Trying to generate the same thing from a different angle will yield wildly different results which won't work well for Nerf. I do think diffusion models may work for generating 3d stuff, but right now a) We need a big labelled dataset of various 3d objects in different styles (yes, I know some exist but they are nowhere near what we have for 2d images since 2d images can be scraped from the web) 2) Memory/computation is a big bottleneck as 3d is orders of magnitude bigger in terms of memory and processing. That said, hopefully some smart person will come up with some shortcuts that make it possible on today's hardware.
KidComputer 4y ago
That doesn't work because of the contradiction of NERFs needing multiple images of the same geometry and stable diffusion not producing images with stable underlying geometry because it's purely image-based.

eutectic 4y ago

I have my doubts that this will converge to anything meaningful.

aliqot 4y ago
In the short term you may be right, but in the long run it's a certainty you won't.

rsp1984 4y ago

The problem with such an approach would be that NERFs require a set of input images with their exact poses and exact poses are only available if the underlying geometry is static. However if you use SD to generate new views it's only an approximation and you wouldn't be able to get the exact poses. Not all hope is lost though. I'm pretty sure in a few years (perhaps sooner) we'll be able to generate entire 3D scenes directly without going through 2D images as an intermediate step.

eutectic 4y ago
You can optimize the poses as part of the model.
londons_explore 4y ago
I don't see it as a blocker... Especially if you alternate NeRF and SD iterations - ie. don't generate a whole image each time, instead just choose a random angle, do NeRF reconstruction, then run a single SD iteration on that NeRF, and do another training step of the NeRF. That way, you know the exact pose for each image, because you chose it when rendering the NeRF.
WhiteNoiz3 4y ago
This 100%. Most images in SD are paintings and have a certain compostion (for instance people usually looking at the camera, posed a certain way, etc). Trying to generate the same thing from a different angle will yield wildly different results which won't work well for Nerf. I do think diffusion models may work for generating 3d stuff, but right now a) We need a big labelled dataset of various 3d objects in different styles (yes, I know some exist but they are nowhere near what we have for 2d images since 2d images can be scraped from the web) 2) Memory/computation is a big bottleneck as 3d is orders of magnitude bigger in terms of memory and processing. That said, hopefully some smart person will come up with some shortcuts that make it possible on today's hardware.

KidComputer 4y ago

That doesn't work because of the contradiction of NERFs needing multiple images of the same geometry and stable diffusion not producing images with stable underlying geometry because it's purely image-based.