3 ms·
DreamFusion doesn't directly build a 3D model from those generated images. It starts with a completely random 3D voxel model, renders it from 6 different angles
by nobbis 4y ago
DreamFusion doesn't directly build a 3D model from those generated images.
It starts with a completely random 3D voxel model, renders it from 6 different angles, then asks Stable Diffusion how plausible an image of "X, side view" it is.
It then sprinkles some noise on the rendering, makes Stable Diffusion improve it a little, then adjusts the voxels to produce that image (using differentiable rendering.)
Rinse and repeat for hours.
- shadowgovt 4y agoThank you for the clarification; I hadn't grokked the algorithm yet. That's interesting for a couple of reasons. I can see why that works. It also implies that for closed objects, the voxel data on the interior (where no images can see it) will be complete noise, as there's no signal to pick any color or lack of a voxel.
- nobbis 4y agoYes, although not complete noise – probably empty. Haven't checked but assume there's regularization of the NeRF parameters.
- FeepingCreature 4y agotext = f"{ref_text}, front cutaway drawing" Maybe?