4 ms·

to my understanding it produces 2D images (from various angles), not 3D models… But sure, it's very close to producing a 3D model.

jsheard 3y ago
There's been some attempts at translating these techniques over to 3D models but the results so far aren't very useful, they have a tendency to produce extremely triangle-dense meshes which are ironically not actually very detailed or well defined, with most of the detailing just painted over the surface as textures, which also have terrible UV mapping. Not to mention the topology is a mess so it's impossible to rig for animation. I'm not sure how good they are at producing proper PBR material channels, but I'm guessing "not good" if they're derived from text-to-image models trained on fully rendered images/photos rather than individual PBR components, which is much more difficult data to source on a massive scale. I suspect this will continue to be an uphill battle because there aren't billions of high quality 3D models and PBR textures just lying around on the internet to slurp up and train a model on, so they're having to build it as a second-order system using images as the training data, and muddle though the rest of the steps to get to a usable 3D model.
- Etherlord87 3y ago
  I'm looking at such techniques that appear here on HN every now and then, and indeed, they are very impressive, but also something an intermediate 3D artist will do better with ease. I think however, that the existence of such tools could perhaps motivate more people to create low quality 3D assets, that for many purposes are good enough - but people might be shy to do that due to many high quality assets shaming them... Once AI 3D stuff will flood the space, amateurs might start competing with it - or so I hope. Also it's just a matter of time until we reach the 80% (from the 80/20 rule), 90% and 98% quality of these tools. The remaining 2% will still be of value in AAA titles though...
dheera 3y ago
MVDream can be combined with a NeRF (or equivalent spatial interpolator such as InstantNGP/TorchNGP) followed by marching cubes to extract a mesh and produce a 3D model. SDS loss and guidance is done from multiple equally-spaced views simultaneously. MVDream has a repo that implements this: https://github.com/bytedance/MVDream-threestudio https://github.com/bytedance/MVDream-threestudio It should be fairly straightforward to adapt this repo (which is based on threestudio) to use SPAD instead of MVDream.
deepnet 3y ago
Are there any NERF to textured or vertex coloured polygon mesh point cloud tools or papers ? I.e. can these be assets for traditional game engines ? Could a sort of photogrammetry like Meshroom or Reality capture do the trick ?

jsheard 3y ago

There's been some attempts at translating these techniques over to 3D models but the results so far aren't very useful, they have a tendency to produce extremely triangle-dense meshes which are ironically not actually very detailed or well defined, with most of the detailing just painted over the surface as textures, which also have terrible UV mapping. Not to mention the topology is a mess so it's impossible to rig for animation. I'm not sure how good they are at producing proper PBR material channels, but I'm guessing "not good" if they're derived from text-to-image models trained on fully rendered images/photos rather than individual PBR components, which is much more difficult data to source on a massive scale. I suspect this will continue to be an uphill battle because there aren't billions of high quality 3D models and PBR textures just lying around on the internet to slurp up and train a model on, so they're having to build it as a second-order system using images as the training data, and muddle though the rest of the steps to get to a usable 3D model.

Etherlord87 3y ago
I'm looking at such techniques that appear here on HN every now and then, and indeed, they are very impressive, but also something an intermediate 3D artist will do better with ease. I think however, that the existence of such tools could perhaps motivate more people to create low quality 3D assets, that for many purposes are good enough - but people might be shy to do that due to many high quality assets shaming them... Once AI 3D stuff will flood the space, amateurs might start competing with it - or so I hope. Also it's just a matter of time until we reach the 80% (from the 80/20 rule), 90% and 98% quality of these tools. The remaining 2% will still be of value in AAA titles though...

dheera 3y ago

MVDream can be combined with a NeRF (or equivalent spatial interpolator such as InstantNGP/TorchNGP) followed by marching cubes to extract a mesh and produce a 3D model. SDS loss and guidance is done from multiple equally-spaced views simultaneously. MVDream has a repo that implements this: https://github.com/bytedance/MVDream-threestudio https://github.com/bytedance/MVDream-threestudio It should be fairly straightforward to adapt this repo (which is based on threestudio) to use SPAD instead of MVDream.

deepnet 3y ago

Are there any NERF to textured or vertex coloured polygon mesh point cloud tools or papers ? I.e. can these be assets for traditional game engines ? Could a sort of photogrammetry like Meshroom or Reality capture do the trick ?