5 ms·

We're headed to text-to-fullyAnimated3DScene while people are still debating ethics of LLMs text responses. We're not ready for the future that is fast approach

by dudus 3y ago

We're headed to text-to-fullyAnimated3DScene while people are still debating ethics of LLMs text responses. We're not ready for the future that is fast approaching.

deleted 3y ago
[deleted]
dr_kiszonka 3y ago
Unfortunately, humans tend to react sufficiently strongly only when something truly tragic happens all of a sudden. Things that creep in don't usually elicit such a response.
janosdebugs 3y ago
I'm very sceptical if this is going to actually manifest. My wife and I have been digging into the text to 3D topic for a while now and the problem seems to be that there is just not enough correctly labeled data and you have to deal with the complexity of having to generate a mesh in the third dimension. There is just a vast amount of 2D pictures with descriptive alt texts or other textual context on the Internet which is simply not true for 3D models. A lot of papers seem to try and extract 3D data from 2D pictures since the dataset is just so much better, but that hits obvious limitations. And we aren't even talking about any animations yet, which anyone who ever tried making them by hand knows, add a huge jump of complexity because you are not just morphing individual vertices, you actually want them to move consistently or else the faces are going to start doing weird glitchy things. Generating 3D stuff is less like generating pixel-based images and more like generating an SVG where the shapes actually make semantic sense, but a lot more complex. (Which, as far as we have found, no AI has managed so far either.) Sure, you'll start seeing animated faces, architectural plans, and you'll also see bone system, based animations for humanoids (humanoid 2D video to 3D bone animation AIs exist today), but don't expect to be able to prompt your way through creating a Pixar movie. However, I'm happy to be convinced otherwise.
- sorenjan 3y ago
  You don't have to generate a mesh token by token like you generate text. You can map the latent space to parametrized 3D models, or generate SDF, or a NeRF, or something else. See for instance: https://blog.metaphysic.ai/3d-morphable-models-3dmms/ https://blog.metaphysic.ai/3d-morphable-models-3dmms/ https://aibusiness.com/ml/meet-imagedream-an-ai-model-that-generates-3d-models-from-images https://aibusiness.com/ml/meet-imagedream-an-ai-model-that-g...

by dudus 3y ago

We're headed to text-to-fullyAnimated3DScene while people are still debating ethics of LLMs text responses. We're not ready for the future that is fast approaching.

deleted 3y ago

[deleted]

dr_kiszonka 3y ago

Unfortunately, humans tend to react sufficiently strongly only when something truly tragic happens all of a sudden. Things that creep in don't usually elicit such a response.

janosdebugs 3y ago

I'm very sceptical if this is going to actually manifest. My wife and I have been digging into the text to 3D topic for a while now and the problem seems to be that there is just not enough correctly labeled data and you have to deal with the complexity of having to generate a mesh in the third dimension. There is just a vast amount of 2D pictures with descriptive alt texts or other textual context on the Internet which is simply not true for 3D models. A lot of papers seem to try and extract 3D data from 2D pictures since the dataset is just so much better, but that hits obvious limitations. And we aren't even talking about any animations yet, which anyone who ever tried making them by hand knows, add a huge jump of complexity because you are not just morphing individual vertices, you actually want them to move consistently or else the faces are going to start doing weird glitchy things. Generating 3D stuff is less like generating pixel-based images and more like generating an SVG where the shapes actually make semantic sense, but a lot more complex. (Which, as far as we have found, no AI has managed so far either.) Sure, you'll start seeing animated faces, architectural plans, and you'll also see bone system, based animations for humanoids (humanoid 2D video to 3D bone animation AIs exist today), but don't expect to be able to prompt your way through creating a Pixar movie. However, I'm happy to be convinced otherwise.

sorenjan 3y ago
You don't have to generate a mesh token by token like you generate text. You can map the latent space to parametrized 3D models, or generate SDF, or a NeRF, or something else. See for instance: https://blog.metaphysic.ai/3d-morphable-models-3dmms/ https://blog.metaphysic.ai/3d-morphable-models-3dmms/ https://aibusiness.com/ml/meet-imagedream-an-ai-model-that-generates-3d-models-from-images https://aibusiness.com/ml/meet-imagedream-an-ai-model-that-g...