5 ms·

Threestudio – Framework for 3D content generation

118 points by victormustar 3y ago

dudus 3y ago
We're headed to text-to-fullyAnimated3DScene while people are still debating ethics of LLMs text responses. We're not ready for the future that is fast approaching.
- deleted 3y ago
  [deleted]
- dr_kiszonka 3y ago
  Unfortunately, humans tend to react sufficiently strongly only when something truly tragic happens all of a sudden. Things that creep in don't usually elicit such a response.
- janosdebugs 3y ago
  I'm very sceptical if this is going to actually manifest. My wife and I have been digging into the text to 3D topic for a while now and the problem seems to be that there is just not enough correctly labeled data and you have to deal with the complexity of having to generate a mesh in the third dimension. There is just a vast amount of 2D pictures with descriptive alt texts or other textual context on the Internet which is simply not true for 3D models. A lot of papers seem to try and extract 3D data from 2D pictures since the dataset is just so much better, but that hits obvious limitations. And we aren't even talking about any animations yet, which anyone who ever tried making them by hand knows, add a huge jump of complexity because you are not just morphing individual vertices, you actually want them to move consistently or else the faces are going to start doing weird glitchy things. Generating 3D stuff is less like generating pixel-based images and more like generating an SVG where the shapes actually make semantic sense, but a lot more complex. (Which, as far as we have found, no AI has managed so far either.) Sure, you'll start seeing animated faces, architectural plans, and you'll also see bone system, based animations for humanoids (humanoid 2D video to 3D bone animation AIs exist today), but don't expect to be able to prompt your way through creating a Pixar movie. However, I'm happy to be convinced otherwise.
  - sorenjan 3y ago
    You don't have to generate a mesh token by token like you generate text. You can map the latent space to parametrized 3D models, or generate SDF, or a NeRF, or something else. See for instance: https://blog.metaphysic.ai/3d-morphable-models-3dmms/ https://blog.metaphysic.ai/3d-morphable-models-3dmms/ https://aibusiness.com/ml/meet-imagedream-an-ai-model-that-generates-3d-models-from-images https://aibusiness.com/ml/meet-imagedream-an-ai-model-that-g...
nextworddev 3y ago
This is like Langchain but for text to 3d. Awesome project!
shlubbert 3y ago
Rather confusing that this is seemingly completely unrelated to three.js
- Trigg3r 3y ago
  +1, would love to see some kind of IDE for three.js!
  - modeless 3y ago
    It does have https://threejs.org/editor/ https://threejs.org/editor/ Also in the category of web 3D engine scene editors is https://playcanvas.com/ https://playcanvas.com/
  - frading 3y ago
    I'm building one, called Polygonjs ( https://polygonjs.com/ https://polygonjs.com/ ), you have a few examples to play with ( https://polygonjs.com/docs/examples https://polygonjs.com/docs/examples ).
deleted 3y ago
[deleted]
mattigames 3y ago
Soon enough the only differentiator for videogames will be the marketing budget and brand power, soon some indie games will have AAA graphics and animations but zero sales as the public gets flooded with offers and become numb to new offerings, instead the differentiator would be what games your friends play (e.g. to play online together) and similar social dynamics, and not much the technical aspects; a world where everyone can paint the Mona Lisa makes it less valuable, but DaVinci would still be a solid brand for historical reason, but new painters not so much.
- capableweb 3y ago
  The real division between games and games will, as it always has been, be if the game is fun or not. Nothing else matters. If it's not fun, it'll flop and disappear, some quicker than others. But if it's fun, people will continue to play it.
  - TrevorJ 3y ago
    this elides the discovery problem somewhat.
  - mattigames 3y ago
    This doesn't ring true, just for example "Among Us" sales only skyrocket after a famous streamer picked it up, the sales were below average before that specific event that happened by mere luck, and it was just as fun before he made it popular.
- Log_out_ 3y ago
  I think the differentiator will be a novel experience and that can only created by novel human characters. The last kind of the hill, standing over the kill, will be a author. What a ironic circle..
  - mattigames 3y ago
    There is no evidence to support such claim, might as well the last king of the hill will be those who can find one amazing story among the miriads created by AI and pretend it was created by him/her.

118 points by victormustar 3y ago

We're headed to text-to-fullyAnimated3DScene while people are still debating ethics of LLMs text responses. We're not ready for the future that is fast approaching.

deleted 3y ago
[deleted]
dr_kiszonka 3y ago
Unfortunately, humans tend to react sufficiently strongly only when something truly tragic happens all of a sudden. Things that creep in don't usually elicit such a response.
janosdebugs 3y ago
I'm very sceptical if this is going to actually manifest. My wife and I have been digging into the text to 3D topic for a while now and the problem seems to be that there is just not enough correctly labeled data and you have to deal with the complexity of having to generate a mesh in the third dimension. There is just a vast amount of 2D pictures with descriptive alt texts or other textual context on the Internet which is simply not true for 3D models. A lot of papers seem to try and extract 3D data from 2D pictures since the dataset is just so much better, but that hits obvious limitations. And we aren't even talking about any animations yet, which anyone who ever tried making them by hand knows, add a huge jump of complexity because you are not just morphing individual vertices, you actually want them to move consistently or else the faces are going to start doing weird glitchy things. Generating 3D stuff is less like generating pixel-based images and more like generating an SVG where the shapes actually make semantic sense, but a lot more complex. (Which, as far as we have found, no AI has managed so far either.) Sure, you'll start seeing animated faces, architectural plans, and you'll also see bone system, based animations for humanoids (humanoid 2D video to 3D bone animation AIs exist today), but don't expect to be able to prompt your way through creating a Pixar movie. However, I'm happy to be convinced otherwise.
- sorenjan 3y ago
  You don't have to generate a mesh token by token like you generate text. You can map the latent space to parametrized 3D models, or generate SDF, or a NeRF, or something else. See for instance: https://blog.metaphysic.ai/3d-morphable-models-3dmms/ https://blog.metaphysic.ai/3d-morphable-models-3dmms/ https://aibusiness.com/ml/meet-imagedream-an-ai-model-that-generates-3d-models-from-images https://aibusiness.com/ml/meet-imagedream-an-ai-model-that-g...

nextworddev 3y ago

This is like Langchain but for text to 3d. Awesome project!

shlubbert 3y ago

Rather confusing that this is seemingly completely unrelated to three.js

Trigg3r 3y ago
+1, would love to see some kind of IDE for three.js!
- modeless 3y ago
  It does have https://threejs.org/editor/ https://threejs.org/editor/ Also in the category of web 3D engine scene editors is https://playcanvas.com/ https://playcanvas.com/
- frading 3y ago
  I'm building one, called Polygonjs ( https://polygonjs.com/ https://polygonjs.com/ ), you have a few examples to play with ( https://polygonjs.com/docs/examples https://polygonjs.com/docs/examples ).

deleted 3y ago

[deleted]

mattigames 3y ago

Soon enough the only differentiator for videogames will be the marketing budget and brand power, soon some indie games will have AAA graphics and animations but zero sales as the public gets flooded with offers and become numb to new offerings, instead the differentiator would be what games your friends play (e.g. to play online together) and similar social dynamics, and not much the technical aspects; a world where everyone can paint the Mona Lisa makes it less valuable, but DaVinci would still be a solid brand for historical reason, but new painters not so much.

capableweb 3y ago
The real division between games and games will, as it always has been, be if the game is fun or not. Nothing else matters. If it's not fun, it'll flop and disappear, some quicker than others. But if it's fun, people will continue to play it.
- TrevorJ 3y ago
  this elides the discovery problem somewhat.
- mattigames 3y ago
  This doesn't ring true, just for example "Among Us" sales only skyrocket after a famous streamer picked it up, the sales were below average before that specific event that happened by mere luck, and it was just as fun before he made it popular.
Log_out_ 3y ago
I think the differentiator will be a novel experience and that can only created by novel human characters. The last kind of the hill, standing over the kill, will be a author. What a ironic circle..
- mattigames 3y ago
  There is no evidence to support such claim, might as well the last king of the hill will be those who can find one amazing story among the miriads created by AI and pretend it was created by him/her.