6 ms·

GALA3D: Towards Text-to-3D Complex Scene Generation

78 points by jfoster 3y ago

yakorevivan 3y ago
[dead]
bbor 3y ago
The media is claiming we’re in an AI Cold War with China, yet here’s a sinoamerican paper casually talking about automating the simple task of the entirety of human spatial understanding. Maybe we can all get along, after all? EDIT: is this paper… broken…? for anyone else? I cant read past the first two pages on phone or Mac, haven’t tried Linux. https://arxiv.org/pdf/2402.07207.pdf https://arxiv.org/pdf/2402.07207.pdf
- ImHereToVote 3y ago
  The US media is claiming. The CIA and Fake News in 1980s https://www.youtube.com/watch?v=XibCflWxZuA https://www.youtube.com/watch?v=XibCflWxZuA
  - bbor 3y ago
    This is more accurate than I gave you initial credit for! I couldn’t find a single Chinese source of note from their government or media. Eg https://en.wikipedia.org/wiki/Artificial_Intelligence_Cold_War https://en.wikipedia.org/wiki/Artificial_Intelligence_Cold_W...
    - yorwba 3y ago
      Not hard to find Chinese sources if you know Chinese. http://old2022.bulletin.cas.cn/publish_article/2024/1/20240113.htm http://old2022.bulletin.cas.cn/publish_article/2024/1/202401... This one even has an English abstract.
- ben_w 3y ago
  Individuals have an easier time cooperating than their respective governments.
  - bbor 3y ago
    Yeah, I guess I was alluding to the idea that the disagreements between the governments are not based on the actual concerns of the people. Tho it’s hard to tell at what level the vague nationalistic ideas of “we deserve to rule X land” and “they’re brainwashed” can be described as actual concerns, much less more openly racist and xenophobic ones.
Larok00 3y ago
I wish there was more progress in text-to-3D mesh for creating basic but very specific and functional shapes. With the last few years of progress, it really feels like it should be possible, but none of the big players are finding it worthwhile to look at. It would give the 3D printing community a massive boost.
- psytrx 3y ago
  Have you seen https://text-to-cad.zoo.dev/ https://text-to-cad.zoo.dev/ ? But yes, there's a lot of potential and, in my uneducated opinion, low hanging fruits for 3D printing especially.
- yorwba 3y ago
  When you want a very specific shape, text is probably not the right input modality. See also image generation, where to get very specific outputs, you're better off defining the large-scale structure spatially with a controlnet and only using text for the visual style and decorative details that do not need to be precisely controlled. What shapes would you ask a text-to-3D model to create for you?
  - alpaca128 3y ago
    That makes me wonder whether an LLM for code generation could actually work for OpenSCAD-style designs as well, especially as it has a quite small set of functions. The only thing that makes me doubtful is that the coordinates have to correctly map to 3D space.
- qwery 3y ago
  Depends what "very specific and functional shapes" you have in mind, I guess. For my use of a 3D printer, I believe something like OpenSCAD is going to be a significantly more efficient textual description of the object than ~english -- for me and the computer.
  - Larok00 3y ago
    OpenSCAD will definitely be more efficient but not as many people speak it fluently. When we learn or talk about objects, we do it in English, even if we have to resort to very technical jargon. I think this is a case of opening it up to more people rather than making it more efficient for people who already do this.
amarant 3y ago
Oh my, now that is exciting! A friend of mine had implemented procedural rigging, and physics based generative animations used in some pretty big name games. Combine those 2 technologies with this, and you've seriously lowered the entry bar for video game production by a lot!
- mdre 3y ago
  Was this at Ubisoft maybe? I remember their presentations, some really amazing tech a decade ago... Never seen anything as straightforward to use in off-the-shelf software since.
  - amarant 3y ago
    It wasn't, my friend certainly wasn't the first to do something similar, this was much more recent. I was just impressed by a demo he showed me of his work. He did procedural rigging for Darktide 40k, and physics based animations for The Finals. Unfortunately I don't think his software is available outside of the companies that hired him.
mihaaly 3y ago
Am I the only one reluctant getting into lengthy discussions with a computer about making 3D models (or any other creation activity in fact) instead of doing it? Especially when I have specific ideas about certain details not that easy to put into words that both me and the receiving end (in this case computer) have the exact same (potentially cultural) understanding about. These panda and things are fun early demos for play but how far is this and more importantly, how far can it be reliable? How useful this can be eventually for serious matter where it really counts to be used? Perhaps there is a reason that practically no real engineering disciplines (work with 3D models professionally) rely on anecdotes and conversations, storytelling, going to the place of manufacture or construction telling with pretty words how to form the product but rely on technical drawings following specific rules that limit what and how is told and what people involved learn to make and read throughout years of training. If casual conversation was enough perhaps other than art and entertainment matters was not having their restricted language either (on top of special formatted documents). Make me a house... no, not that tall... a bit taller now... good... and a family home only... also it is ugly, make it look like in England... no, not modern England but old....not that old, from after Victorian era... ah, not the ones made for coal miners, those are ugly, make it detached and something a middle class family in Surrey would like... no, that's too big ... put it on the shelf darling... hey, I did not talk to you, do not put the house on a shelf! ... good, but I do not have that much land, make it narrow... and have American kitchen in it... no, not with American stove but an induction one... and do not start the stairs right at the entrance that is unpractical... have the bathroom in the ground floor... I don't care if it is not a proper English home anymore just do what I tell.... but not exactly how I tell but do it the way I want!....
- amarant 3y ago
  I think the best we can hope for, at least for the foreseeable future, is a workflow where you ask the AI to do a thing it gets it 80% right (someone who's good at writing prompts might even have it get a bit more correct) and then manually correct the things that it didn't get exactly like we wanted them. I think we're quite far from replacing skilled professionals entirely, but making them a lot more productive is within reach!

78 points by jfoster 3y ago

[dead]

bbor 3y ago

The media is claiming we’re in an AI Cold War with China, yet here’s a sinoamerican paper casually talking about automating the simple task of the entirety of human spatial understanding. Maybe we can all get along, after all? EDIT: is this paper… broken…? for anyone else? I cant read past the first two pages on phone or Mac, haven’t tried Linux. https://arxiv.org/pdf/2402.07207.pdf https://arxiv.org/pdf/2402.07207.pdf

ImHereToVote 3y ago
The US media is claiming. The CIA and Fake News in 1980s https://www.youtube.com/watch?v=XibCflWxZuA https://www.youtube.com/watch?v=XibCflWxZuA
- bbor 3y ago
  This is more accurate than I gave you initial credit for! I couldn’t find a single Chinese source of note from their government or media. Eg https://en.wikipedia.org/wiki/Artificial_Intelligence_Cold_War https://en.wikipedia.org/wiki/Artificial_Intelligence_Cold_W...
  - yorwba 3y ago
    Not hard to find Chinese sources if you know Chinese. http://old2022.bulletin.cas.cn/publish_article/2024/1/20240113.htm http://old2022.bulletin.cas.cn/publish_article/2024/1/202401... This one even has an English abstract.
ben_w 3y ago
Individuals have an easier time cooperating than their respective governments.
- bbor 3y ago
  Yeah, I guess I was alluding to the idea that the disagreements between the governments are not based on the actual concerns of the people. Tho it’s hard to tell at what level the vague nationalistic ideas of “we deserve to rule X land” and “they’re brainwashed” can be described as actual concerns, much less more openly racist and xenophobic ones.

Larok00 3y ago

I wish there was more progress in text-to-3D mesh for creating basic but very specific and functional shapes. With the last few years of progress, it really feels like it should be possible, but none of the big players are finding it worthwhile to look at. It would give the 3D printing community a massive boost.

psytrx 3y ago
Have you seen https://text-to-cad.zoo.dev/ https://text-to-cad.zoo.dev/ ? But yes, there's a lot of potential and, in my uneducated opinion, low hanging fruits for 3D printing especially.
yorwba 3y ago
When you want a very specific shape, text is probably not the right input modality. See also image generation, where to get very specific outputs, you're better off defining the large-scale structure spatially with a controlnet and only using text for the visual style and decorative details that do not need to be precisely controlled. What shapes would you ask a text-to-3D model to create for you?
- alpaca128 3y ago
  That makes me wonder whether an LLM for code generation could actually work for OpenSCAD-style designs as well, especially as it has a quite small set of functions. The only thing that makes me doubtful is that the coordinates have to correctly map to 3D space.
qwery 3y ago
Depends what "very specific and functional shapes" you have in mind, I guess. For my use of a 3D printer, I believe something like OpenSCAD is going to be a significantly more efficient textual description of the object than ~english -- for me and the computer.
- Larok00 3y ago
  OpenSCAD will definitely be more efficient but not as many people speak it fluently. When we learn or talk about objects, we do it in English, even if we have to resort to very technical jargon. I think this is a case of opening it up to more people rather than making it more efficient for people who already do this.

amarant 3y ago

Oh my, now that is exciting! A friend of mine had implemented procedural rigging, and physics based generative animations used in some pretty big name games. Combine those 2 technologies with this, and you've seriously lowered the entry bar for video game production by a lot!

mdre 3y ago
Was this at Ubisoft maybe? I remember their presentations, some really amazing tech a decade ago... Never seen anything as straightforward to use in off-the-shelf software since.
- amarant 3y ago
  It wasn't, my friend certainly wasn't the first to do something similar, this was much more recent. I was just impressed by a demo he showed me of his work. He did procedural rigging for Darktide 40k, and physics based animations for The Finals. Unfortunately I don't think his software is available outside of the companies that hired him.

mihaaly 3y ago

Am I the only one reluctant getting into lengthy discussions with a computer about making 3D models (or any other creation activity in fact) instead of doing it? Especially when I have specific ideas about certain details not that easy to put into words that both me and the receiving end (in this case computer) have the exact same (potentially cultural) understanding about. These panda and things are fun early demos for play but how far is this and more importantly, how far can it be reliable? How useful this can be eventually for serious matter where it really counts to be used? Perhaps there is a reason that practically no real engineering disciplines (work with 3D models professionally) rely on anecdotes and conversations, storytelling, going to the place of manufacture or construction telling with pretty words how to form the product but rely on technical drawings following specific rules that limit what and how is told and what people involved learn to make and read throughout years of training. If casual conversation was enough perhaps other than art and entertainment matters was not having their restricted language either (on top of special formatted documents). Make me a house... no, not that tall... a bit taller now... good... and a family home only... also it is ugly, make it look like in England... no, not modern England but old....not that old, from after Victorian era... ah, not the ones made for coal miners, those are ugly, make it detached and something a middle class family in Surrey would like... no, that's too big ... put it on the shelf darling... hey, I did not talk to you, do not put the house on a shelf! ... good, but I do not have that much land, make it narrow... and have American kitchen in it... no, not with American stove but an induction one... and do not start the stairs right at the entrance that is unpractical... have the bathroom in the ground floor... I don't care if it is not a proper English home anymore just do what I tell.... but not exactly how I tell but do it the way I want!....

amarant 3y ago
I think the best we can hope for, at least for the foreseeable future, is a workflow where you ask the AI to do a thing it gets it 80% right (someone who's good at writing prompts might even have it get a bit more correct) and then manually correct the things that it didn't get exactly like we wanted them. I think we're quite far from replacing skilled professionals entirely, but making them a lot more productive is within reach!