4 ms·

It doesn't feel like the AI is making meaningful decisions in this, so I'm a bit underwhelmed. It seems like they've created a new sculptural style based on a

by cwkoss 4y ago

It doesn't feel like the AI is making meaningful decisions in this, so I'm a bit underwhelmed. It seems like they've created a new sculptural style based on a system of constraints that yields an interesting new style, but I'm left wondering if all of those constraints are imposed by the humans: feels like the AI is being used as a tool to render the ideas of the humans rather than making interesting decisions itself. I tried to get chatGPT to write OpenSCAD a couple weeks ago. Made a good outline and structure for the code, but the spatial relationships were all wrong (somewhat unsurprisingly) I think taking a collection of 2d images and sculpting a plausible rigged 3d model is the direction of AI I'm more interested in. We have photogrammetry already, but what if the subjects are 'pokemon': and every image of them is from a different angle in a different pose - some features may be squishy, some rigid. The ability to take a character, extract a model and poses and propose additional poses would be incredible: such a tool could let a kid who learns how to draw create a character and drop it into a video game, and use it.

knicholes 4y ago
So this, then? https://www.youtube.com/watch?v=shy51E-MU8Y https://www.youtube.com/watch?v=shy51E-MU8Y (not rigging, but maybe you're talking about adding Mixamo to it?)
- joewhatkins 4y ago
  From my understanding the model in the linked video only stylizes existing meshes from text. There’s plenty of papers that have tried text -> 3D model generation using photogrammetric-esque methods similar to what the parent comment suggested - the two minute papers video on one example is here. https://youtu.be/L3G0dx1Q0R8 https://youtu.be/L3G0dx1Q0R8 Outside of cherry-picked examples this style of model tends to suffer from what people are calling the “Janus Problem” - the easiest way for it to satisfy the loss is to simply make the object look like the input prompt from as many angles as possible. So if you enter “a rubber duck”, it tends to generate yellow blobs with multiple head-like appendages sprouting off from it. Google’s paper that tried this approach using Imagen as the text->image generator had great results, but they might be cherry-picked. Someone replicated it with stable diffusion as the text->image backend - still major Janus problems.

by cwkoss 4y ago

knicholes 4y ago

So this, then? https://www.youtube.com/watch?v=shy51E-MU8Y https://www.youtube.com/watch?v=shy51E-MU8Y (not rigging, but maybe you're talking about adding Mixamo to it?)

joewhatkins 4y ago
From my understanding the model in the linked video only stylizes existing meshes from text. There’s plenty of papers that have tried text -> 3D model generation using photogrammetric-esque methods similar to what the parent comment suggested - the two minute papers video on one example is here. https://youtu.be/L3G0dx1Q0R8 https://youtu.be/L3G0dx1Q0R8 Outside of cherry-picked examples this style of model tends to suffer from what people are calling the “Janus Problem” - the easiest way for it to satisfy the loss is to simply make the object look like the input prompt from as many angles as possible. So if you enter “a rubber duck”, it tends to generate yellow blobs with multiple head-like appendages sprouting off from it. Google’s paper that tried this approach using Imagen as the text->image generator had great results, but they might be cherry-picked. Someone replicated it with stable diffusion as the text->image backend - still major Janus problems.