9 ms·

> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close.

> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. I feel like we're close too, but for another reason. For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer can immediately spot that. However I'm willing to bet that we'll soon have something much better: you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. And the scene shall be sent into Blender and you'll click on a button and have an actual rendering made by Blender, with correct lighting. Wanna move that bicycle? Move it in the 3D scene exactly where you want. That is coming. And for audio it's the same: why generate an audio file when soon models shall be able to generate the various tracks, with all the instruments and whatnots, allowing to create the audio file? That is coming too.

p1esk 3y ago
Are you working on all that?
- cptaj 3y ago
  Probably not. But there does seem to be a clear path to it. The main issue is going to be having the right dataset. You basically need to record user actions in something like blender (ie: moving a model of a bike to the left of a scene), match it to a text description of the action (ie; "move bike to the left") and match those to before/after snapshots of the resulting file format. You need a whole metric fuckton of these. After that, you train your model to produce those 3d scene files instead of image bitmaps. You can do this for a lot of other tasks. These general purpose models can learn anything that you can usefully represent in data. I can imagine AGI being, at least in part, a large set of these purpose trained models. Heck, maybe our brains work this way. When we learn to throw a ball, we train a model in a subset of our brain to do just this and then this model is called on by our general consciousness when needed. Sorry, I'm just rambling here but its very exciting stuff.
  - sterlind 3y ago
    The hard part of AGI is the self-training and few examples. Your parents didn't attach strings to your body and puppeteer you through a few hundred thousand games of baseball. And the humans that invented baseball had zero training data to go on.
    - p1esk 3y ago
      Your body is a result of a billion year old evolutionary optimization process. GPT-4 was trained from scratch in a few months.
  - filipezf 3y ago
    I have for some time planning to do a 'Wikipedia for AI' (even bought a domain), where people could contribute all sorts of these skills ( not only 3d video, but also manual skills, or anything). Given the current climate of 'AI will save/doom us', and that users would in some sense be training their own replacements, I don't know how much love such site would have, though.
atentaten 3y ago
Whats your reasoning for feeling that we're close?
- cptaj 3y ago
  We do it for text, audio and bitmapped images. A 3D scene file format is no different, you could train a model to output a blender file format instead of a bitmap. It can learn anything you have data for. Heck, we do it with geospatial data already, generating segmentation vectors. Why not 3D?
  - boppo1 3y ago
    >3D scene file format is no different Not in theory, but the level of complexity is way higher and the amount of data available is much smaller. Compare bitmaps to this: https://fossies.org/linux/blender/doc/blender_file_format/mystery_of_the_blend.html https://fossies.org/linux/blender/doc/blender_file_format/my...
    - kaibee 3y ago
      Also the level of fault tolerance... if your pixels are a bit blurry, chances are no one notices at a high enough resolution. If your json is a bit blurry you have problems.
      - astrange 3y ago
        You can do "constrained decoding" on a code model which keeps it grammatically correct. But we haven't gotten diffusion working well for text/code, so generating long files is a problem.
        DougBTX 3y ago
        Recent results for code diffusion here: https://www.microsoft.com/en-us/research/publication/codefusion-a-pre-trained-diffusion-model-for-code-generation/ https://www.microsoft.com/en-us/research/publication/codefus... I'm not experienced enough to validate their claims, but I love the choice of languages to evaluate on: > Python, Bash and Excel conditional formatting rules.
bob1029 3y ago
> However I'm willing to bet that we'll soon have something much better: you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I agree with this philosophy - Teach the AI to work with the same tools the human does. We already have a lot of human experts to refer to. Training material is everywhere. There isn't a "text-to-video" expert we can query to help us refine the capabilities around SD. It's a one-shot, Jupiter-scale model with incomprehensible inertia. Contrast this with an expert-tuned model (i.e. natural language instructions) that can be nuanced precisely and to the the point of imperceptibility with a single sentence. The other cool thing about the "use existing tools" path is that if the AI fails part way through, it's actually possible for a human operator to step in and attempt recovery.
epr 3y ago
> you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I'm always confused why I don't hear more about projects going in this direction. Controlnets are great, but there's still quite a lot of hallucination and other tiny mistakes that a skilled human would never make.
- boppo1 3y ago
  Blender files are dramatically more complex than any image format, which are basically all just 2D arrays of 3-value vectors. The blender filetype uses a weird DNA/RNA struct system that would probably require its own training run. More on the Blender file format: https://fossies.org/linux/blender/doc/blender_file_format/mystery_of_the_blend.html https://fossies.org/linux/blender/doc/blender_file_format/my...
  - mikepurvis 3y ago
    But surely you wouldn't try to emit that format directly, but rather some higher level scene description? Or even just a set of instructions for how to manipulate the UI to create the imagined scene?
    - BirdieNZ 3y ago
      I've seen this but producing Python scripts that you run in Blender, e.g. https://www.youtube.com/watch?v=x60zHw_z4NM https://www.youtube.com/watch?v=x60zHw_z4NM (but I saw something marginally more impressive, not sure where though!)
      - bsenftner 3y ago
        My god that is an irritating video style, "AI woweee!"
    - deleted 3y ago
      [deleted]
    - mikebelanger 3y ago
a_bouncing_bean 3y ago
Thanks! this is exactly what I have been thinking, only you've expressed it much more eloquently than I would be able.
internet101010 3y ago
I am guessing it will be similar to inpainting in normal stable diffusion, which is easy when using the workflow feature InvokeAI ui.
coldtea 3y ago
>For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer can immediately spot that. The question is whether the 99% of the audience would even care...
- COAGULOPATH 3y ago
  Of course they would. The internet spent a solid month laughing at the Sonic the Hedgehog movie because Sonic had weird-looking teeth.
  - ekianjo 3y ago
    No they laughed at it because it looked awful in every single way
  - coldtea 3y ago
    Since that movie did well and spawned 2 sequels, the real conclusion is that the viewers didn't really care. As for "the internet", there will always some small part of it which will obsess and/or laught over anything, doesn't mean they represent anything significant - not even when they're vocal.
    - PawgerZ 3y ago
      Viewers did care: the teeth got changed before the movie was released. And, I don't know if you missed it, but it wasn't just one niche of the internet commenting on his teeth. The "outrage" went mainstream; even dentists were making hit-pieces on Sonic's teeth. I'm not gonna lie, it was amazing marketing for the movie, intentional or not.
sheepscreek 3y ago
Excellent point. Perhaps a more computationally expensive but better looking method will be to pull all objects in the scene from a 3D model library, then programmatically set the scene and render it.
Kuinox 3y ago
This isn't coming, it's already here. https://github.com/gsgen3d/gsgen https://github.com/gsgen3d/gsgen Yes, it's just 3D models for now, but it can do whole scenes generations, it's just not great yet at it. The tech is there but just need to improve.
solarkraft 3y ago
Where is the training data coming from?
Veraslane 3y ago
[dead]
jwoodbridge 3y ago
we're working on this if you want to give it a try - dream3d.com
- hackerlight 3y ago
  You should put a demo on the landing page
  - jwoodbridge 3y ago
    just redid the ux and making a new one, but here's a quick example: https://www.loom.com/share/fa84ba92d7144179ac17ece9bf7fbd99 https://www.loom.com/share/fa84ba92d7144179ac17ece9bf7fbd99
wruza 3y ago
Not that I’m against the described 3d way, but personally I don’t care about light and shadows until it’s so bad that I do. This obsession with realism is irrational in video games. In real life people don’t understand why light works like this or like that. We just accept it. And if you ask someone to paint how it should work, the result is rarely physical but acceptable. It literally doesn’t matter until it’s very bad.
btbuildem 3y ago
That indeed sounds like a very plausible solution -- working with AI on the level of scene definitions, model geometries etc. However, 3D is just one approach to rendering visuals. There are so many other styles and methods how people create images, and if I understand correctly, we can do image-to-text to analyze image content, as well as text-to-image to generate it - regardless of the orginal method (3d render or paintbrush or camera lens). There are some "fuzzy primitives" in the layers there that translate to the visual elements. I'm hoping we see "editors" that let us manipulate / edit / iterate over generated images in terms of those.
whywhywhywhy 3y ago
Nah I disagree, this feels like a glorification of the process not the end result. Just because having the 3D model in the scene with all the lighting makes the end result feel more solid to you because you feel you can see the work that's going into it. In the end diffusion technology can make a more realistic image faster than a rendering engine can. I feel pretty strongly that this pipeline will be the foundation for most of the next decade of graphics and making things by hand in 3D will become extremely niche because lets face it anyone who has worked in 3D it's tedious, it's time consuming, takes large teams and it's not even well paid. The future is just tools that give us better controls and every frame will be coming from latent space not simulated photons. I say this as someone who had done 3D professionally in the past.
- pegasus 3y ago
  Nah, I agree with GP. Who didn't suggest making 3D scenes by hand, but the opposite: create those 3D scenes using the generative method, use ray-tracing or the like to render the image. Maybe have another pass through a model to apply any touch-ups to make it more gritty and less artificial. This way things can stay consistent and sane, avoiding all those flaws which are so easy to spot today.
  - whywhywhywhy 3y ago
    I know exactly what OP suggested but why are you both glorifying the fact there is a 3D scene graph made in the middle and then slower rendering at the end when the tech can just go from the first thing to a better finished thing?
    - pegasus 3y ago
      Because it just can't. And it won't. It can't even reliably produce consistent shadows in a still image, so when we talk video with a moving camera, all bets are off. To create flawless movie simulations through a dynamic and rich 3D world, requires an ability of internally represent that scene with a level of accuracy which is beyond what we can hope generative models to achieve, even with the gargantuan amount of GPU-power behind ChatGPT, for example. ChatGPT, may I remind you, can't even properly simulate large-ish multiplications. I think you may need to slightly recalibrate your expectations for generative tech here.

p1esk 3y ago

Are you working on all that?

cptaj 3y ago
Probably not. But there does seem to be a clear path to it. The main issue is going to be having the right dataset. You basically need to record user actions in something like blender (ie: moving a model of a bike to the left of a scene), match it to a text description of the action (ie; "move bike to the left") and match those to before/after snapshots of the resulting file format. You need a whole metric fuckton of these. After that, you train your model to produce those 3d scene files instead of image bitmaps. You can do this for a lot of other tasks. These general purpose models can learn anything that you can usefully represent in data. I can imagine AGI being, at least in part, a large set of these purpose trained models. Heck, maybe our brains work this way. When we learn to throw a ball, we train a model in a subset of our brain to do just this and then this model is called on by our general consciousness when needed. Sorry, I'm just rambling here but its very exciting stuff.
- sterlind 3y ago
  The hard part of AGI is the self-training and few examples. Your parents didn't attach strings to your body and puppeteer you through a few hundred thousand games of baseball. And the humans that invented baseball had zero training data to go on.
  - p1esk 3y ago
    Your body is a result of a billion year old evolutionary optimization process. GPT-4 was trained from scratch in a few months.
- filipezf 3y ago
  I have for some time planning to do a 'Wikipedia for AI' (even bought a domain), where people could contribute all sorts of these skills ( not only 3d video, but also manual skills, or anything). Given the current climate of 'AI will save/doom us', and that users would in some sense be training their own replacements, I don't know how much love such site would have, though.

atentaten 3y ago

Whats your reasoning for feeling that we're close?

cptaj 3y ago
We do it for text, audio and bitmapped images. A 3D scene file format is no different, you could train a model to output a blender file format instead of a bitmap. It can learn anything you have data for. Heck, we do it with geospatial data already, generating segmentation vectors. Why not 3D?
- boppo1 3y ago
  >3D scene file format is no different Not in theory, but the level of complexity is way higher and the amount of data available is much smaller. Compare bitmaps to this: https://fossies.org/linux/blender/doc/blender_file_format/mystery_of_the_blend.html https://fossies.org/linux/blender/doc/blender_file_format/my...
  - kaibee 3y ago
    Also the level of fault tolerance... if your pixels are a bit blurry, chances are no one notices at a high enough resolution. If your json is a bit blurry you have problems.
    - astrange 3y ago
      You can do "constrained decoding" on a code model which keeps it grammatically correct. But we haven't gotten diffusion working well for text/code, so generating long files is a problem.
      - DougBTX 3y ago
        Recent results for code diffusion here: https://www.microsoft.com/en-us/research/publication/codefusion-a-pre-trained-diffusion-model-for-code-generation/ https://www.microsoft.com/en-us/research/publication/codefus... I'm not experienced enough to validate their claims, but I love the choice of languages to evaluate on: > Python, Bash and Excel conditional formatting rules.

bob1029 3y ago

> However I'm willing to bet that we'll soon have something much better: you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I agree with this philosophy - Teach the AI to work with the same tools the human does. We already have a lot of human experts to refer to. Training material is everywhere. There isn't a "text-to-video" expert we can query to help us refine the capabilities around SD. It's a one-shot, Jupiter-scale model with incomprehensible inertia. Contrast this with an expert-tuned model (i.e. natural language instructions) that can be nuanced precisely and to the the point of imperceptibility with a single sentence. The other cool thing about the "use existing tools" path is that if the AI fails part way through, it's actually possible for a human operator to step in and attempt recovery.

epr 3y ago

> you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I'm always confused why I don't hear more about projects going in this direction. Controlnets are great, but there's still quite a lot of hallucination and other tiny mistakes that a skilled human would never make.

boppo1 3y ago
Blender files are dramatically more complex than any image format, which are basically all just 2D arrays of 3-value vectors. The blender filetype uses a weird DNA/RNA struct system that would probably require its own training run. More on the Blender file format: https://fossies.org/linux/blender/doc/blender_file_format/mystery_of_the_blend.html https://fossies.org/linux/blender/doc/blender_file_format/my...
- mikepurvis 3y ago
  But surely you wouldn't try to emit that format directly, but rather some higher level scene description? Or even just a set of instructions for how to manipulate the UI to create the imagined scene?
  - BirdieNZ 3y ago
    I've seen this but producing Python scripts that you run in Blender, e.g. https://www.youtube.com/watch?v=x60zHw_z4NM https://www.youtube.com/watch?v=x60zHw_z4NM (but I saw something marginally more impressive, not sure where though!)
    - bsenftner 3y ago
      My god that is an irritating video style, "AI woweee!"
  - deleted 3y ago
    [deleted]
  - mikebelanger 3y ago

a_bouncing_bean 3y ago

Thanks! this is exactly what I have been thinking, only you've expressed it much more eloquently than I would be able.

internet101010 3y ago

I am guessing it will be similar to inpainting in normal stable diffusion, which is easy when using the workflow feature InvokeAI ui.

coldtea 3y ago

>For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer can immediately spot that. The question is whether the 99% of the audience would even care...

COAGULOPATH 3y ago
Of course they would. The internet spent a solid month laughing at the Sonic the Hedgehog movie because Sonic had weird-looking teeth.
- ekianjo 3y ago
  No they laughed at it because it looked awful in every single way
- coldtea 3y ago
  Since that movie did well and spawned 2 sequels, the real conclusion is that the viewers didn't really care. As for "the internet", there will always some small part of it which will obsess and/or laught over anything, doesn't mean they represent anything significant - not even when they're vocal.
  - PawgerZ 3y ago
    Viewers did care: the teeth got changed before the movie was released. And, I don't know if you missed it, but it wasn't just one niche of the internet commenting on his teeth. The "outrage" went mainstream; even dentists were making hit-pieces on Sonic's teeth. I'm not gonna lie, it was amazing marketing for the movie, intentional or not.

sheepscreek 3y ago

Excellent point. Perhaps a more computationally expensive but better looking method will be to pull all objects in the scene from a 3D model library, then programmatically set the scene and render it.

Kuinox 3y ago

This isn't coming, it's already here. https://github.com/gsgen3d/gsgen https://github.com/gsgen3d/gsgen Yes, it's just 3D models for now, but it can do whole scenes generations, it's just not great yet at it. The tech is there but just need to improve.

solarkraft 3y ago

Where is the training data coming from?

Veraslane 3y ago

[dead]

jwoodbridge 3y ago

we're working on this if you want to give it a try - dream3d.com

hackerlight 3y ago
You should put a demo on the landing page
- jwoodbridge 3y ago
  just redid the ux and making a new one, but here's a quick example: https://www.loom.com/share/fa84ba92d7144179ac17ece9bf7fbd99 https://www.loom.com/share/fa84ba92d7144179ac17ece9bf7fbd99

wruza 3y ago

Not that I’m against the described 3d way, but personally I don’t care about light and shadows until it’s so bad that I do. This obsession with realism is irrational in video games. In real life people don’t understand why light works like this or like that. We just accept it. And if you ask someone to paint how it should work, the result is rarely physical but acceptable. It literally doesn’t matter until it’s very bad.

btbuildem 3y ago

That indeed sounds like a very plausible solution -- working with AI on the level of scene definitions, model geometries etc. However, 3D is just one approach to rendering visuals. There are so many other styles and methods how people create images, and if I understand correctly, we can do image-to-text to analyze image content, as well as text-to-image to generate it - regardless of the orginal method (3d render or paintbrush or camera lens). There are some "fuzzy primitives" in the layers there that translate to the visual elements. I'm hoping we see "editors" that let us manipulate / edit / iterate over generated images in terms of those.

whywhywhywhy 3y ago

Nah I disagree, this feels like a glorification of the process not the end result. Just because having the 3D model in the scene with all the lighting makes the end result feel more solid to you because you feel you can see the work that's going into it. In the end diffusion technology can make a more realistic image faster than a rendering engine can. I feel pretty strongly that this pipeline will be the foundation for most of the next decade of graphics and making things by hand in 3D will become extremely niche because lets face it anyone who has worked in 3D it's tedious, it's time consuming, takes large teams and it's not even well paid. The future is just tools that give us better controls and every frame will be coming from latent space not simulated photons. I say this as someone who had done 3D professionally in the past.

pegasus 3y ago
Nah, I agree with GP. Who didn't suggest making 3D scenes by hand, but the opposite: create those 3D scenes using the generative method, use ray-tracing or the like to render the image. Maybe have another pass through a model to apply any touch-ups to make it more gritty and less artificial. This way things can stay consistent and sane, avoiding all those flaws which are so easy to spot today.
- whywhywhywhy 3y ago
  I know exactly what OP suggested but why are you both glorifying the fact there is a 3D scene graph made in the middle and then slower rendering at the end when the tech can just go from the first thing to a better finished thing?
  - pegasus 3y ago
    Because it just can't. And it won't. It can't even reliably produce consistent shadows in a still image, so when we talk video with a moving camera, all bets are off. To create flawless movie simulations through a dynamic and rich 3D world, requires an ability of internally represent that scene with a level of accuracy which is beyond what we can hope generative models to achieve, even with the gargantuan amount of GPU-power behind ChatGPT, for example. ChatGPT, may I remind you, can't even properly simulate large-ish multiplications. I think you may need to slightly recalibrate your expectations for generative tech here.