3 ms·

Whenever I see these text to video diffusion models, I always think that a much better result would come out of some sort of multi-modal vision-aware AI agent w

Whenever I see these text to video diffusion models, I always think that a much better result would come out of some sort of multi-modal vision-aware AI agent were able to use 3d modeling tools like blender, continuously iterate it, work with physics and particle simulation tools then refine the less realistic details with a diffusion model based filter. It would also allow all the assets created along the way to be reused and refined. I imagine it will be the de-facto way AI generates video in the next few years, once huge models are smart enough to use tools and work in high breadth and depth jobs like humans do.

red75prime 4y ago
I concur. Throwing in more and more compute to make digital dreams more and more realistic doesn't seem to be an efficient strategy for video generation where you need to keep lots of out-of-the-picture context in memory.
jiggawatts 4y ago
This actually exists, already... sort-of. A major topic of research is to make differentiable renderers, which can then have their parameters tuned via gradient descent, much like how AIs are trained. The problem is that most renderers are very discrete, and hence aren't differentiable. For the same reason that reverse rendering is difficult on such systems, it's hard to train an AI to use them also. Having said that, I can imagine it happening. Look at Google's Alpha Star and related AIs, which can play games with discrete events at grandmaster level. You could train an AI to operate a 3D design program similarly. The input training could be recordings of actions taken by human designers, or simply train the AI by making it match target images. Start out simple, and then crank up the complexity over time.
- Maken 4y ago
  Differentiable renderers have started to volumetric representations and distance fields to represent geometry, which kinda solves the discretization of the scene.
Closi 4y ago
The same wasn't true for images - it has been easier (so far) to directly compute the images than it has been to train them to use photoshop or illustrator. Similarly, neural networks have already been shown to be able to directly simulate physics simulations (smoke, particles) with suprising accuracy. I agree that logically your statement makes sense, however I'm not sure it matches what has been successful in the space so far. Tools may make sense for these NN's to use, however they probably won't look anything like the tools we use today.
- gitfan86 4y ago
  The future is stacking models in feedback loops. One model could detect bad anatomy and tell the other models to rerender the frame. AutoGTP is an example of this
  - barking_biscuit 4y ago
    I've definitely been seeing things trending in this direction.
- al_be_back 4y ago
  >> it has been easier (so far) to directly compute the images than it has been to train them to use photoshop or illustrator. possible because training models are fed by an abundance of existing human-generated content, which is ever increasing in quality, dirt-cheap, and easily accessible; which doesn't make AI Generated Content disruptive in this respect (my view). not sure what you mean by "easier", since computers are better at following rules and people better at reasoning. >> I agree that logically your statement makes sense ... Isn't that the best case for AI generated content, logical-sense? or are you suggesting more (e.g 'source of meaning')?
  - Closi 4y ago
    > Isn't that the best case for AI generated content, logical-sense? or are you suggesting more (e.g 'source of meaning')? I just mean that the approach sounds logical, although the best approach isn’t always the one that appears the most logical. This appears particularly true in AI - for instance diffusion models feel more like an incredibly surprising discovery rather than an approach that’s immediately obvious by applying logic. > Not sure what you mean by "easier", since computers are better at following rules and people better at reasoning. By easier I just mean we have had better results so far. Maybe easier wasn't the right word. On your second point - computers will be better at most reasoning tasks shortly.

red75prime 4y ago

I concur. Throwing in more and more compute to make digital dreams more and more realistic doesn't seem to be an efficient strategy for video generation where you need to keep lots of out-of-the-picture context in memory.

jiggawatts 4y ago

This actually exists, already... sort-of. A major topic of research is to make differentiable renderers, which can then have their parameters tuned via gradient descent, much like how AIs are trained. The problem is that most renderers are very discrete, and hence aren't differentiable. For the same reason that reverse rendering is difficult on such systems, it's hard to train an AI to use them also. Having said that, I can imagine it happening. Look at Google's Alpha Star and related AIs, which can play games with discrete events at grandmaster level. You could train an AI to operate a 3D design program similarly. The input training could be recordings of actions taken by human designers, or simply train the AI by making it match target images. Start out simple, and then crank up the complexity over time.

Maken 4y ago
Differentiable renderers have started to volumetric representations and distance fields to represent geometry, which kinda solves the discretization of the scene.

Closi 4y ago

The same wasn't true for images - it has been easier (so far) to directly compute the images than it has been to train them to use photoshop or illustrator. Similarly, neural networks have already been shown to be able to directly simulate physics simulations (smoke, particles) with suprising accuracy. I agree that logically your statement makes sense, however I'm not sure it matches what has been successful in the space so far. Tools may make sense for these NN's to use, however they probably won't look anything like the tools we use today.

gitfan86 4y ago
The future is stacking models in feedback loops. One model could detect bad anatomy and tell the other models to rerender the frame. AutoGTP is an example of this
- barking_biscuit 4y ago
  I've definitely been seeing things trending in this direction.
al_be_back 4y ago
>> it has been easier (so far) to directly compute the images than it has been to train them to use photoshop or illustrator. possible because training models are fed by an abundance of existing human-generated content, which is ever increasing in quality, dirt-cheap, and easily accessible; which doesn't make AI Generated Content disruptive in this respect (my view). not sure what you mean by "easier", since computers are better at following rules and people better at reasoning. >> I agree that logically your statement makes sense ... Isn't that the best case for AI generated content, logical-sense? or are you suggesting more (e.g 'source of meaning')?
- Closi 4y ago
  > Isn't that the best case for AI generated content, logical-sense? or are you suggesting more (e.g 'source of meaning')? I just mean that the approach sounds logical, although the best approach isn’t always the one that appears the most logical. This appears particularly true in AI - for instance diffusion models feel more like an incredibly surprising discovery rather than an approach that’s immediately obvious by applying logic. > Not sure what you mean by "easier", since computers are better at following rules and people better at reasoning. By easier I just mean we have had better results so far. Maybe easier wasn't the right word. On your second point - computers will be better at most reasoning tasks shortly.