6 ms·

Yeah but... it's good enough? There were movies with horrible VFX that still sold perfectly well at the time.

jsheard 2y ago
An important contrast is that early VFX offered strong control with weak fidelity, and these prompt-based AI systems offer high fidelity with weak control. Intent matters if you want to make something more than a tech demo or throwaway B-roll and you can't communicate much intent in a 30 word prompt, assuming the model even follows the prompt accurately.
- Ajedi32 2y ago
  Just need to wait for someone to develop a version of ControlNet that works with this system.
- marcosdumay 2y ago
  This is such an important problem of the entire genAI idea. It's absurd that people keep focusing on quality instead of talking about it. But then, a lot of people have financial reasons to ignore the problem. What's too bad, because it's hindering the creation of stuff that are actually useful.
- dageshi 2y ago
  Yeah, that's a fair point.
- blueblisters 2y ago
  Yeah controlnet-style conditioning doesn't solve for consistent assets, or lighting, framing etc. Maybe its early but it seems hard to get around traditional 3D assets + rendering, at least for serious use-cases. These models do seem like they could be great photorealism/stylization shaders. And they are also pretty good at stuff like realistic explosions, fluid renders etc. That stuff is really hard with CG.
- mrandish 2y ago
  > AI systems offer high fidelity with weak control You are spot on. I've been involved in creating technologies used by film and video creators for decades, so I have some understanding of what would be useful to them. The best video AIs I've seen only seem capable of replacing some stock video clip creation because, so far, I haven't seen any ability to maintain robust consistency from shot to shot and scene to scene. There's also no granular control other than the text prompt. At first glance, these demos are very impressive but when you try to map the capability shown to a real production workflow for a movie, TV show or commercial, they're not even close because they aren't even trying to solve the problem. To be clear, I think it's probably possible to create a video AI that would be truly useful in a real production workflow, it's just that I haven't seen anything working in that direction yet.
  - chefandy 2y ago
    > You are spot on. I've been involved in creating technologies used by film and video creators for decades, so I have some understanding of what would be useful to them. The best video AIs I've seen only seem capable of replacing some stock video clip creation because, so far, I haven't seen any ability to maintain robust consistency from shot to shot and scene to scene. There's also no granular control other than the text prompt. At first glance, these demos are very impressive but when you try to map the capability shown to a real production workflow for a movie, TV show or commercial, they're not even close because they aren't even trying to solve the problem. Yeah it's really hard to get across to a lot of folks that are really amped up about these tools that what they're focused on refining is not getting them any closer to their imagined goal in most professional workflows. This will be great right off the bat for what most developers would need images for-- making a hero image for a blog post, making a blurb of video for a background, or a joke, or making assets for their video game that would never cut it for a non-cheapo commercial project but are better than what they'd have been able to cobble together themselves. But those workflows are fundamentally so different from the very first steps in the process. It's a larger-scale version of trying to explain to no-compromise FOSS zealots 20 years ago that Gimp was nowhere near able to replace Photoshop in a professional toolkit because they're completely disinterested in taking feedback about professional use cases, and that being able to write your own filters in Perl doesn't really help graphic designers-- well 20 years later, the gap is as wide as it ever has been, and there are even more people, almost exclusively FOSS nerds with to professional visual work experience, that insist it's better. That said, it's nearly as hard to get this across to ADs who are like "what do you mean this shot is going to take you 3 days? I just made these stills which are like 70% there in midjourney in 10 minutes." > To be clear, I think it's probably possible to create a video AI that would be truly useful in a real production workflow, it's just that I haven't seen anything working in that direction yet. I think that neural networks, generally, are already fantastically useful in tools like Nuke's Copycat node. Nobody misses masking frame-by-frame if they don't have to do it. But prompt-based tools? Nah. If even 200 words in a prompt was enough information to convey work that needed to be done, why do creative workflows need so many revisions and why are there so many meetings with sketches and mood boards and concept art among career professionals? Text prompts are great for people that are working with a medium they don't really know how to create in because the real artistic decisions are already made by the artists whose artwork was ingested into the models. If you don't understand that level of nuance, you don't see how unbelievably consequential it is to the final product, and not having granular control of it seems nearly inconsequential. Most professionals look at it and see a toy because they know it will never be capable of making what they want it to make.

An important contrast is that early VFX offered strong control with weak fidelity, and these prompt-based AI systems offer high fidelity with weak control. Intent matters if you want to make something more than a tech demo or throwaway B-roll and you can't communicate much intent in a 30 word prompt, assuming the model even follows the prompt accurately.

Ajedi32 2y ago
Just need to wait for someone to develop a version of ControlNet that works with this system.
marcosdumay 2y ago
This is such an important problem of the entire genAI idea. It's absurd that people keep focusing on quality instead of talking about it. But then, a lot of people have financial reasons to ignore the problem. What's too bad, because it's hindering the creation of stuff that are actually useful.
dageshi 2y ago
Yeah, that's a fair point.
blueblisters 2y ago
Yeah controlnet-style conditioning doesn't solve for consistent assets, or lighting, framing etc. Maybe its early but it seems hard to get around traditional 3D assets + rendering, at least for serious use-cases. These models do seem like they could be great photorealism/stylization shaders. And they are also pretty good at stuff like realistic explosions, fluid renders etc. That stuff is really hard with CG.
mrandish 2y ago
> AI systems offer high fidelity with weak control You are spot on. I've been involved in creating technologies used by film and video creators for decades, so I have some understanding of what would be useful to them. The best video AIs I've seen only seem capable of replacing some stock video clip creation because, so far, I haven't seen any ability to maintain robust consistency from shot to shot and scene to scene. There's also no granular control other than the text prompt. At first glance, these demos are very impressive but when you try to map the capability shown to a real production workflow for a movie, TV show or commercial, they're not even close because they aren't even trying to solve the problem. To be clear, I think it's probably possible to create a video AI that would be truly useful in a real production workflow, it's just that I haven't seen anything working in that direction yet.
- chefandy 2y ago
  > You are spot on. I've been involved in creating technologies used by film and video creators for decades, so I have some understanding of what would be useful to them. The best video AIs I've seen only seem capable of replacing some stock video clip creation because, so far, I haven't seen any ability to maintain robust consistency from shot to shot and scene to scene. There's also no granular control other than the text prompt. At first glance, these demos are very impressive but when you try to map the capability shown to a real production workflow for a movie, TV show or commercial, they're not even close because they aren't even trying to solve the problem. Yeah it's really hard to get across to a lot of folks that are really amped up about these tools that what they're focused on refining is not getting them any closer to their imagined goal in most professional workflows. This will be great right off the bat for what most developers would need images for-- making a hero image for a blog post, making a blurb of video for a background, or a joke, or making assets for their video game that would never cut it for a non-cheapo commercial project but are better than what they'd have been able to cobble together themselves. But those workflows are fundamentally so different from the very first steps in the process. It's a larger-scale version of trying to explain to no-compromise FOSS zealots 20 years ago that Gimp was nowhere near able to replace Photoshop in a professional toolkit because they're completely disinterested in taking feedback about professional use cases, and that being able to write your own filters in Perl doesn't really help graphic designers-- well 20 years later, the gap is as wide as it ever has been, and there are even more people, almost exclusively FOSS nerds with to professional visual work experience, that insist it's better. That said, it's nearly as hard to get this across to ADs who are like "what do you mean this shot is going to take you 3 days? I just made these stills which are like 70% there in midjourney in 10 minutes." > To be clear, I think it's probably possible to create a video AI that would be truly useful in a real production workflow, it's just that I haven't seen anything working in that direction yet. I think that neural networks, generally, are already fantastically useful in tools like Nuke's Copycat node. Nobody misses masking frame-by-frame if they don't have to do it. But prompt-based tools? Nah. If even 200 words in a prompt was enough information to convey work that needed to be done, why do creative workflows need so many revisions and why are there so many meetings with sketches and mood boards and concept art among career professionals? Text prompts are great for people that are working with a medium they don't really know how to create in because the real artistic decisions are already made by the artists whose artwork was ingested into the models. If you don't understand that level of nuance, you don't see how unbelievably consequential it is to the final product, and not having granular control of it seems nearly inconsequential. Most professionals look at it and see a toy because they know it will never be capable of making what they want it to make.