4 ms·

Are any image / video generation tools giving just the output or the layers, timelines, transitions, audio as things to work with in our old fashioned toolsets?

Are any image / video generation tools giving just the output or the layers, timelines, transitions, audio as things to work with in our old fashioned toolsets? The problem: In my limited playing of these tools they don't quite make the mark and I would easily be able to tweak something if I had all the layers used. I imagine in the future products could be used to tweak this to match what I think the output should be.... At least the code generation tools are providing source code. Imagine them only giving compiled bytecode.

swatcoder 2y ago
Keep in mind that these technologies produce more stuff like what they've been trained on, and they need tremendous amounts of training data to pull that off. It so happens that there are innumerable samples of prose and source code and rendered songs and videos and images to use as this training data. But that's not so much the case for professional workflows (outside of software development). If the tools can evolve to generating usefully detailed and coherent media projects instead of just perceptually convincing media assets, it's going to be a while before they get there.
botanical76 2y ago
They definitely do not give you an Adobe After Effects project. This is because of the way they are trained. I suspect a vast proportion of its training data is not annotated with the corresponding layers, timelines, etc so the model is unable to reproduce it like that. You basically just get video AFAIK.
vunderba 2y ago
If you have experience as a graphic designer, you can get very far with any layer based graphic tools like Krita or Affinity in conjunction with proper inpainting against generative image models - in fact that's InvokeAI's entire target user base.
woodson 2y ago
There are some approaches that use an LLM to generate “scripts” (you can think of them as a DSL) for composing/arranging media, essentially driving other models to generate parts of the media. One example is WavJourney: https://audio-agi.github.io/WavJourney_demopage/ https://audio-agi.github.io/WavJourney_demopage/

swatcoder 2y ago

Keep in mind that these technologies produce more stuff like what they've been trained on, and they need tremendous amounts of training data to pull that off. It so happens that there are innumerable samples of prose and source code and rendered songs and videos and images to use as this training data. But that's not so much the case for professional workflows (outside of software development). If the tools can evolve to generating usefully detailed and coherent media projects instead of just perceptually convincing media assets, it's going to be a while before they get there.

botanical76 2y ago

They definitely do not give you an Adobe After Effects project. This is because of the way they are trained. I suspect a vast proportion of its training data is not annotated with the corresponding layers, timelines, etc so the model is unable to reproduce it like that. You basically just get video AFAIK.

vunderba 2y ago

If you have experience as a graphic designer, you can get very far with any layer based graphic tools like Krita or Affinity in conjunction with proper inpainting against generative image models - in fact that's InvokeAI's entire target user base.

woodson 2y ago

There are some approaches that use an LLM to generate “scripts” (you can think of them as a DSL) for composing/arranging media, essentially driving other models to generate parts of the media. One example is WavJourney: https://audio-agi.github.io/WavJourney_demopage/ https://audio-agi.github.io/WavJourney_demopage/