5 ms·

Video is already possible as is shown by e.g. SALT_VERSE [1] (which may or may not use SD/DALL-E/... but that is not that interesting), what is not there yet is

Video is already possible as is shown by e.g. SALT_VERSE [1] (which may or may not use SD/DALL-E/... but that is not that interesting), what is not there yet is a --txt2vid script option. Implementing this would not be that hard but the processing time needed is quite substantial. Something else which would be possible is the use of a model like SD in combination with a frame interpolation model like [2] as a video generator. Use SD to generate key frames, feed these to FILM and let it generate the intermediate frames and you should get video. [1] https://twitter.com/SALT_VERSE https://twitter.com/SALT_VERSE [2] https://film-net.github.io/ https://film-net.github.io/

deleted 4y ago
[deleted]
quitit 4y ago
Amongst the many video experiments, I find Karen Cheng is making some interesting concepts by piping AI output to a range of other interpolation tools. Below is a mini tut where the content from Dall-e it piped to Ebsynth and then DAIN. https://www.instagram.com/reel/Ch7aV2mjWOD/ https://www.instagram.com/reel/Ch7aV2mjWOD/ In summary: Dall-e generated the outfits, Ebsynth mapped those outfits to a range of frames (instead of having a new random artwork on each frame) & Dain smoothed the transitions between each outfit change. Another example of this concept is here: https://www.instagram.com/reel/ChmyFNoDHZY/ https://www.instagram.com/reel/ChmyFNoDHZY/
- cma 4y ago
  is there a mirror without a login wall anywhere?
  - quitit 4y ago
    search for “instagram downloader”
- quitit 4y ago
  Piggy-backing off my comment, because I forgot to mention this very interesting project: https://replicate.com/andreasjansson/stable-diffusion-animation https://replicate.com/andreasjansson/stable-diffusion-animat... The idea is that it takes two txt2img prompts and animates between them. The result isn't 100% there yet, but I think this idea has some legs on the 2D side of things because of the way it's able to keep a consistent composition. (One of the core issues with txt2img is that while it can produce nice images, it can product just one - animation, or even a series of story boards requires a lot of manual intervention from the creative.
aqme28 4y ago
> Implementing this would not be that hard The fact that no one has been able to demonstrate it convincingly makes me think that it might actually be quite hard.
Vetch 4y ago
The article is thorough and covers why your [2], under tweening, is insufficient. Video beyond stereotyped repetitive movements is likely AI-complete. Presumably you'd want to be director and be able to control camera angles. You might want to have a cartoon or 3D render visual style. Unreal engine prompt tag even more literal. Most important, you will want actors. Anything alive will have to move convincingly and intentionally. Some might even bring up the ethics of allowing the movie to end. Before proper video, I think we'll first have to see tools for music, 3D rendering and animation that bring down difficulty by orders of magnitude.
O__________O 4y ago
Comment by SALT creator on HN yesterday saying they are using Midjourney, though Midjourney supports Stable Diffusion too: https://news.ycombinator.com/item?id=32651855 https://news.ycombinator.com/item?id=32651855
contingo 4y ago
SALT_VERSE just uses still images generated by DALL•E-2 and Midjourney. They're animated using techniques that have already been around for a long time: panning around and zooming in and out of a larger frame, fake raster-to-3d effects (auto projection mapping), typical After Effects transitions. There are some overlaid face animations done with something along the lines of NVIDIA's Audio2Face. None of it is video generated by SD, DALL•E-2 and Midjourney. There have been some actual video explorations of SD's latent space, this example amazes me: https://twitter.com/karpathy/status/1559343616270557184 https://twitter.com/karpathy/status/1559343616270557184

deleted 4y ago

[deleted]

quitit 4y ago

Amongst the many video experiments, I find Karen Cheng is making some interesting concepts by piping AI output to a range of other interpolation tools. Below is a mini tut where the content from Dall-e it piped to Ebsynth and then DAIN. https://www.instagram.com/reel/Ch7aV2mjWOD/ https://www.instagram.com/reel/Ch7aV2mjWOD/ In summary: Dall-e generated the outfits, Ebsynth mapped those outfits to a range of frames (instead of having a new random artwork on each frame) & Dain smoothed the transitions between each outfit change. Another example of this concept is here: https://www.instagram.com/reel/ChmyFNoDHZY/ https://www.instagram.com/reel/ChmyFNoDHZY/

cma 4y ago
is there a mirror without a login wall anywhere?
- quitit 4y ago
  search for “instagram downloader”
quitit 4y ago
Piggy-backing off my comment, because I forgot to mention this very interesting project: https://replicate.com/andreasjansson/stable-diffusion-animation https://replicate.com/andreasjansson/stable-diffusion-animat... The idea is that it takes two txt2img prompts and animates between them. The result isn't 100% there yet, but I think this idea has some legs on the 2D side of things because of the way it's able to keep a consistent composition. (One of the core issues with txt2img is that while it can produce nice images, it can product just one - animation, or even a series of story boards requires a lot of manual intervention from the creative.

aqme28 4y ago

> Implementing this would not be that hard The fact that no one has been able to demonstrate it convincingly makes me think that it might actually be quite hard.

Vetch 4y ago

The article is thorough and covers why your [2], under tweening, is insufficient. Video beyond stereotyped repetitive movements is likely AI-complete. Presumably you'd want to be director and be able to control camera angles. You might want to have a cartoon or 3D render visual style. Unreal engine prompt tag even more literal. Most important, you will want actors. Anything alive will have to move convincingly and intentionally. Some might even bring up the ethics of allowing the movie to end. Before proper video, I think we'll first have to see tools for music, 3D rendering and animation that bring down difficulty by orders of magnitude.

O__________O 4y ago

Comment by SALT creator on HN yesterday saying they are using Midjourney, though Midjourney supports Stable Diffusion too: https://news.ycombinator.com/item?id=32651855 https://news.ycombinator.com/item?id=32651855

contingo 4y ago

SALT_VERSE just uses still images generated by DALL•E-2 and Midjourney. They're animated using techniques that have already been around for a long time: panning around and zooming in and out of a larger frame, fake raster-to-3d effects (auto projection mapping), typical After Effects transitions. There are some overlaid face animations done with something along the lines of NVIDIA's Audio2Face. None of it is video generated by SD, DALL•E-2 and Midjourney. There have been some actual video explorations of SD's latent space, this example amazes me: https://twitter.com/karpathy/status/1559343616270557184 https://twitter.com/karpathy/status/1559343616270557184