3 ms·

Fascinating leap forward. It makes me think of the difference between ancestral and non-ancestral samplers, e.g. Euler vs Euler Ancestral. With Euler, the outp

Fascinating leap forward. It makes me think of the difference between ancestral and non-ancestral samplers, e.g. Euler vs Euler Ancestral. With Euler, the output is somewhat deterministic and doesn't vary with increasing sampling steps, but with Ancestral, noise is added to each step which creates more variety but is more random/stochastic. I assume to create video, the sampler needs to lean heavily on the previous frame while injecting some kind of sub-prompt, like rotate <object> to the left by 5 degrees, etc. I like the phrase another commenter used, "temporal consistency". Edit: Indeed the special sauce is "temporal layers". [0] > Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets [0] https://stability.ai/research/stable-video-diffusion-scaling-latent-video-diffusion-models-to-large-datasets https://stability.ai/research/stable-video-diffusion-scaling...

adventured 3y ago
The hardest problem the Stable Diffusion community has dealt with in terms of quality has been in the video space, largely in relation to the consistency between frames. It's probably the most commonly discussed problem for example on r/stablediffusion. Temporal consistency is the popular term for that. So this example was posted an hour ago, and it's jumping all over the place frame to frame (somewhat weak temporal consistency). The author appears to have used pretty straight-forward text2img + Animatediff: https://www.reddit.com/r/StableDiffusion/comments/180no09/on_the_swing_animatediff_text2image_only/ https://www.reddit.com/r/StableDiffusion/comments/180no09/on... Fixing that frame to frame jitter related to animation is probably the most in-demand thing around Stable Diffusion right now. Animatediff motion painting made a splash the other day: https://www.reddit.com/r/StableDiffusion/comments/17xnqn7/roll_your_own_motion_brush_with_animatediff_and/ https://www.reddit.com/r/StableDiffusion/comments/17xnqn7/ro... It's definitely an exciting time around SD + animation. You can see how close it is to reaching the next level of generation.

adventured 3y ago

The hardest problem the Stable Diffusion community has dealt with in terms of quality has been in the video space, largely in relation to the consistency between frames. It's probably the most commonly discussed problem for example on r/stablediffusion. Temporal consistency is the popular term for that. So this example was posted an hour ago, and it's jumping all over the place frame to frame (somewhat weak temporal consistency). The author appears to have used pretty straight-forward text2img + Animatediff: https://www.reddit.com/r/StableDiffusion/comments/180no09/on_the_swing_animatediff_text2image_only/ https://www.reddit.com/r/StableDiffusion/comments/180no09/on... Fixing that frame to frame jitter related to animation is probably the most in-demand thing around Stable Diffusion right now. Animatediff motion painting made a splash the other day: https://www.reddit.com/r/StableDiffusion/comments/17xnqn7/roll_your_own_motion_brush_with_animatediff_and/ https://www.reddit.com/r/StableDiffusion/comments/17xnqn7/ro... It's definitely an exciting time around SD + animation. You can see how close it is to reaching the next level of generation.