11 ms·
Stable Diffusion: Is Video Coming Soon?
- bob_paulson 4y agohmm noise added to pictures using AI. Is Midjourney and DALL-E2 not there yet? Just asking. Don't know.
- adamsmith143 4y ago>hmm noise added to pictures Hope you don't actually think that's what these models do.
- bob_paulson 4y agohumor...
- prophesi 4y agoI think even taking Poe's Law into consideration it would be difficult to parse that comment as a joke.
- the_third_wave 4y agoVideo is already possible as is shown by e.g. SALT_VERSE [1] (which may or may not use SD/DALL-E/... but that is not that interesting), what is not there yet is a --txt2vid script option. Implementing this would not be that hard but the processing time needed is quite substantial. Something else which would be possible is the use of a model like SD in combination with a frame interpolation model like [2] as a video generator. Use SD to generate key frames, feed these to FILM and let it generate the intermediate frames and you should get video. [1] https://twitter.com/SALT_VERSE https://twitter.com/SALT_VERSE [2] https://film-net.github.io/ https://film-net.github.io/
- deleted 4y ago[deleted]
- quitit 4y agoAmongst the many video experiments, I find Karen Cheng is making some interesting concepts by piping AI output to a range of other interpolation tools. Below is a mini tut where the content from Dall-e it piped to Ebsynth and then DAIN. https://www.instagram.com/reel/Ch7aV2mjWOD/ https://www.instagram.com/reel/Ch7aV2mjWOD/ In summary: Dall-e generated the outfits, Ebsynth mapped those outfits to a range of frames (instead of having a new random artwork on each frame) & Dain smoothed the transitions between each outfit change. Another example of this concept is here: https://www.instagram.com/reel/ChmyFNoDHZY/ https://www.instagram.com/reel/ChmyFNoDHZY/
- cma 4y agois there a mirror without a login wall anywhere?
- quitit 4y agosearch for “instagram downloader”
- quitit 4y agoPiggy-backing off my comment, because I forgot to mention this very interesting project: https://replicate.com/andreasjansson/stable-diffusion-animation https://replicate.com/andreasjansson/stable-diffusion-animat... The idea is that it takes two txt2img prompts and animates between them. The result isn't 100% there yet, but I think this idea has some legs on the 2D side of things because of the way it's able to keep a consistent composition. (One of the core issues with txt2img is that while it can produce nice images, it can product just one - animation, or even a series of story boards requires a lot of manual intervention from the creative.
- aqme28 4y ago> Implementing this would not be that hard The fact that no one has been able to demonstrate it convincingly makes me think that it might actually be quite hard.
- yeldarb 4y agoI’m not sure if this is real, but this video of a stable diffusion powered video editor was posted on Reddit a couple weeks ago and looks insane: https://www.reddit.com/r/MachineLearning/comments/wmypmh/a_demo_of_stable_diffusion_a_texttoimage_model/ https://www.reddit.com/r/MachineLearning/comments/wmypmh/a_d...
- yreg 4y agoI'm not as excited by the demo as you are. It seems that they just detect foreground, key it and put an arbitrary picture (in this case from stable diff) on a skybox behind. It seems closer to what Mac OS X Leopard's Photo Booth was able to do 15 years ago[1], than to a "Stable Diffusion for Video". [1] - https://web.archive.org/web/20071018033504/https://www.apple.com/macosx/features/300.html#photobooth https://web.archive.org/web/20071018033504/https://www.apple...
- deleted 4y ago[deleted]
- grumbel 4y agoFor object/background replacement one can use EBSynth[1] and combine it with in-painting of DALL-E2 and Co. [1] https://www.youtube.com/watch?v=Ki9NVyD3It4 https://www.youtube.com/watch?v=Ki9NVyD3It4
- null_shift 4y agoTrue video capability would entail describing a scene as a prompt and getting a video in return. Not interpolating between a handful of images as is being done now (not to discredit those). This will be a huge game changer when it occurs. Whether it be for deep fake videos, creating custom content, or making a new season of your favorite tv show that was cancelled too early. The possibilities are endless. This is probably not in the near future (i.e. this year), but I doubt it is very far off.
- seydor 4y agoWikiSeries
- gitfan86 4y agoYou could "hack it" by using a couple of other models as part of your pipeline. Similarly to how you have to use GAN after SD to "fix" faces sometimes. You also could put a language model on top of your prompting system. So "gandolff kicking ass" gets translated into " Page XXX, Paragraph XX from LOTR "
- ShamelessC 4y agoThe article shows a model that does this. It's only a few frames, but they are entirely generated from text - no seed image or interpolation required.
- MuffinFlavored 4y agoWhat is referred to/defined as "interpolation" because as an outsider... isn't "Stable Diffusion interpolating text into images/frames/video" in a "literal" (maybe not technical) sense?
- FlyingAvatar 4y agoIt's to be interpreted in the quasi-mathematical sense where you have images for frame A and frame B representing your data points. To interpolate between those frames, a flow of plausible images simulating the transition from A to B is generated.
- O__________O 4y agoCurious, what other notable features or use cases are people aware of that are already existing in other systems, but not present in Stable Diffusion? Anyone aware of any other open source projects that have a known list of requested features and way for community to express support for them, for example bounties?
- deleted 4y ago[deleted]
- hnthrowaway0328 4y agoI'm wondering is there a service to create tilemap spritesheet? Say I want to make a retro style Ultima-ish spritesheet with 64*64 sprites, but I don't have any drawing skill, is there an AI to generate some for me?
- heliophobicdude 4y agoTry using Dalle for it. I just searched "sprite sheet of a cute 16 bit character"
- hnthrowaway0328 4y agoThanks, will try it out.
- lucidrains 4y agoyea of course https://video-diffusion.github.io/ https://video-diffusion.github.io/ we are going all the way to the holodeck (my guess is latent diffusion on weights, aka hypernetwork, of a 4d NERF)
- deleted 4y ago[deleted]
- alphabetting 4y agoNot close in short term but it's inevitable https://sites.google.com/view/transframer https://sites.google.com/view/transframer
- lucidrains 4y agoyup, training on DCT representation in general is an untapped research direction. can also help with larger images, longer audio, etc
- astrange 4y agoDCT is a linear transformation so the model ought to be able to learn it (and typically already does). If you're going to feed in some non-learned transformation it seems like it should be nonlinear to add value.
- lucidrains 4y agoTrue, it goes against the spirit of deep learning Still the Transframer results are interesting and has encouraged me to look into it
- Tenoke 4y agoWhile on the software side we are close, the amount of compute required to train a model to produce longer, high quality, sensible, and non-surrealistic videos - especially with a plot - is simply too much for the time being.
- kachurovskiy 4y agoGoogle Photos already has image to video capabilities with the Cinematic photo feature - https://blog.google/products/photos/new-cinematic-photos-and-more-ways-relive-your-memories/ https://blog.google/products/photos/new-cinematic-photos-and...
- throwaway920102 4y agoVideo will suffer even more than still imagery or text from the inherent lack of continuity/self-consistent memory that these autocomplete/prediction algorithms have. So cool for abstract art but not for storytelling or following a script. Unless you are OK with the content being visually inconsistent like an acid trip.
- FairlyInvolved 4y agoI think that's just a scaling issue, fundamentally there's no reason why a model trained on video couldn't come to create coherent motion in the same way that image models can now product coherent lighting/themes. Smaller image models had the same problems with logical inconsistency just because they didn't have sufficient general understanding of how visual concepts. The same is almost certainly true of video - early smaller models will likely create janky movements/motion, however once they've seen enough video to understand how a person walks, how a scene is framed etc.. there's no reason we couldn't get to the same level of maturity as today's image models. I think the real issue will come from labelling - most video is only going to be labelled simply with basic info/captions without detailed descriptions of the camera pan, movement of subjects. The amount of text required to accurately describe a scene is much larger than a still image and I'm not sure how once would go about collecting this.
- babyshake 4y agoIt does seem to be the case that this type of generative image AI makes somewhat surreal imagery unless given very specific literal directions (aka, Tom Cruise hugging Ben Affleck). If using this AI right now it is probably best to work within these constraints.
- dqpb 4y agoFor video, I feel like we’d be better off generating models, textures, and universal scene descriptions.
- teddyh 4y agoAn image is already a 2-dimensional flattening of a 3-dimensional world. A generator of 2-dimensional images would already have a disadvantage, since they have not learned anything about the three dimensions, and are simply faking what other, real, 2-dimensional images look like, including all artifacts which comes from 3-dimensionality being flattened. But these artifacts will always be weird and not consistent with each other, since the image generator does not really know anything about the third dimension, it’s just mimicking what artifacts it has learned by seeing them. Video is one additional dimension: time. I would guess that similar artifacts would also show up here, and be difficult to eliminate.
- l33tman 4y agoThe current image generators certainly know about 3D internally. It's not put there explicitly but the wealth of knowledge they possess and the perfect lighting and shadowing and material reflections compared to the tiny amount of weights strongly suggest they have proper representations (just like humans by the way). Even a small model in early training learns to do proper shadowing and lighting and mostly makes mistakes in the 3D high-level object space, like wrong number of wheels or legs or fingers and stuff like that.
- teddyh 4y ago> It's not put there explicitly but the wealth of knowledge they possess and the perfect lighting and shadowing and material reflections compared to the tiny amount of weights strongly suggest they have proper representations I don’t think they do, but I can’t prove it, either way. > (just like humans by the way). Oh, I don’t think humans have it either (mostly), since humans don’t have 3D vision; humans have the equivalent of wiggle stereoscopy¹, which gives some hints of depth, and humans have the additional advantage of the time dimension. People do have some intellectual capacity to reason about 3-dimensional shapes, and some people can even rotate things in their heads with ease. Blind people might have it too, since their concept of the world was not created by this pseudo-3D visual input. But mostly, people don’t think in 3D. I think we can see this by looking at drawings by children. Child drawings are dominated by the concepts important to children: Faces, hands, etc. Concepts, not actual images. And it’s all in 2D, as was the majority of art for much of history. 1. https://en.wikipedia.org/wiki/Wiggle_stereoscopy https://en.wikipedia.org/wiki/Wiggle_stereoscopy
- ksec 4y agoMay be before Video it would be 3D Models? I am thinking it should dramatically lower the cost of current Gaming development budget.
- astlouis44 4y agoAgreed, this will be incredibly disruptive to the video game industry.
- SpaceManNabs 4y agoit is still a significant endeavor for the same reason audio is still difficult. will be a huge breakthrough because it means having made quantifying uncertainty in the near continuous limit tractable.
- jnurmine 4y agoIs there something like Stable Diffusion for text only? Like, feed in some text and pose questions, like people have done with GPT-3.
- GaggiX 4y agoThere are many open source models like GPT-3, the problem is that you need a GPU cluster if you want to run something yourself that has similar performance to GPT-3.
- astrange 4y agoNone of them really work though; the current GPT-3 (InstructGPT) is fine tuned to answer questions, which isn't the same as the general text completion problem they're all trained on. You can try TextSynth and see the result is nowhere near as good.
- deleted 4y ago[deleted]
- mcbuilder 4y agoThe pieces are coming into place https://github.com/microsoft/VideoX/tree/master/X-CLIP https://github.com/microsoft/VideoX/tree/master/X-CLIP
- jasondigitized 4y agoVideo game level design and artifacts seems like a closer next step.
- tarunmuvvala 4y agoI guess the next step is to make stable diffusion better a lot of local context driven and multi- image prediction . Today still the AI misses local context as the images are more of trained from Open images annotated in English by experts. But imagine if you have single image annotated by multiple people in different languages then what happens to AI capabilities
- m3kw9 4y agoYes but training it and massaging the data set properly would be pretty mind blowing
- spyder 4y agoAnother cool trick: to achieve video in-painting, combine diffusion models with ebsyth: https://www.tiktok.com/@karenxcheng/video/7138067105211075883?is_copy_url=1&is_from_webapp=v1 https://www.tiktok.com/@karenxcheng/video/713806710521107588...
- zone411 4y ago> "Stable Diffusion was trained primarily on LAION-Aesthetics, a collection of subsets of the LAION 5B dataset, which is itself a subset of LAION-400M" Err, no? As you can tell by the name itself, the 5B dataset is larger. > "Unlike autoencoder-based deepfake content, or the human recreations that can be achieved by Neural Radiance Fields (NeRF) and Generative Adversarial Networks (GANs), diffusion-based systems learn to generate images by adding noise" This is confused. Diffusion is orthogonal to NeRF. For example, here is a paper that uses both: https://arxiv.org/abs/2112.12390 https://arxiv.org/abs/2112.12390 > "Within days of release, the open sourced Stable Diffusion code and weights were packaged into a free Windows executable" That's not how it became popular. > "Additionally, at the time of writing, Google Research has just released a similar system called DreamBooth, which likewise ‘tokenizes’ a desired element into a distinct ‘object’." The approaches are actually very different. DreamBooth uses fine-tuning, unlike textual inversion.
- astrange 4y ago> This is confused. Diffusion is orthogonal to NeRF. Also, diffusion models learn to remove noise (guided by a description of the undiffused image), not add it. Though, it seems you don't have to use noise as the information destruction mechanism; blurring works, and I wonder if there isn't something better for animation.
- amelius 4y agoHow about video game 3d assets? And physics simulations?
- pc2g4d 4y agoThe "initial frame -> video" problem seems way harder than generating video from a text input. Once a good dataset for that problem is assembled, it seems like Stable Diffusion would naturally accept an additional "time" dimension, and generate cohesive output, with corresponding massively increased hardware requirements. Though I'd venture that the first "novel to feature film" or "novel to TV series" algorithm won't just be an upscaling of this tech....