5 ms·
Stablevideo: Text-driven consistency-aware diffusion video editing
- stale2002 3y agoDid these guys just straight up solve the generative video problem? The results are better than anything that I've ever seen. What's the catch? Large processing times? Are the results cherry picked? Or what? I guess it only works for video to video, but that's still amazing!
- satvikpendem 3y agoVideo to video is not too special, we've had temporally stable solutions to those for some time now, and they're even in commercially available apps. The real test is true text to video, which is much harder. To my knowledge, the only open source solution that works well for text to video is Zeroscope v2 XL, and v3 is coming soon. v2 is already on par with RunwayML's Gen-2 while v3 is better.
- varelse 3y agoRunway runs circles around zeroscope. But that shouldn't be a blocker for zeroscope to catch up and/or surpass. Both runway and pika Labs deliver better quality at the moment. Evidence: struggling with all three of them. Runway outputs the best video quality and options for video length whilst pika delivers better fidelity to an input image as inspiration. All of this subject to change without notice
- atorodius 3y agoBeing able to condition on a video vs. just text massively simplifies the task. Mainly because you get consistent camera motion and movement in the scene for free!
- progbits 3y agoI'm the car example the wheels don't spin which is interesting. The original frames have different wheel angles so simple text prompted img2img frame by frame approach would preserve the motion, but at the cost of interframe consistency. Here you get consistent look of the scene and no rapid transitions, but the wheel motion is gone.
- remixer-dec 3y agoIf I understand correctly, you have to train a custom NLA model for each video segment before actually using it.
- alonfnt 3y agoTotally unrelated to the paper itself, but is there a GitHub page template to showcase a paper/project as they have? I keep seeing these type of webpages to promote papers but haven't found the template yet.
- Joeri 3y agoYou can see the source of the github pages site on github: https://github.com/rese1f/StableVideo/tree/web https://github.com/rese1f/StableVideo/tree/web It seems they forked from somebody else and then changed the content to match their paper.
- bawolff 3y agoWeird, its about video but there is no example video (?)
- progbits 3y agoThere are few examples on top of the page. Check your content blocker I guess?
- TaylorAlexander 3y agoI see a carousel with sample image frames. There’s video examples?
- satvikpendem 3y agoYou might have autoplay turned off in your browser. Those are videos, not images.
- TaylorAlexander 3y agoAh apparently. It works in firefox on desktop linux, but did not work on iphone even when I clicked on them.
- dmazzoni 3y agoMore examples on this page: https://github.com/rese1f/StableVideo https://github.com/rese1f/StableVideo
- veb 3y agoThanks mate, I really wanted to see some demos myself, but OPs link didn't show me much. This is pretty exciting, I know it's in its infancy but still... exciting times!
- madacol 3y agoIt's the same 3 examples shown in web page (It's a carousel, you have to click the "righ/left" buttons to see the others)
- LoveMortuus 3y agoThis looks exciting. Can't wait to see what the future holds!
- runeks 3y agoIt's fun to watch how it blends together videos, but none of the three example videos are actually of high enough quality to be useful. 1. The boat video transforms the coast into rolling waves and looks super weird 2. The swan/duck video looks better than the others, but the lighting is obviously wrong when looking closer. Looks like a cardboard cutout bird. 3. The car video looks like a video game from the 2000s, with low quality textures and wheels not turning. Again, super interesting to see how it can come up with these "by itself", but utterly useless at the moment.
- strbean 3y agoYou missed "the duck looks like a swan wearing a duck skin-suit" :) I wouldn't call it entirely useless though - it makes for an interesting surreal effect. I could see something really cool coming out of this in the A Scanner Dark movie vein. I particularly liked Rusty Car in the Desert.
- nasir 3y agoI suppose this will be a stepping stone for it to get there.
- runeks 3y agoThat's what everyone seems to suppose. I don't see why it's warranted, though.
- lhl 3y agoAlso recently released is "CoDeF: Content Deformation Fields for Temporally Consistent Video Processing" https://qiuyu96.github.io/CoDeF/ https://qiuyu96.github.io/CoDeF/ "We present the content deformation field (CoDeF) as a new type of video representation, which consists of a canonical content field aggregating the static contents in the entire video and a temporal deformation field recording the transformations from the canonical image (i.e., rendered from the canonical content field) to each individual frame along the time axis. Given a target video, these two fields are jointly optimized to reconstruct it through a carefully tailored rendering pipeline. We advisedly introduce some regularizations into the optimization process, urging the canonical content field to inherit semantics (e.g., the object shape) from the video." The results also look very stable/impressive.
- deleted 3y ago[deleted]
- infofarmer 3y agoLayman thoughts: (1) with enough high-quality training data, «AI» models should be able to output H265 / H266 / AV1 directly, can achieve simplicity and reduce artifacts by skipping an inferior compression step and leveraging temporal elements (2) if AI video compression (as demoed by nvidia) becomes standard, the training data and generated data will become [more] «AI-native», boosting these efforts by miles
- goeiedaggoeie 3y agowhy would you want 1? having the raster frames is surely better for post production. I agree that models should take a stab at compression but I think it should be independent. At the end of the day you also don't want to be doing video compression on your GPU, using a dedicated chip for that is so much more efficient. lastly, you don't want to compress the same all the time. FOr low latency we compress with no b-frames and a smallish GOP, with VOD we have a long GOP and b-frames are great for compression. 2. as long as we can again port the algo's to dedicated hardware, which are on mobiles a must for energy efficiency for both encode and decode
- CWuestefeld 3y agoIt makes me wonder if the car video might have come out better if they'd prompted to have it put in the desert, rather than in a dessert.