30 ms·
Video to video with Stable Diffusion
- USB5 3y ago[dead]
- politelemon 3y agoI've only been using stable diffusion in a very basic way, give some text, get some images back. Seeing what others produce with it is amazing and also just feels beyond me, I'm simply not familiar with all these other extensions and scripts. That said I appreciate a tutorial like this which isn't glossing over steps, so it's worth a try.
- tetris11 3y agoIt's insane the speed of progress. That being said, the most smooth example in that tutorial (Temporal Kit) required a lot of outside processing. I wonder if AUTOMATIC1111 ui will start handling workflow scripts, a.l.a blender workflow boxes.
- Kuinox 3y agoThere is another UI for workflow scripts: https://github.com/comfyanonymous/ComfyUI https://github.com/comfyanonymous/ComfyUI Give it enough times and it will also have the UI simplicity of automatic1111 UI.
- andybak 3y agoautomatic1111 isn't what immediately springs to mind when I hear the phrase "UI simplicity"
- Kuinox 3y agoComfyUI allow you to design your own workflow. It also mean understanding how things works, it's far harder to use than automatic1111 UI.
- washadjeffmad 3y agoComfy is harder if you've only used gradio, just like Blender's modal interface was "too complicated" if you'd only used Autodesk products. Each has its strengths, but I do wonder whether I'd feel automatic1111 was too random and restrictive if Comfy had been released first.
- tetris11 3y agoThat's really impressive. This workflow in particular speaks of major potential: https://comfyanonymous.github.io/ComfyUI_examples/controlnet/mixing_controlnets.png https://comfyanonymous.github.io/ComfyUI_examples/controlnet...
- gareth_untether 3y agoCould people please add newbie resources to this post?
- DrSiemer 3y agoNot going to bother with this until the temporal cohesion issue is solved. The results are cool, but the variety in frames makes it look like a very specific and distracting art style instead of true animation.
- colordrops 3y agoDid you scroll down and see the different methods used, specifically method 5? Not perfect, but getting pretty close.
- justinclift 3y agoIt's a lot better than earlier work, but still not really at the level it needs to get to.
- moonchrome 3y agoThe last one looks the creepiest - like a dystopian matrix glitch. Could be a cool effect to illustrate that.
- rpastuszak 3y agoThe light reflection on the shoulder doesn't make much sense. I think this is less pronounced when soft light is involved. I wonder if sharp light and reflections will take longer to be handled properly. Every frame on the shoulder looks ok on its own, but when animated it feels like it was taken in a different environment than the rest. I suspect depth maps (SD2 already supports them) could be used to achieve that in the future. I wonder if a diffusion model could accept an "onion skin" noise, so the transitions between frames would be less jarring. Can someone with more knowledge than me explain what's the most promising approach here?
- bondarchuk 3y ago>The light reflection on the shoulder doesn't make much sense. People will really grasp at anything to find something to critique in generative AI, huh?
- baddhidurgarao 3y ago[flagged]
- bulbosaur123 3y agoDid they finally fix the annoying flickering?
- varelse 3y ago[dead]
- bulbosaur123 3y agoIsn't runway vid2vid superior in the aspect of coherency? This flickering is horrible.
- andybak 3y agoYes but people want to run stuff on their own hardware or at least have a deep ability to configure and mix and match tools.
- padolsey 3y agoOnce the temporal fixes are applied, the residual morphing and flickering actually quite appeals to me. Reminds me of the animation style in A Scanner Darkly.
- IsTom 3y agoAnd looks like scramble suits without it
- horsawlarway 3y agoA Scanner Darkly was also immediately where my mind went. I enjoy the style, but I'm also not sure it's the right medium for most works.
- ajani 3y agoAnd to think ASD was entirely hand rotoscoped!
- villgax 3y ago[flagged]
- karles 3y agoSame aesthetics as A-ha's "Take on me". Even with the flaws, I can see how this can simply be used with that exact aesthetic in mind, without the need for any improvement.
- cnschn 3y agoI had the exact same reaction!
- GeoAtreides 3y agoSomething like this (music video, NSFW ): https://www.youtube.com/watch?v=edKo3y2cFUg https://www.youtube.com/watch?v=edKo3y2cFUg
- deleted 3y ago[deleted]
- torginus 3y agoIt looks to me there's a tradeoff between how much SD changes the input, and how temporally coherent it is.
- hospitalJail 3y agoI was going to get a 4090 this year, but I just don't think 24gb VRAM is going to bee enough in the short term future(for AI related stuff). Ended up getting a 3060 for 1/4 of the cost, and I'm planning on using/paying for Colab until some 6090 comes out with 128gb vram. Something that kind of bothers me that I don't understand. Why is there such an obsession about having small computers/servers? I don't care if I have to put my computer in the basement because its the size of a closet or two. Maybe there is an EE issue, like too much current or EMF if you make a large computer. I'd usually use colab, for 99% of the jobs... but I admittedly want to do some nsfw stuff with me and the wife with AI Art.
- dist-epoch 3y ago> Why is there such an obsession about having small computers/servers More likely just cost. If a "small" computer costs $2k, how much do you think a "big" one will cost? $10k+ probably, what is the market for that? And you can buy a rack in your home and put 10 servers in it if you really want a closet sized computer.
- hospitalJail 3y agoYou'd expect manufacturing costs to go down.
- dist-epoch 3y agoMost of the cost is in CPU/GPU/RAM.
- hospitalJail 3y agoYes, so if your CPU doesnt need to be the size of a box of cards, your GPU doesnt need to be the size of a library book, the manufacturing costs would go down.
- mensetmanusman 3y ago
- mherrmann 3y agoDoes someone know how long rendering these videos takes? Can it be done in real-time?
- dmingod666 3y agoNo can't do it realtime. Each image would be about 20-60 seconds (depends on what you're doing with it)
- yarg 3y agoSo all that's happening is people are getting increasingly good at hiding violations of spatiotemporal invariants? This is treating the symptom and ignoring the disease.
- erybodyknows 3y agoDoes anyone “in the know” have approximations on cost to run this on collab or similar? Maybe a very rough Dollars per minute of video?
- dmingod666 3y agoA frame is like 20-30 seconds for one pass on my 3090, you may have multiple tools and multiple passes.. say 60 seconds a frame x 300 frames about 5 hours -- 3090 costs 0.44$ per hour on runpod.. a little more than 2$
- unshavedyak 3y agoKinda crazy how averse to ongoing costs i am. I have a 3080TI for Blender rendering and sculpting and general blender, so i definitely need something good local.. but i'm debating buying top of the line on the next iteration so i can do this stuff locally. Thousands of dollars perhaps just to avoid a $2 cost lol.
- dmingod666 3y agoThe 2$ adds up + thats an ephemeral environment and these things are super finiky to get working properly (dependency and initial setup + adding tools to your workflow). Plus there is a more of a craft like - ad-hoc approach to how you want to do things, plus a learning curve, with what tools and scripts you want to do or write simple stuff on your own -- runpod sounds good on paper but after a while, you then want AWS/Google VM so things are stable and your environment is sane and stable. Then if you calculate, you'll come up to a cost of buying a nice card in 6-10 monthly payments.. So if you want to play with it in peace you spend once and only worry about electricity costs :D
- swayvil 3y agoThat first video is fabulous. It reminds me of a standard trope in generative art : You generate a whole bunch of prospective images and then pick the nicest ones. Except in this case we get to see all the prospects (The variations in chestplate, wings etc). And in such a cool style.
- throwaway22032 3y agoDoes anyone else find it weird that a huge amount of AI video/imagery in tutorials etc seem to be sexy anime schoolgirls? I'm a bloke, I've never been that particularly bothered by like, Lara Croft, Lena, muscular guys in action games etc, I get it, I want to look at idealized/attractive things as much as the next person, but something about this makes me super uncomfortable.
- varelse 3y ago[dead]
- m3kw9 3y agoThe end result look like a a cartoon filter applied frame by frame. Or, each frame is basically going through stable diffusion in photo mode
- aantix 3y agoThe Temporal Kit write up was amazing.
- plandis 3y ago> Something that kind of bothers me that I don't understand. Why is there such an obsession about having small computers/servers? I don't care if I have to put my computer in the basement because its the size of a closet or two. Maybe there is an EE issue, like too much current or EMF if you make a large computer. I’m not using my GPU for heavy machine learning tasks at home. I have a Ncase M1 with a ryzen 3950x (16core/32 thread) and a 3090 and a few nvme m2 drives. It’s water cooled but for me the video card is the only pcie card I have, and water cooling keeps the temps reasonable even in the small space. It was a little tricky to build in but why take up a bunch of unnecessary space?