18 ms·
Real-time image editing using latent consistency models
- joerambo808 3y agowhy is it faster than latent diffusion models?
- vipermu 3y agoit uses a new technique called "consistency" that lets latent diffusion models to predict images in much fewer steps. some links here: - https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378 - https://arxiv.org/abs/2311.05556 https://arxiv.org/abs/2311.05556
- billconan 3y agolatent diffusion is an iterative process, the image becomes clearer one step at a time. The process can be viewed as a particle moving in the image space, one step at a time to its final position in the image space, which is the generated image. consistency model tries to predict the movement trajectory by providing the current position in the image space. Hence, what used to be a step-by-step process becomes a one-step process.
- dvrp 3y agooh wow, never thought of it that way
- swyx 3y ago> consistency model tries to predict the movement trajectory by providing the current position in the image space. Hence, what used to be a step-by-step process becomes a one-step process. no that wasnt a sufficient explanation for me. what is the prediction method here? why was diffusion necessary in the past? what tradeoffs does this approach have?
- quadrature 3y agofrom https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378 it sounds like its a form of distillation of an SDL model. So im guessing it can't be directly trained, but once you have a trained diffusion model you can distil a predictor which cuts out the iterative steps. While it can do 1-step the output quality looks a ton better with additional steps.
- dvrp 3y agoYou can train it directly. This is from the paper “An LCM demands merely 32 A100 GPUs Hours training for 2-step inference [...]”
- T-A 3y agoLet's quote the whole sentence, shall we? An LCM demands merely 32 A100 GPUs Hours training for 2-step inference, as de- picted in Figure 1. [1] Now let's look at the caption under Figure 1: LCMs can be distilled from any pre-trained Stable Diffusion (SD) in only 4,000 training steps (∼32 A100 GPU Hours) for generating high quality 768×768 resolution images in 2∼4 steps or even one step, significantly accelerating text-to-image generation. The training mentioned in your quote is distillation: it requires a previously trained SD model. You could have told just by reading the introduction: We propose a one-stage guided distillation method to efficiently convert a pretrained guided diffusion model into a latent consistency model by solving an augmented PF-ODE. [1] Section 4.2 in https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378
- deleted 3y ago[deleted]
- billconan 3y agoto my defense, if you look at the original paper https://arxiv.org/pdf/2303.01469.pdf https://arxiv.org/pdf/2303.01469.pdf the exact neural network used for the prediction method is omitted. apparently many neural networks can be used for this prediction method as long as they fulfill certain requirements. > why was diffusion necessary in the past? in the paper, one way to train a consistency model is distilling an existing diffusion model. But it can be trained from scratch too. "why was it necessary in the past " doesn't bother me that much. Before people know to use molding to make candles, they did it by dipping threads into wax. Why was thread dipping necessary? it's just a stepping stone of technology development.
- foota 3y agoWould that motion vector have to include like a color delta?
- billconan 3y agoif you are talking about latent diffusion, No, the "particle" is in a hyper-dimensional space, like for example a 10k-dimensional space. We are not supposed to interpret the meaning of that vector. and when that particle has moved to the right location, there is a decoder that converts it into an image. The decoder network knows how to interpret it.
- dvrp 3y agoExactly. I actually wrote a micro essay on Twitter the other day about the meaning of the classic encoder decoder network. It’s beautiful. But yeah! - For reference: https://x.com/asciidiego/status/1722544108252836119 https://x.com/asciidiego/status/1722544108252836119
- stavros 3y agoDoes this mean that you need this vector state in order to generate the next step? Ie I can't take an image (the pixels) and the prompt and run a few more steps on it?
- orbital-decay 3y agoThe architecture is the same but it has an additional model trained on the sampling trajectory computation, speeding up the process. So instead of 20 iterations with the original sampler that solves a SDE, you could use 8, 4, or sometimes even 2 iterations with this new ML-based sampler. That's not a new idea, but the author of this adapter figured out how to train it efficiently using very few GPU-hours.
- jiggawatts 3y agoI wonder if they could do a real-time preview using 2-4 iterations, and then 'refine' the quality with a higher number of iterations over time, much like how 3D editors preview ray-tracing.
- orbital-decay 3y agoI've lost the link, but IIRC someone on twitter demoed exactly that in Blender. "Neural rendering" in a 3D editor has some nice properties - the scene can be rendered into separate layers with separate depth maps and openpose skeletons generated from the rigs to serve as input for controlnets, objects can be tagged and tags composed into the prompt for the particular tile in case of tiled rendering, etc.
- dvrp 3y agoprobably not but are you talking about the guy who used control net and animatediff to make a video that looked relatively consistent?
- orbital-decay 3y agoNo, it was running in the Blender viewport using the original LCM Dreamshaper model.
- dvrp 3y agowe’re thinking of doing that as well! but first we want to make sure we can get this in the hands of a tight cohort of creatives and see how they use it.
- tasgon 3y agoDisclaimer: I'm actively working on this tool. This is in a closed beta for now (while we work on provisioning enough GPU compute) but we're hoping to make this public later.
- ttul 3y agoYeah, indeed. I know LCMs are way faster than diffusion models, but surely this demo is running on a beastly GPU...
- dvrp 3y agoyup, see comment above!
- boppo1 3y agoIs the model designed to run locally or does it run in the cloud somewhere? Is it your own model or is it based on sdxl or something like that?
- gailees 3y agoHow do you plan on stopping people from using this tool maliciously?
- monkellipse 3y agoThe same way you stop people from using a hammer maliciously?
- dvrp 3y agoIn the general sense yes, but I wonder if there will be unexpected things that we’ll need to take into account with this new generation of tools. Part of me thinks that this is another revolution in graphics the same way Photoshop was where you can work 10x faster. But another one ponders about what happens when we’re dealing with intelligence.
- jrflowers 3y ago>But another one ponders about what happens when we’re dealing with intelligence. This is a good point, diffusion models are an example of intelligence. Proof of that is that they became ubiquitous in the same year as large language models, thus they are the same.
- boppo1 3y agoI'm an illustrator. I'm fully capable of drawing/painting political figures in compromising or worse situations. Should the government blind me because my skills are dangerous?
- jiggawatts 3y agoSome politicians would say: "Yes." Artists, intellectuals, journalists, and critics of all sorts have been jailed, beaten, or simply murdered in the past for expressing themselves in ways the government of the time did not like.
- dvrp 3y ago
- byrneml 3y agoCould you apply the same technique for real-time video editing?
- vipermu 3y agoit would be interesting to play with LCM-LoRAs and AnimateDiff and see how much much this technique can speed up video generation. not sure if it's possible to just plug-and-play it or if we would need an extra LCM-LoRA for the motion module. once we have these sort of models producing frames in milliseconds we should be able to do something similar to this demo but with videos.
- kmavm 3y ago(Disclaimer: I'm an investor in Krea AI.) When Diego first showed me this animation, I wasn't completely sure what I was looking at, because I assumed the left and right sides were like composited together or something. But it's a unified screen recording; the right, generated side is keeping pace with the riffing the artist does in the little paint program on the left. There is no substitute for low latency in creative tools; if you have to sit there holding your breath every time you try something, you aren't just linearly slowed down. There are points that are just too hard to reach in slow, deliberate, 30+ second steps that a classical diffusion generation requires. When I first heard about consistency, my assumption was that it was just an accelerator. I expected we'd get faster, cheaper versions of the same kinds of interactions with visual models we're used to seeing. The fine hackers at Krea did not take long to prove me wrong!
- dvrp 3y agoExactly. There is no substitute for real-time when you're doing creative work. That's why GitHub Copilot works so well; that's why ChatGPT struck a chord with people—it streamed the characters back to you quite fast. At first, I was skeptical too. I asked myself “what about Photoshop 1.0? They surely couldn't do it in real-time.”. It turns out that even then you needed it. Of course, compute wasn't there to do a simple translation of all rasterized pixel values that form an image within a layer, but there was a trick they did: they showed you the outline that would tell you, the user, where the content _will_ render if you let the mouse go. You can see the workflow here: > https://www.youtube.com/watch?v=ftaIzyrMDqE https://www.youtube.com/watch?v=ftaIzyrMDqE It applies to general tools too; you can see the same on this MacOS 8 demo (it runs on the browser!): > https://infinitemac.org/1998/Mac%20OS%208.1 https://infinitemac.org/1998/Mac%20OS%208.1
- stavros 3y ago> that's why ChatGPT struck a chord with people—it streamed the characters back to you quite fast. So did GPT-3, though. ChatGPT (3.5) was a bit faster, but not overly so.
- 3y ago
- vmatsiiako 3y agoDoes this have to do anything at all with LoRAs?
- vipermu 3y agoindeed; we're able to make it work with SDXL thanks to a new technique that got released yesterday called LCM-LoRA. with LCM-LoRA you can turn models like SDXL into LCMs without need for training and you can add other style LoRAs like the ones you find on civit.ai in case you're interested, here's the technical report about LCM-LoRA: https://arxiv.org/abs/2311.05556 https://arxiv.org/abs/2311.05556
- GaggiX 3y agoYes, they are probably using: https://huggingface.co/blog/lcm_lora https://huggingface.co/blog/lcm_lora
- ShamelessC 3y agoColor me underwhelmed.
- dvrp 3y agowhat does it take to impress you?
- ShamelessC 3y agoThere was a webcam demo a little while back that was pretty cool. It's mostly that the right-hand side seems to only mildly follow the patterns drawn on the left hand side. It still seems sort of useful but once it's running GAN speeds (30-60fps) and adheres more strongly to the input is when people will find it both useful and a joy to work with. Based on my general experience with text-to-image stuff, I assume this lack of adherence isn't always the case? Maybe another demo could show what it's like under ideal conditions. edit: Webcam demo https://github.com/radames/Real-Time-Latent-Consistency-Model?tab=readme-ov-file#demo-on-hugging-face https://github.com/radames/Real-Time-Latent-Consistency-Mode... It does however exhibit similar issues but the realtime constraints of live video make it quit interesting.
- dvrp 3y agoI agree with your points but I think this is the beginning of something bigger
- ShamelessC 3y agoDefinitely. I've been looking at all this stuff way too closely for awhile now so I doubt my opinion is representative. Looks like there's positive reception coming from others in the comments here. I think the general cycle of "new model/technique - rush to implement - immediately obsoleted by another new model/technique - rush to implement..." has me a little burned out. Sorry about the dismissive comment.
- 3y ago
- TheRealNGenius 3y ago[dead]
- percentcer 3y agoStill too slow to be used for real work but it's cool to see progress
- dvrp 3y agoReally? Some users are already using KREA professionally. FCB for example.
- percentcer 3y ago...yes? If you're marketing a tool as being "real time" I'd expect it to not have this much latency (e.g. when the blue is added to the mushroom). Imagine if Photoshop made you wait a second between each brush stroke.
- dvrp 3y agoThat’s how it started!
- ncr100 3y agoI disagree. A compromise: How about near-real time, would that pass under your BS-threshold?
- percentcer 3y agoNot sure what you mean by my BS-threshold, but yes I would describe this tool as "near real time". It has too much latency to be used as a professional tool for artists, though perhaps that is not the intended audience.
- dvrp 3y agoOur users are already using our tool professionally—even though latency is >5s at times. I have to disagree with you. Of course, people will use it in a weird way—which is exciting.
- 3y ago
- gmerc 3y agoNvidia had a solution like this several years ago. Something something studio.
- dvrp 3y agothat’s kind of different though
- pandominium 3y agoIs it different in the underlying methods and models that are used? This is the Nvidia tool: https://www.nvidia.com/en-us/studio/canvas/ https://www.nvidia.com/en-us/studio/canvas/
- dvrp 3y agoYup! It is different; that one is based on GAN models. That's why you can only use what they let you paint. In our demo, you can put whatever text you want. And we want to make it so that you can assign a (custom) label to each shape. So you can type what each shape represents.
- iandanforth 3y agoThis is awesome. For future inputs please give me a 3d scene with camera controls into which I can drop primitive shapes, and human pose dolls so that I can frame shots like I would with a camera. I can't draw so even figuring out how to suggest "draw the image from the top left as the person looks over their shoulder" to the model is something I struggle with.
- dvrp 3y agoWe haven't even deployed an open beta of this and you're already asking for 3D? I'll never get tired of seeing the adaptation capabilities of human nature! (But of course, we'd love to work on this!)
- mungoman2 3y agoCome on, this is the wrong attitude. What they asked for could be prototyped in a day to see if it makes sense. Have the left image in the demo be fed from a rendering from a simple 3d scene of a camera and simple colored primitives.
- lovepronmostly 3y agoGreat idea! It takes a surprising amount of artist know how to illustrate actions using stick figures but probably less so with something like this https://www.ikea.com/jp/ja/p/gestalta-artists-dummy-natural-80257609/ https://www.ikea.com/jp/ja/p/gestalta-artists-dummy-natural-... Or maybe just pose yourself and upload a picture, trace it?
- vicentwu 3y agoCool! The real-time feedback will have enormous ramifications on the art creation workflows.
- dvrp 3y agoExactly! Everything will change. And this is without taking into account WebGPU and other advances in adjacent fields.
- radarsat1 3y agoI searched for "latent consistency models" out of curiosity, I guess this is the project (with paper link): https://github.com/luosiallen/latent-consistency-model https://github.com/luosiallen/latent-consistency-model
- smusamashah 3y agoSD community has been experimenting with Realtime generation using LCM for pas few weeks now https://www.reddit.com/r/StableDiffusion/comments/17ovb4j/this_is_the_future_of_digital_painting_i_combined/ https://www.reddit.com/r/StableDiffusion/comments/17ovb4j/th... https://www.reddit.com/r/StableDiffusion/comments/17kvpxn/realtime_latent_consistency_model_demos/ https://www.reddit.com/r/StableDiffusion/comments/17kvpxn/re... https://www.reddit.com/r/StableDiffusion/comments/17kekea/demo_of_realtime15fps_camera_capture_plus_sd/ https://www.reddit.com/r/StableDiffusion/comments/17kekea/de... https://www.reddit.com/r/StableDiffusion/comments/17ecdab/we_are_now_at_10_frames_a_second_512x512_with/k63nwyk/ https://www.reddit.com/r/StableDiffusion/comments/17ecdab/we... There are more examples on that subreddit. Given the pace at which features are added to popular automatic1111 repo, this will be added soon allowing you to try this for free on your own machine.
- smusamashah 3y agoSomeone just shared this on twitter https://huggingface.co/spaces/latent-consistency/super-fast-lcm-lora-sd1.5 https://huggingface.co/spaces/latent-consistency/super-fast-... which you can play with right now in your browser.
- dvrp 3y agoI think we are going to see insane things before end of year :)