9 ms·

why is it faster than latent diffusion models?

vipermu 3y ago
it uses a new technique called "consistency" that lets latent diffusion models to predict images in much fewer steps. some links here: - https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378 - https://arxiv.org/abs/2311.05556 https://arxiv.org/abs/2311.05556
billconan 3y ago
latent diffusion is an iterative process, the image becomes clearer one step at a time. The process can be viewed as a particle moving in the image space, one step at a time to its final position in the image space, which is the generated image. consistency model tries to predict the movement trajectory by providing the current position in the image space. Hence, what used to be a step-by-step process becomes a one-step process.
- dvrp 3y ago
  oh wow, never thought of it that way
  - swyx 3y ago
    > consistency model tries to predict the movement trajectory by providing the current position in the image space. Hence, what used to be a step-by-step process becomes a one-step process. no that wasnt a sufficient explanation for me. what is the prediction method here? why was diffusion necessary in the past? what tradeoffs does this approach have?
    - quadrature 3y ago
      from https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378 it sounds like its a form of distillation of an SDL model. So im guessing it can't be directly trained, but once you have a trained diffusion model you can distil a predictor which cuts out the iterative steps. While it can do 1-step the output quality looks a ton better with additional steps.
      - dvrp 3y ago
        You can train it directly. This is from the paper “An LCM demands merely 32 A100 GPUs Hours training for 2-step inference [...]”
        T-A 3y ago
        Let's quote the whole sentence, shall we? An LCM demands merely 32 A100 GPUs Hours training for 2-step inference, as de- picted in Figure 1. [1] Now let's look at the caption under Figure 1: LCMs can be distilled from any pre-trained Stable Diffusion (SD) in only 4,000 training steps (∼32 A100 GPU Hours) for generating high quality 768×768 resolution images in 2∼4 steps or even one step, significantly accelerating text-to-image generation. The training mentioned in your quote is distillation: it requires a previously trained SD model. You could have told just by reading the introduction: We propose a one-stage guided distillation method to efficiently convert a pretrained guided diffusion model into a latent consistency model by solving an augmented PF-ODE. [1] Section 4.2 in https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378
    - deleted 3y ago
      [deleted]
    - billconan 3y ago
      to my defense, if you look at the original paper https://arxiv.org/pdf/2303.01469.pdf https://arxiv.org/pdf/2303.01469.pdf the exact neural network used for the prediction method is omitted. apparently many neural networks can be used for this prediction method as long as they fulfill certain requirements. > why was diffusion necessary in the past? in the paper, one way to train a consistency model is distilling an existing diffusion model. But it can be trained from scratch too. "why was it necessary in the past " doesn't bother me that much. Before people know to use molding to make candles, they did it by dipping threads into wax. Why was thread dipping necessary? it's just a stepping stone of technology development.
      - dvrp 3y ago
        Exactly. The stepping stone way of seeing things reminds me a lot to the thesis behind the book “ Why Greatness Cannot Be Planned: The Myth of the Objective” (2015).
    - ativzzz 3y ago
      > what tradeoffs does this approach have? From playing around with it a bit locally, LCM is much, much faster, but generally the detail is much lower using the latest SDXL model. If your prompt is very simple, such as "a boy looking at the moon in a forest" it does pretty well. If your prompt is much more complex and asks for a lot more detail or uses other LoRas, it doesn't do nearly as well as other samplers and generates lower quality, worse matching images. These other samplers take 30-40 steps so it's several times slower. From what I've seen though, if you use control net, or passing some guideline images in and rely on a simple prompt, like an existing video that you're trying to change the style of, LCM can generate images in near real time like the OP on an RTX4090 and maybe slower cards on smaller/older models Another benefit is the decreased experimental time. So you can more quickly iterate over seeds to find output you like, and then maybe you can spend some time with other samplers/upscalers with that seed to make the result higher quality
      - dvrp 3y ago
        exactly. and there’s more things that you can do in terms of what you use as input for the NN!
      - stavros 3y ago
        Is it possible to use the seed from the LCM on the full-blown model to get a more detailed image, or is it a different latent space/decoder?
      - ttul 3y ago
        If you use a tool like ComfyUI, you can use LCM-LoRa to generate fast but not perfect outputs and then refine them with the old school sampler. I’ve been playing around and have found the quality of LCM to be excellent when used in combination with IP Adapter instead of text prompts for conditioning.
- foota 3y ago
  Would that motion vector have to include like a color delta?
  - billconan 3y ago
    if you are talking about latent diffusion, No, the "particle" is in a hyper-dimensional space, like for example a 10k-dimensional space. We are not supposed to interpret the meaning of that vector. and when that particle has moved to the right location, there is a decoder that converts it into an image. The decoder network knows how to interpret it.
    - dvrp 3y ago
      Exactly. I actually wrote a micro essay on Twitter the other day about the meaning of the classic encoder decoder network. It’s beautiful. But yeah! - For reference: https://x.com/asciidiego/status/1722544108252836119 https://x.com/asciidiego/status/1722544108252836119
    - stavros 3y ago
      Does this mean that you need this vector state in order to generate the next step? Ie I can't take an image (the pixels) and the prompt and run a few more steps on it?
orbital-decay 3y ago
The architecture is the same but it has an additional model trained on the sampling trajectory computation, speeding up the process. So instead of 20 iterations with the original sampler that solves a SDE, you could use 8, 4, or sometimes even 2 iterations with this new ML-based sampler. That's not a new idea, but the author of this adapter figured out how to train it efficiently using very few GPU-hours.
- jiggawatts 3y ago
  I wonder if they could do a real-time preview using 2-4 iterations, and then 'refine' the quality with a higher number of iterations over time, much like how 3D editors preview ray-tracing.
  - orbital-decay 3y ago
    I've lost the link, but IIRC someone on twitter demoed exactly that in Blender. "Neural rendering" in a 3D editor has some nice properties - the scene can be rendered into separate layers with separate depth maps and openpose skeletons generated from the rigs to serve as input for controlnets, objects can be tagged and tags composed into the prompt for the particular tile in case of tiled rendering, etc.
    - dvrp 3y ago
      probably not but are you talking about the guy who used control net and animatediff to make a video that looked relatively consistent?
      - orbital-decay 3y ago
        No, it was running in the Blender viewport using the original LCM Dreamshaper model.
  - dvrp 3y ago
    we’re thinking of doing that as well! but first we want to make sure we can get this in the hands of a tight cohort of creatives and see how they use it.

it uses a new technique called "consistency" that lets latent diffusion models to predict images in much fewer steps. some links here: - https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378 - https://arxiv.org/abs/2311.05556 https://arxiv.org/abs/2311.05556

billconan 3y ago

latent diffusion is an iterative process, the image becomes clearer one step at a time. The process can be viewed as a particle moving in the image space, one step at a time to its final position in the image space, which is the generated image. consistency model tries to predict the movement trajectory by providing the current position in the image space. Hence, what used to be a step-by-step process becomes a one-step process.

dvrp 3y ago
oh wow, never thought of it that way
- swyx 3y ago
  > consistency model tries to predict the movement trajectory by providing the current position in the image space. Hence, what used to be a step-by-step process becomes a one-step process. no that wasnt a sufficient explanation for me. what is the prediction method here? why was diffusion necessary in the past? what tradeoffs does this approach have?
  - quadrature 3y ago
    from https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378 it sounds like its a form of distillation of an SDL model. So im guessing it can't be directly trained, but once you have a trained diffusion model you can distil a predictor which cuts out the iterative steps. While it can do 1-step the output quality looks a ton better with additional steps.
    - dvrp 3y ago
      You can train it directly. This is from the paper “An LCM demands merely 32 A100 GPUs Hours training for 2-step inference [...]”
      - T-A 3y ago
        Let's quote the whole sentence, shall we? An LCM demands merely 32 A100 GPUs Hours training for 2-step inference, as de- picted in Figure 1. [1] Now let's look at the caption under Figure 1: LCMs can be distilled from any pre-trained Stable Diffusion (SD) in only 4,000 training steps (∼32 A100 GPU Hours) for generating high quality 768×768 resolution images in 2∼4 steps or even one step, significantly accelerating text-to-image generation. The training mentioned in your quote is distillation: it requires a previously trained SD model. You could have told just by reading the introduction: We propose a one-stage guided distillation method to efficiently convert a pretrained guided diffusion model into a latent consistency model by solving an augmented PF-ODE. [1] Section 4.2 in https://arxiv.org/abs/2310.04378 https://arxiv.org/abs/2310.04378
  - deleted 3y ago
    [deleted]
  - billconan 3y ago
    to my defense, if you look at the original paper https://arxiv.org/pdf/2303.01469.pdf https://arxiv.org/pdf/2303.01469.pdf the exact neural network used for the prediction method is omitted. apparently many neural networks can be used for this prediction method as long as they fulfill certain requirements. > why was diffusion necessary in the past? in the paper, one way to train a consistency model is distilling an existing diffusion model. But it can be trained from scratch too. "why was it necessary in the past " doesn't bother me that much. Before people know to use molding to make candles, they did it by dipping threads into wax. Why was thread dipping necessary? it's just a stepping stone of technology development.
    - dvrp 3y ago
      Exactly. The stepping stone way of seeing things reminds me a lot to the thesis behind the book “ Why Greatness Cannot Be Planned: The Myth of the Objective” (2015).
  - ativzzz 3y ago
    > what tradeoffs does this approach have? From playing around with it a bit locally, LCM is much, much faster, but generally the detail is much lower using the latest SDXL model. If your prompt is very simple, such as "a boy looking at the moon in a forest" it does pretty well. If your prompt is much more complex and asks for a lot more detail or uses other LoRas, it doesn't do nearly as well as other samplers and generates lower quality, worse matching images. These other samplers take 30-40 steps so it's several times slower. From what I've seen though, if you use control net, or passing some guideline images in and rely on a simple prompt, like an existing video that you're trying to change the style of, LCM can generate images in near real time like the OP on an RTX4090 and maybe slower cards on smaller/older models Another benefit is the decreased experimental time. So you can more quickly iterate over seeds to find output you like, and then maybe you can spend some time with other samplers/upscalers with that seed to make the result higher quality
    - dvrp 3y ago
      exactly. and there’s more things that you can do in terms of what you use as input for the NN!
    - stavros 3y ago
      Is it possible to use the seed from the LCM on the full-blown model to get a more detailed image, or is it a different latent space/decoder?
    - ttul 3y ago
      If you use a tool like ComfyUI, you can use LCM-LoRa to generate fast but not perfect outputs and then refine them with the old school sampler. I’ve been playing around and have found the quality of LCM to be excellent when used in combination with IP Adapter instead of text prompts for conditioning.
foota 3y ago
Would that motion vector have to include like a color delta?
- billconan 3y ago
  if you are talking about latent diffusion, No, the "particle" is in a hyper-dimensional space, like for example a 10k-dimensional space. We are not supposed to interpret the meaning of that vector. and when that particle has moved to the right location, there is a decoder that converts it into an image. The decoder network knows how to interpret it.
  - dvrp 3y ago
    Exactly. I actually wrote a micro essay on Twitter the other day about the meaning of the classic encoder decoder network. It’s beautiful. But yeah! - For reference: https://x.com/asciidiego/status/1722544108252836119 https://x.com/asciidiego/status/1722544108252836119
  - stavros 3y ago
    Does this mean that you need this vector state in order to generate the next step? Ie I can't take an image (the pixels) and the prompt and run a few more steps on it?

orbital-decay 3y ago

The architecture is the same but it has an additional model trained on the sampling trajectory computation, speeding up the process. So instead of 20 iterations with the original sampler that solves a SDE, you could use 8, 4, or sometimes even 2 iterations with this new ML-based sampler. That's not a new idea, but the author of this adapter figured out how to train it efficiently using very few GPU-hours.

jiggawatts 3y ago
I wonder if they could do a real-time preview using 2-4 iterations, and then 'refine' the quality with a higher number of iterations over time, much like how 3D editors preview ray-tracing.
- orbital-decay 3y ago
  I've lost the link, but IIRC someone on twitter demoed exactly that in Blender. "Neural rendering" in a 3D editor has some nice properties - the scene can be rendered into separate layers with separate depth maps and openpose skeletons generated from the rigs to serve as input for controlnets, objects can be tagged and tags composed into the prompt for the particular tile in case of tiled rendering, etc.
  - dvrp 3y ago
    probably not but are you talking about the guy who used control net and animatediff to make a video that looked relatively consistent?
    - orbital-decay 3y ago
      No, it was running in the Blender viewport using the original LCM Dreamshaper model.
- dvrp 3y ago
  we’re thinking of doing that as well! but first we want to make sure we can get this in the hands of a tight cohort of creatives and see how they use it.