3 ms·

Stable Diffusion Gets a Major Boost with RTX Acceleration

101 points by ortusdux 3y ago

ortusdux 3y ago
Link to the support page: https://nvidia.custhelp.com/app/answers/detail/a_id/5487 https://nvidia.custhelp.com/app/answers/detail/a_id/5487
- k12sosse 3y ago
  Thanks! Can't wait to get a minute to check this out. Promising 2x performance over pytorch+xformers sounds too good to be true for the same card. As someone with a lowly 10gb card sdxl is beyond my reach with a1111 it seems. It functions well enough in comfyui but I can't make anything but garbage with it in automatic. So I'm happy to see 1.5 gets a big boost, I know there's a million of us out there who can't quite squeeze SDXL out so the maturing of the "legacy" versions is a positive note to see.
  - thot_experiment 3y ago
    I haven't been paying attention for a few months, is there a lot of development happening around SDXL or are people still mostly focused on 1.5? My expectation was that SDXL would probably not see wide adoption for a while because it's hard to make up for the worse training data, is that bearing out?
    - GaggiX 3y ago
      >My expectation was that SDXL would probably not see wide adoption for a while because it's hard to make up for the worse training data Where do you read that SDXL has worse training data? They didn't disclose what the model was trained on like the previous models.
    - Veraticus 3y ago
      Most people (and models) still use 1.5 because of hardware and performance issues with SDXL.
    - dragonwriter 3y ago
      > I haven't been paying attention for a few months, is there a lot of development happening around SDXL or are people still mostly focused on 1.5? There's been a lot happening on SDXL, yes. > My expectation was that SDXL would probably not see wide adoption for a while because it's hard to make up for the worse training data, is that bearing out? What worse training data?
      - thot_experiment 3y ago
        > There's been a lot happening on SDXL, yes. Nice! Care to share any projects/advancements/gens you've found particularly compelling if you have anything handy? > What worse training data? I don't know this, but I think it's a pretty safe assumption given the state of 2.0/the political climate around this stuff, along with the comparative lower diversity of gens I've seen come out of SDXL. I haven't seen anything to the contrary but I'd love to be wrong.
  - iLoveOncall 3y ago
    I've used SDXL with a 3080 (10GB) mostly without issues. You have to give some command line parameters when launching it to reduce memory use. > --medvram-sdxl --no-half-vae
    - Filligree 3y ago
      "--no-half-vae" doesn't reduce memory usage, it forces A1111 to use 32-bit floats for the VAE decoding.
      - dragonwriter 3y ago
        --no-half-vae, IIRC, is required for the stock sdxl vae model because otherwise it frequently produces errors. I believe there’s a half-precision fixed VAE floating around and baked into many SDXL-derived checkpoints that avoids this problem.
  - DrSiemer 3y ago
    Runs just fine on my 8GB card. You probably did not swap the VAE. Without that it will only produce junk.
  - dragonwriter 3y ago
    > As someone with a lowly 10gb card sdxl is beyond my reach with a1111 it seems SDXL reportedly works at below 8GiB in A1111 with --low-vram, and at 8GiB to anything short of 16GiB with --med-vram-sdxl, and 16GiB and up with no options. > but I can't make anything but garbage with it in automatic. If you aren't getting out of memory errors with SDXL but the output sucks, its probably not a VRAM problem. The most likely thing, IME, is using the SDXL base model or another checkpoint without a baked VAE but still having A1111 configured to use an SD1.x VAE (or using one with a baked VAE but having A1111 configured to override it.) SDXL needs an SDXL-specific VAE, but A1111 doesn't bundle one, IIRC, you need to download it separately.
freitzkriesler2 3y ago
I want to make an RTX on joke but my card isn't supported...
- happycube 3y ago
  Mr. Miyagi: RTX on, RTX off...
neonbjb 3y ago
Its cool that this is starting to approach real time video territory (30 images per second, this claims close to 1 image/sec).
- 01100011 3y ago
  Once you reach a few fps, you can use other techniques to interpolate frames.
  - smusamashah 3y ago
    SD is trained on static images and don't have a clue about motion. Interpolation won't help with that. See deforum videos. They don't make any sense. Fast generation will help iterate faster though.
lxe 3y ago
Before you jump into installing the extension for your A1111, keep in mind the limitations: - You have to "pre-bake" each model (create optimized engines) for only a pre-defined set of resolutions - ~~No LoRA support~~ I guess there is a LoRA support, but you also will need to convert them to TRT It's been like this for quite a few months now, not sure what's new this time.
- KRAKRISMOTT 3y ago
  Why is LoRA not supported? Isn't it just an additional two matrix multiplications per layer?
  - Zetobal 3y ago
    You have to convert them and you need to adept the model loader in your tooling.
- dragonwriter 3y ago
  LoRA are explicitly supported, though you have to convert them, as well as the base model. This seems likely to be more useful for people with a "production" workflow with particular checkpoint, LoRA, & resolution combo than what I get the impression is the more common hobbyist situation of having a decently diverse collection of checkpoints and LoRA and fairly freely mixing and matching.
geuis 3y ago
Here's a direct link to the extension that's referenced in the page. https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT
ziddoap 3y ago
Played around with it, only a few dozen images so far, and the difference is pretty noticeable. On a 2060S, no launch arguments, going from ~4 it/s to ~7.5 it/s.
deleted 3y ago
[deleted]
ChatGTP 3y ago
Legit can't wait to generate images even faster.
hunkins 3y ago
For developers, TensorRT support also exists for the diffusers library via community pipelines. [1] It's limited, but if you're only supporting a subset of features, it can help. In general, these insane speed boosts come at the cost of bleeding edge features. [1] https://github.com/huggingface/diffusers/blob/28e8d1f6ec82a60a9b69554d83f02d385b79d1cc/examples/community/stable_diffusion_tensorrt_txt2img.py#L590 https://github.com/huggingface/diffusers/blob/28e8d1f6ec82a6...
- falsedang 3y ago
  [flagged]
kopyl 3y ago
Useless. I don't see instructions neither for Linux nor for Mac. + there are no drivers for Linux or Mac.

101 points by ortusdux 3y ago

Link to the support page: https://nvidia.custhelp.com/app/answers/detail/a_id/5487 https://nvidia.custhelp.com/app/answers/detail/a_id/5487

k12sosse 3y ago
Thanks! Can't wait to get a minute to check this out. Promising 2x performance over pytorch+xformers sounds too good to be true for the same card. As someone with a lowly 10gb card sdxl is beyond my reach with a1111 it seems. It functions well enough in comfyui but I can't make anything but garbage with it in automatic. So I'm happy to see 1.5 gets a big boost, I know there's a million of us out there who can't quite squeeze SDXL out so the maturing of the "legacy" versions is a positive note to see.
- thot_experiment 3y ago
  I haven't been paying attention for a few months, is there a lot of development happening around SDXL or are people still mostly focused on 1.5? My expectation was that SDXL would probably not see wide adoption for a while because it's hard to make up for the worse training data, is that bearing out?
  - GaggiX 3y ago
    >My expectation was that SDXL would probably not see wide adoption for a while because it's hard to make up for the worse training data Where do you read that SDXL has worse training data? They didn't disclose what the model was trained on like the previous models.
  - Veraticus 3y ago
    Most people (and models) still use 1.5 because of hardware and performance issues with SDXL.
  - dragonwriter 3y ago
    > I haven't been paying attention for a few months, is there a lot of development happening around SDXL or are people still mostly focused on 1.5? There's been a lot happening on SDXL, yes. > My expectation was that SDXL would probably not see wide adoption for a while because it's hard to make up for the worse training data, is that bearing out? What worse training data?
    - thot_experiment 3y ago
      > There's been a lot happening on SDXL, yes. Nice! Care to share any projects/advancements/gens you've found particularly compelling if you have anything handy? > What worse training data? I don't know this, but I think it's a pretty safe assumption given the state of 2.0/the political climate around this stuff, along with the comparative lower diversity of gens I've seen come out of SDXL. I haven't seen anything to the contrary but I'd love to be wrong.
- iLoveOncall 3y ago
  I've used SDXL with a 3080 (10GB) mostly without issues. You have to give some command line parameters when launching it to reduce memory use. > --medvram-sdxl --no-half-vae
  - Filligree 3y ago
    "--no-half-vae" doesn't reduce memory usage, it forces A1111 to use 32-bit floats for the VAE decoding.
    - dragonwriter 3y ago
      --no-half-vae, IIRC, is required for the stock sdxl vae model because otherwise it frequently produces errors. I believe there’s a half-precision fixed VAE floating around and baked into many SDXL-derived checkpoints that avoids this problem.
- DrSiemer 3y ago
  Runs just fine on my 8GB card. You probably did not swap the VAE. Without that it will only produce junk.
- dragonwriter 3y ago
  > As someone with a lowly 10gb card sdxl is beyond my reach with a1111 it seems SDXL reportedly works at below 8GiB in A1111 with --low-vram, and at 8GiB to anything short of 16GiB with --med-vram-sdxl, and 16GiB and up with no options. > but I can't make anything but garbage with it in automatic. If you aren't getting out of memory errors with SDXL but the output sucks, its probably not a VRAM problem. The most likely thing, IME, is using the SDXL base model or another checkpoint without a baked VAE but still having A1111 configured to use an SD1.x VAE (or using one with a baked VAE but having A1111 configured to override it.) SDXL needs an SDXL-specific VAE, but A1111 doesn't bundle one, IIRC, you need to download it separately.

freitzkriesler2 3y ago

I want to make an RTX on joke but my card isn't supported...

happycube 3y ago
Mr. Miyagi: RTX on, RTX off...

neonbjb 3y ago

Its cool that this is starting to approach real time video territory (30 images per second, this claims close to 1 image/sec).

01100011 3y ago
Once you reach a few fps, you can use other techniques to interpolate frames.
- smusamashah 3y ago
  SD is trained on static images and don't have a clue about motion. Interpolation won't help with that. See deforum videos. They don't make any sense. Fast generation will help iterate faster though.

lxe 3y ago

Before you jump into installing the extension for your A1111, keep in mind the limitations: - You have to "pre-bake" each model (create optimized engines) for only a pre-defined set of resolutions - ~~No LoRA support~~ I guess there is a LoRA support, but you also will need to convert them to TRT It's been like this for quite a few months now, not sure what's new this time.

KRAKRISMOTT 3y ago
Why is LoRA not supported? Isn't it just an additional two matrix multiplications per layer?
- Zetobal 3y ago
  You have to convert them and you need to adept the model loader in your tooling.
dragonwriter 3y ago
LoRA are explicitly supported, though you have to convert them, as well as the base model. This seems likely to be more useful for people with a "production" workflow with particular checkpoint, LoRA, & resolution combo than what I get the impression is the more common hobbyist situation of having a decently diverse collection of checkpoints and LoRA and fairly freely mixing and matching.

geuis 3y ago

Here's a direct link to the extension that's referenced in the page. https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT

ziddoap 3y ago

Played around with it, only a few dozen images so far, and the difference is pretty noticeable. On a 2060S, no launch arguments, going from ~4 it/s to ~7.5 it/s.

deleted 3y ago

[deleted]

ChatGTP 3y ago

Legit can't wait to generate images even faster.

hunkins 3y ago

For developers, TensorRT support also exists for the diffusers library via community pipelines. [1] It's limited, but if you're only supporting a subset of features, it can help. In general, these insane speed boosts come at the cost of bleeding edge features. [1] https://github.com/huggingface/diffusers/blob/28e8d1f6ec82a60a9b69554d83f02d385b79d1cc/examples/community/stable_diffusion_tensorrt_txt2img.py#L590 https://github.com/huggingface/diffusers/blob/28e8d1f6ec82a6...

falsedang 3y ago
[flagged]

kopyl 3y ago

Useless. I don't see instructions neither for Linux nor for Mac. + there are no drivers for Linux or Mac.