4 ms·

I accidentally stumbled upon this and did not expect such a speedup. It seems anything less than cu118 does not properly support the RTX4090 (or H100). Bumping

I accidentally stumbled upon this and did not expect such a speedup. It seems anything less than cu118 does not properly support the RTX4090 (or H100). Bumping to cuda12.2 with pytorch2.0.1+cu118 made my SDXL go 50% faster and ESRGAN 80% faster on the 4090.

latchkey 3y ago
Good find!
brucethemoose2 3y ago
You can also run PyTorch cu121 nightly builds, These also allow `torch.compile` to function properly with dynamic input, which should net another 30%+ boost to SD.
- voz_ 3y ago
  This comment brings a tear to my eye.
  - doctorpangloss 3y ago
    The underlying problem is the community's decision to make users manage this in the first place. This is an example of a setup.py that correctly installs the accelerated PyTorch for your platform: https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19866a0e2d1354a9afc169c627904/setup.py https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19... As you can see, never merged. For philosophical reasons I believe. The author wanted to merge it earlier and changed his mind. Like why make end users deal with this at all? The ROI from a layperson choosing these details is very low. Python has a packaging problem, this is well known. Fixing setuptools would be highest yield. Other package tooling can't install PyTorch, for example: https://github.com/python-poetry/poetry/issues/6409#issuecomment-1483932932 https://github.com/python-poetry/poetry/issues/6409#issuecom.... PyTorch itself is wonkily packaged. But I'm sure they have a good reason for this. Anyway, it goes to show that you can put a huge amount of effort into fixing this particular problem that everyone touching this technology has, and the maintainers everywhere will go nowhere with it. And I don't think this is a "me" problem, because there is so much demand for packaging PyTorch correctly - all the easy UIs, etc.
    - brucethemoose2 3y ago
      > But I'm sure they have a good reason for this. CUDA and ROCM make this an intractable problem. Basically there is no way to sanely package everything users need, and the absolutely enormous, cude/rocm versioned pytorch packages with missing libs are already a compromise. TBH the whole ecosystem is not meant to be for end user inference anyway.
    - voz_ 3y ago
      Sorry, no idea what you are talking about. I am talking about dynamic shapes in torch.compile. You seem to be talking about software packaging. You also make heavy use of the word "this" without it being clear what "this" is.
      - brucethemoose2 3y ago
        The two most popular stable diffusion UIs (automatic1111 and comfy) have longstanding issues with a few known but poorly documented bugs, like the ADA performance issue. For instance, the torch.compile thing we are talking about is (last I checked) totally irrelevant for those UIs because they are still using the Stability AI implementation, not diffusers package that Huggingface checks for graph breaks. This may extend to SDXL.
- cheald 3y ago
  Is there a trick to getting pytorch+cu121 and xformers to play nicely together? All the xformers packages I can find are torch==2.01+cu118. Edit: After a bit more research it looks like scaled dot product attention in Pytorch 2 provides much the same benefit as xformers without the need for xformers proper. Nice.
  - brucethemoose2 3y ago
    xformers has to match the PyTorch build. For PyTorch nightly, you need to build from source. xformers still has a tiny performance benefit (especially at higher resolutions IIRC), but yeah, PyTorch's SDP is good.
- vans554 3y ago
  Pretty interesting. Using nightly + cu121 im getting 8.18 it/s, another 5% improvement vs 7.78 that cu118 gave.
hospitalJail 3y ago
This was one of the reasons I skipped the 4090. So few people have the technology that I knew I'd be spending significant time figuring out solutions to problems. The other reason is that I'd wait a few years and get some 6090 with 4x the VRAM.
- VHRanger 3y ago
  I doubt future generations will up the consumerarket cards past 24gb. They know it's a bottleneck for LM training and inference, so they'll want to extract value by reserving it for the professional line cards

latchkey 3y ago

Good find!

brucethemoose2 3y ago

You can also run PyTorch cu121 nightly builds, These also allow `torch.compile` to function properly with dynamic input, which should net another 30%+ boost to SD.

voz_ 3y ago
This comment brings a tear to my eye.
- doctorpangloss 3y ago
  The underlying problem is the community's decision to make users manage this in the first place. This is an example of a setup.py that correctly installs the accelerated PyTorch for your platform: https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19866a0e2d1354a9afc169c627904/setup.py https://github.com/comfyanonymous/ComfyUI/blob/9aeaac4af5e19... As you can see, never merged. For philosophical reasons I believe. The author wanted to merge it earlier and changed his mind. Like why make end users deal with this at all? The ROI from a layperson choosing these details is very low. Python has a packaging problem, this is well known. Fixing setuptools would be highest yield. Other package tooling can't install PyTorch, for example: https://github.com/python-poetry/poetry/issues/6409#issuecomment-1483932932 https://github.com/python-poetry/poetry/issues/6409#issuecom.... PyTorch itself is wonkily packaged. But I'm sure they have a good reason for this. Anyway, it goes to show that you can put a huge amount of effort into fixing this particular problem that everyone touching this technology has, and the maintainers everywhere will go nowhere with it. And I don't think this is a "me" problem, because there is so much demand for packaging PyTorch correctly - all the easy UIs, etc.
  - brucethemoose2 3y ago
    > But I'm sure they have a good reason for this. CUDA and ROCM make this an intractable problem. Basically there is no way to sanely package everything users need, and the absolutely enormous, cude/rocm versioned pytorch packages with missing libs are already a compromise. TBH the whole ecosystem is not meant to be for end user inference anyway.
  - voz_ 3y ago
    Sorry, no idea what you are talking about. I am talking about dynamic shapes in torch.compile. You seem to be talking about software packaging. You also make heavy use of the word "this" without it being clear what "this" is.
    - brucethemoose2 3y ago
      The two most popular stable diffusion UIs (automatic1111 and comfy) have longstanding issues with a few known but poorly documented bugs, like the ADA performance issue. For instance, the torch.compile thing we are talking about is (last I checked) totally irrelevant for those UIs because they are still using the Stability AI implementation, not diffusers package that Huggingface checks for graph breaks. This may extend to SDXL.
cheald 3y ago
Is there a trick to getting pytorch+cu121 and xformers to play nicely together? All the xformers packages I can find are torch==2.01+cu118. Edit: After a bit more research it looks like scaled dot product attention in Pytorch 2 provides much the same benefit as xformers without the need for xformers proper. Nice.
- brucethemoose2 3y ago
  xformers has to match the PyTorch build. For PyTorch nightly, you need to build from source. xformers still has a tiny performance benefit (especially at higher resolutions IIRC), but yeah, PyTorch's SDP is good.
vans554 3y ago
Pretty interesting. Using nightly + cu121 im getting 8.18 it/s, another 5% improvement vs 7.78 that cu118 gave.

hospitalJail 3y ago

This was one of the reasons I skipped the 4090. So few people have the technology that I knew I'd be spending significant time figuring out solutions to problems. The other reason is that I'd wait a few years and get some 6090 with 4x the VRAM.

VHRanger 3y ago
I doubt future generations will up the consumerarket cards past 24gb. They know it's a bottleneck for LM training and inference, so they'll want to extract value by reserving it for the professional line cards