5 ms·

Nvidia's banking on TensorRT to expand its generative AI dominance

19 points by webmaven 3y ago

brucethemoose2 3y ago
TensorRT, like many other frameworks, has always been really cool in GenAI demos and essentially useless and unused because no one integrates those demos into the more fleshed out projects. And TensorRT in particular is a pain for end-users to install correctly, especially when they gate parts of the stack behind sign-up walls.
- vhold 3y ago
  That's what this article is about, they're doing the integration work. An integration with a very popular tool is already available for use. I've personally tested it and did get a 2x speed up. I didn't have to sign-up for anything. https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT
  - brucethemoose2 3y ago
    Ugh... I know everyone uses automatic, but I don't really like automatic. Is janky as heck, especially with SDXL, especially if you want to dig into the code and make some manual hacks. Still, you are right, a native a1111 extensions is quite impressive and exactly the kind of thing Nvidia should pursue.
    - qiine 3y ago
      What do you use instead? confyUI ?
      - nickthegreek 3y ago
        ComfyUI or fooocus handle SDXL way better than A1111 with 10gb VRAM or under. While I really like A1111, it just isnt usable with XL for me.
        brucethemoose2 3y ago
        I run them on a 3090, and fooocus is still way better.
      - brucethemoose2 3y ago
        Fooocus MRE for SDXL. The quality is just so much better than a1111 or vlad's fork. And its stable, with a good UI: https://github.com/MoonRide303/Fooocus-MRE/commits/moonride-main https://github.com/MoonRide303/Fooocus-MRE/commits/moonride-... I use VoltaML for SD 1.5: https://github.com/VoltaML/voltaML-fast-stable-diffusion/tree/experimental https://github.com/VoltaML/voltaML-fast-stable-diffusion/tre... Next on my todo list is trying ComfyUI with AIT: https://github.com/FizzleDorf/ComfyUI-AIT https://github.com/FizzleDorf/ComfyUI-AIT
        qiine 3y ago
        so many projects... its overwhelming
        brucethemoose2 3y ago
        FYI I just learned that Fooocus-MRE has been depreciated, but many of the same options are now in the debug menu upstream. ...Yes, it is impossible to keep up, lol
    - diggan 3y ago
      diffusers also has example usage for TensorRT in the Stable Diffusion context: https://github.com/huggingface/diffusers/blob/85dccab7fd280cf085cc8007decc458d5fe0f032/examples/community/stable_diffusion_tensorrt_txt2img.py https://github.com/huggingface/diffusers/blob/85dccab7fd280c... Seems those particular examples were submitted by NVIDIA employee too: https://github.com/huggingface/diffusers/commit/9d44e2fb6600e80f410b2c05139c001fb0fa9794 https://github.com/huggingface/diffusers/commit/9d44e2fb6600... Shouldn't be too hard to adopt to other similar pipelines if you've used diffusers before.
      - brucethemoose2 3y ago
        That's a good example of a "hit and run" demo. None of the community Diffusers pipelines I know of have successfully integrated this. VoltaML made a valiant effort and gave up in favor of Facebooks AITemplate (for now).
        diggan 3y ago
        Worked for me and was easy to integrate, but I use diffusers directly. Not sure what the others are doing, but if you can manage to implement your own pipeline with diffusers, you should be able to integrate the TensorRT pipeline with relative ease. https://github.com/VoltaML/voltaML https://github.com/VoltaML/voltaML seems to have successfully integrated it as far as can tell, it says TensorRT is one of the compatible backends.
        brucethemoose2 3y ago
        VoltaML abandoned it (but may revisit it) and replaced it with AIT. The problem with a vanilla diffusers pipe is that you miss out on all the cool community augmentations that really make sd shine. Crazy img2img stuff aside, some random examples include: - Fooocus's implementation preserves 'momentum' when switching from the sdxl base model to the refiner. - freeu augmentation - tomesd - fooocus image style presets, with known styles from the dataset strait from SAI and others. - auto prompt expansion - performance hacks not in vanilla diffusers
71a54xd 3y ago
Did they also release new drivers for linux that carry these features? I'm going to really not be happy if you have to use Winblows for these...
- diggan 3y ago
  I don't think you have to use any new drivers to take advantage of it? At least the TensorRT example from diffusers run on my 535.113.01 drivers (Linux).
- brucethemoose2 3y ago
  TensorRT was originally linux (or WSL) only. Windows support is one of the new revelations. Its not like ROCM where official consumer gpu support is only on Windows.
  - Tommstein 3y ago
    > Its not like ROCM where official consumer gpu support is only on Windows. I don't follow AMD's GPUs, but lolwat. Is this really their master plan to finally catch up to NVIDIA? Those antitrust investigators in France should be given this as Exhibit A for why the rest of the industry is so far behind NVIDIA, not NVIDIA misbehavior.
    - brucethemoose2 3y ago
      AMD has lots of blame for underbaking rocm, but the problem isnt that simple either. Nvidia essentially shares their architecture across gaming GPUs and servers. Whatever work is done for A100s is "inherited" by the 3000 series and such. AMD, on the other hand, split their architecture between desktop (rdna) and server (cdna).
  - 65a 3y ago
    These work completely fine on linux.
wejick 3y ago
I'm still don't understand. The opening of the article seems discussing about consumer end use cases. But then H100 comes to the discussion. So TensorRT SDK is targeted to the end user usecases integration or not?
- brucethemoose2 3y ago
  That is TheVerge being TheVerge and largely missing the original post: https://blogs.nvidia.com/blog/2023/10/17/tensorrt-llm-windows-stable-diffusion-rtx/ https://blogs.nvidia.com/blog/2023/10/17/tensorrt-llm-window...

19 points by webmaven 3y ago

TensorRT, like many other frameworks, has always been really cool in GenAI demos and essentially useless and unused because no one integrates those demos into the more fleshed out projects. And TensorRT in particular is a pain for end-users to install correctly, especially when they gate parts of the stack behind sign-up walls.

vhold 3y ago
That's what this article is about, they're doing the integration work. An integration with a very popular tool is already available for use. I've personally tested it and did get a 2x speed up. I didn't have to sign-up for anything. https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT
- brucethemoose2 3y ago
  Ugh... I know everyone uses automatic, but I don't really like automatic. Is janky as heck, especially with SDXL, especially if you want to dig into the code and make some manual hacks. Still, you are right, a native a1111 extensions is quite impressive and exactly the kind of thing Nvidia should pursue.
  - qiine 3y ago
    What do you use instead? confyUI ?
    - nickthegreek 3y ago
      ComfyUI or fooocus handle SDXL way better than A1111 with 10gb VRAM or under. While I really like A1111, it just isnt usable with XL for me.
      - brucethemoose2 3y ago
        I run them on a 3090, and fooocus is still way better.
    - brucethemoose2 3y ago
      Fooocus MRE for SDXL. The quality is just so much better than a1111 or vlad's fork. And its stable, with a good UI: https://github.com/MoonRide303/Fooocus-MRE/commits/moonride-main https://github.com/MoonRide303/Fooocus-MRE/commits/moonride-... I use VoltaML for SD 1.5: https://github.com/VoltaML/voltaML-fast-stable-diffusion/tree/experimental https://github.com/VoltaML/voltaML-fast-stable-diffusion/tre... Next on my todo list is trying ComfyUI with AIT: https://github.com/FizzleDorf/ComfyUI-AIT https://github.com/FizzleDorf/ComfyUI-AIT
      - qiine 3y ago
        so many projects... its overwhelming
        brucethemoose2 3y ago
        FYI I just learned that Fooocus-MRE has been depreciated, but many of the same options are now in the debug menu upstream. ...Yes, it is impossible to keep up, lol
  - diggan 3y ago
    diffusers also has example usage for TensorRT in the Stable Diffusion context: https://github.com/huggingface/diffusers/blob/85dccab7fd280cf085cc8007decc458d5fe0f032/examples/community/stable_diffusion_tensorrt_txt2img.py https://github.com/huggingface/diffusers/blob/85dccab7fd280c... Seems those particular examples were submitted by NVIDIA employee too: https://github.com/huggingface/diffusers/commit/9d44e2fb6600e80f410b2c05139c001fb0fa9794 https://github.com/huggingface/diffusers/commit/9d44e2fb6600... Shouldn't be too hard to adopt to other similar pipelines if you've used diffusers before.
    - brucethemoose2 3y ago
      That's a good example of a "hit and run" demo. None of the community Diffusers pipelines I know of have successfully integrated this. VoltaML made a valiant effort and gave up in favor of Facebooks AITemplate (for now).
      - diggan 3y ago
        Worked for me and was easy to integrate, but I use diffusers directly. Not sure what the others are doing, but if you can manage to implement your own pipeline with diffusers, you should be able to integrate the TensorRT pipeline with relative ease. https://github.com/VoltaML/voltaML https://github.com/VoltaML/voltaML seems to have successfully integrated it as far as can tell, it says TensorRT is one of the compatible backends.
        brucethemoose2 3y ago
        VoltaML abandoned it (but may revisit it) and replaced it with AIT. The problem with a vanilla diffusers pipe is that you miss out on all the cool community augmentations that really make sd shine. Crazy img2img stuff aside, some random examples include: - Fooocus's implementation preserves 'momentum' when switching from the sdxl base model to the refiner. - freeu augmentation - tomesd - fooocus image style presets, with known styles from the dataset strait from SAI and others. - auto prompt expansion - performance hacks not in vanilla diffusers

71a54xd 3y ago

Did they also release new drivers for linux that carry these features? I'm going to really not be happy if you have to use Winblows for these...

diggan 3y ago
I don't think you have to use any new drivers to take advantage of it? At least the TensorRT example from diffusers run on my 535.113.01 drivers (Linux).
brucethemoose2 3y ago
TensorRT was originally linux (or WSL) only. Windows support is one of the new revelations. Its not like ROCM where official consumer gpu support is only on Windows.
- Tommstein 3y ago
  > Its not like ROCM where official consumer gpu support is only on Windows. I don't follow AMD's GPUs, but lolwat. Is this really their master plan to finally catch up to NVIDIA? Those antitrust investigators in France should be given this as Exhibit A for why the rest of the industry is so far behind NVIDIA, not NVIDIA misbehavior.
  - brucethemoose2 3y ago
    AMD has lots of blame for underbaking rocm, but the problem isnt that simple either. Nvidia essentially shares their architecture across gaming GPUs and servers. Whatever work is done for A100s is "inherited" by the 3000 series and such. AMD, on the other hand, split their architecture between desktop (rdna) and server (cdna).
- 65a 3y ago
  These work completely fine on linux.

wejick 3y ago

I'm still don't understand. The opening of the article seems discussing about consumer end use cases. But then H100 comes to the discussion. So TensorRT SDK is targeted to the end user usecases integration or not?

brucethemoose2 3y ago
That is TheVerge being TheVerge and largely missing the original post: https://blogs.nvidia.com/blog/2023/10/17/tensorrt-llm-windows-stable-diffusion-rtx/ https://blogs.nvidia.com/blog/2023/10/17/tensorrt-llm-window...