4 ms·

Thanks. Based on your comment, I added a note that the GPU recommendations are for GPU clouds (where if you're running stable diffusion, ~$0.50/hr for a 4090 i

by tikkun 3y ago

Thanks. Based on your comment, I added a note that the GPU recommendations are for GPU clouds (where if you're running stable diffusion, ~$0.50/hr for a 4090 is pretty practical, and it's not worth going to a cheaper card for most), and I added this note re local GPUs now: If you're using a local GPU: * Same as above, but you probably won't be able to train or fine tune an LLM! * Most of the open LLMs have versions available that can run on lower VRAM cards e.g. [Falcon on a 40GB card](https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ) * Thanks Bruce for prompting me to add this section

brucethemoose2 3y ago
The UIs/containers people use for 4090s specifically have some kind of performance issue with Stable Diffusion, last I checked. It runs no faster than a 3090. I think this can be solved by running RTX 4000 cards with CUDA 12.1 and pytorch nightly cu121... But thats a very uncommon setup. Also, if we are talking cloud deployments, cheap CPU instances with 32GB-64GB of RAM are a somewhat sane way to test LLMs, thanks to llama.cpp. A GPU will give you better throughput, of course. But CPU support for Falcon and MPT specifically are in flux, see this post for instance: https://huggingface.co/TheBloke/mpt-30B-instruct-GGML https://huggingface.co/TheBloke/mpt-30B-instruct-GGML
- tikkun 3y ago
  For people reading using a 4090, this comment from March might help: https://www.reddit.com/r/StableDiffusion/comments/y71q5k/comment/jcm67lu/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/StableDiffusion/comments/y71q5k/com...