4 ms·
I hope someday there’s a version of this or something comparable to it that can run on <8gb consumer hardware. The main selling point of Stable Diffusion was it
by jrflowers 3y ago
I hope someday there’s a version of this or something comparable to it that can run on <8gb consumer hardware. The main selling point of Stable Diffusion was its ability to run in that environment.
- minsc_and_boo 3y agoI feel like this is the greatest demand for LLMs at the moment too. It's hard to believe we're only 8 months into this industry, so I imagine we'll start seeing smaller footprints soon.
- simbolit 3y ago8 months from what point? Gpt3 is 36 months old. Dalle-e is 28 months old. Even StableDiffusion is like 11 months old.
- minsc_and_boo 3y agoFair, I should have said 8 months since the market exploded.
- brucethemoose2 3y agoWe already do. MLC-LLM and Llama.cpp have Vulkan/OpenCL/Metal 3 bit implementations. That can run llama 7B (or maybe even 13b?) in 8GB. TBH devices just need more ram for coherent output though. Llama 13b and 33b are so much "smarter" and more coherent than 7B with 3 bit quant.
- TeddyDD 3y ago13b q5 llama report „total VRAM used: 8321 MB” so 3bit will most likely fit into 8GB.
- naillo 3y agoYou can do this if you select the `pipe.enable_model_cpu_offload()` option. See this https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9 https://huggingface.co/stabilityai/stable-diffusion-xl-base-...
- liuliu 3y agoSDXL 0.9 runs on iPad Pro 8GiB just fine.
- JeffeFawkes 3y agoIs this using Draw Things, or another app? Did you have to quantize the model first?
- liuliu 3y agoYeah, Draw Things. It will be submitted as soon as SDXL v1.0 weights available. Quantized model should run on iPhones (4GiB / 6GiB models), but we haven't done that yet. So no, these are just typical FP16 weights on iPad.
- JeffeFawkes 3y agoThanks! I guess I'll stick to running it on my Macbook for the time being until the quantized model gets uploaded. What kind of performance are you seeing with the FP16 weights on the iPad? I've run a few SD2.0-based (unquantized) models on my 2020 iPad Pro but it seems like it gets thermally throttled after a while.
- liuliu 3y agoWill be more info upon release. SDXL v0.9 performs generally the same as SD v1 / v2 on the same resolution. But because you tend to run it at larger resolution, you might feel it slower.
- brucethemoose2 3y agoThere are several papers on 4/8 bit quantization, and a few implementations for Vulkan/CUDA/ROCm compilation. TBH the UIs people run for SD 1.5 are pretty unoptimized.
- dragonwriter 3y ago> I hope someday there’s a version of this or something comparable to it that can run on <8gb consumer hardware. Someday is today: from the official announcement: “SDXL 1.0 should work effectively on consumer GPUs with 8GB VRAM or readily available cloud instances.” https://stability.ai/blog/stable-diffusion-sdxl-1-announcement https://stability.ai/blog/stable-diffusion-sdxl-1-announceme...
- capybara_2020 3y agoGive InvokeAI a try. https://github.com/invoke-ai/InvokeAI https://github.com/invoke-ai/InvokeAI Edit: Spec required from the documentation You will need one of the following: An NVIDIA-based graphics card with 4 GB or more VRAM memory. 6-8 GB of VRAM is highly recommended for rendering using the Stable Diffusion XL models An Apple computer with an M1 chip. An AMD-based graphics card with 4GB or more VRAM memory (Linux only), 6-8 GB for XL rendering.
- kristianp 3y agoThanks for the recommendation. As an aside, does this irritate anyone else? "You must have Python 3.9 or 3.10 installed on your machine. Earlier or later versions are not supported. Node.js also needs to be installed along with yarn" I don't like having to install npm when an existing dev stack (python) is already present.
- cmdr2 3y agoEasy Diffusion (previously cmdr2 UI) can run SDXL in 768x768 in about 7 GB of VRAM. And SDXL 512x512 in about 5 GB of VRAM. Regular SD can run in less than 2 GB of VRAM with Easy Diffusion. 1. Installation (no dependencies, python etc): https://github.com/easydiffusion/easydiffusion#installation https://github.com/easydiffusion/easydiffusion#installation 2. Enable beta to get access to SDXL: https://github.com/easydiffusion/easydiffusion/wiki/The-beta-channel#test-diffusers-in-beta https://github.com/easydiffusion/easydiffusion/wiki/The-beta... 3. Use the "Low" VRAM Usage model in the Settings tab.
- jrflowers 3y agoI should clarify that by <8gb I meant “less than 8gb”, which is what SD 1.5 and 2 were able to do. I’m aware that it can run on ==8gb.