4 ms·
Llama.cpp/ggml is uniquely suited to llms. The memory requirements are huge, quantization is effective, and token generation is surprisingly serial and bandwidt
by brucethemoose2 3y ago
Llama.cpp/ggml is uniquely suited to llms. The memory requirements are huge, quantization is effective, and token generation is surprisingly serial and bandwidth bound, making it good for CPUs, and an even better fit for ggml's unique pipelined CPU/GPU inference.
...But Stable Diffusion is not the same. It doesn't quantize as well, the unet is very compute intense, and batched image generation is effective and useful to single users. Its a better fit for GPUs/IGPs. Additionally, it massively benefits from the hackability of the Python implementations.
I think ML compilation to executables is the way for SD. AITemplate is already blazing fast [1], and TVM Vulkan is very promising if anyone will actually flesh out the demo implementation [2]. And they preserve most of the hackability of the pure PyTorch implementations.
1: https://github.com/VoltaML/voltaML-fast-stable-diffusion https://github.com/VoltaML/voltaML-fast-stable-diffusion
2: https://github.com/mlc-ai/web-stable-diffusion https://github.com/mlc-ai/web-stable-diffusion
- WinLychee 3y agoThe above project somewhat supports GPUs if you pass the correct GGML compile flags to it. `GGML_CUBLAS` for example is supported when compiling. You get a decent speedup relative to pure C/C++.
- brucethemoose2 3y agoInteresting. It still doesn't seem to be very quick: https://github.com/leejet/stable-diffusion.cpp/issues/6 https://github.com/leejet/stable-diffusion.cpp/issues/6 But don't get me wrong, I look forward to playing with ggml SD and its development.
- WinLychee 3y agoYeah for comparison, `tinygrad` takes a little over a second per iteration on my machine. https://github.com/tinygrad/tinygrad/blob/master/examples/stable_diffusion.py https://github.com/tinygrad/tinygrad/blob/master/examples/st...
- brucethemoose2 3y agoIs that on GPU or CPU? 1 it/s would be very respectable on CPU. The fastest implementation on my 2060 laptop is AITemplate, being about 2x faster than pure optimized HF diffusers.
- WinLychee 3y agoThat was on GPU, and there are various CPU implementations (e.g. based on Tencent/ncnn) on github that have similar runtime (1-3s / iteration).
- voz_ 3y agoIirc we had good speedups on it with torch.compile, and I remember working on it. Let me see if I can find numbers…
- brucethemoose2 3y agoIts about 20-40% depending on the GPU, from my tests. And only very recent builds of torch 2.1 (with dynamic input) work properly, and it still doesn't like certain input changes, or augmentations like controlnet. AIT is the most usable compiled implementation I have personally tested, but SHARK (running IREE/MLIR/Vulkan) and torch-mlir are said to be very good. Hidet is promising but doesn't really work yet. TVM doesn't have a complete implementation outside of the WebGPU demo.
- voz_ 3y agoTry head of master. If there’s any bugs or graph breaks you hit, lmk, I can take a look. My numbers say 71% with a few custom hacks. Glad the dynamic stuff is working out tho!
- brucethemoose2 3y agoI will, thanks! I have been away for a month, but I will start testing it again later and submit some issues I run into.
- voz_ 3y agoMy username without underscore, at meta. Email me any bugs, I can help file them on GH and lend a hand fixing.
- skykooler 3y agoOn the other hand, this is nice for anyone who wants to play with these networks locally and does not have a nvidia GPU with 6+ gigabytes of VRAM. I can run this on an old laptop, even if it takes a while.
- brucethemoose2 3y agoI would also highly recommend https://tinybots.net/artbot https://tinybots.net/artbot You can even run CLIP or (if its fast enough) llama.cpp on the CPU to contribute to the network, if you wish.
- gsharma 3y agoComfyUI works pretty well on old computers using CPU. It takes over 30 seconds per sampling step on a 2015 MacBook Air (I7, 8GB RAM).
- dragonwriter 3y ago> On the other hand, this is nice for anyone who wants to play with these networks locally and does not have a nvidia GPU with 6+ gigabytes of VRAM. I SD 1.x works in stable-diffusion-web-ui (aka, A1111, one of the more popular frontends) with (reportedly) as little as 2GB of VRAM (I’ve personally used 1.5 and 2.1 without any problems with a 4GB card.)