4 ms·
Mixtral works great at 3-bit quantization. It fits onto a single RTX 3090 and runs at about 50 tokens/s. The output quality is not "ruined" at all. For the amo
by coder543 3y ago
Mixtral works great at 3-bit quantization. It fits onto a single RTX 3090 and runs at about 50 tokens/s. The output quality is not "ruined" at all.
For the amount of money you're talking about, you could also buy two 3090s (~$750 each on eBay) and have 48GB of VRAM to run with less quantization at full speed.
M-series Macs are surprisingly flexible platforms, but they're not "the only" consumer platform that can do Mixtral.
- Casteil 3y agoFair enough. I did put 'ruining' in quotes for a reason - I haven't compared output between Q3 and Q4_K_M that I use, but you do generally sacrifice output quality at higher quantization levels. And you're right, you can run it on a multi-GPU setup if you're so inclined.
- coder543 3y agoYou can also choose to run at 4-bit quantization, offloading ~27 out of 33 layers to the GPU, and that runs at about 25 tokens/s for me. I think that's about the same speed as you get out of an M1 Max running at 4 bits? Although I'm not sure about the newer M2 or M3 Max chips. Googling around, I didn't immediately see clear benchmarks for those.
- Casteil 3y agoNice - that's still pretty solid.. although on a more typical 3060 or 3070 with less vram available, I probably wouldn't expect numbers quite that good. My 14" M1 Max does around 30t/s on Mixtral Q4_K_M.
- patrakov 3y agoJust as another data point, a CPU-only setup with Q5_K_M would give you roughly 4 tokens per second on a Ryzen laptop (Dell Inspiron 7415 upgraded to 64 GB of RAM).
- eyegor 3y agoSo you don't see significantly worse performance on 3bit quantized models compared to 4? Every 7/13b model I tried quantized gave much worse responses at 3 bit and below, whereas the differences from 4 bit to 6 or even 8 bit is more subtle.
- coder543 3y agoMixtral is a larger model, so maybe that makes it more tolerant of that level of quantization? I’ve been impressed with 3-bit Mixtral, but I haven’t done a ton of side by sides against 4-bit because I haven’t felt the need.
- 3abiton 3y agoHave you tried the 2x 3090 setup? Using nvlink or SLI?
- coder543 3y agoI have not personally gotten to test things that way
- Tepix 3y agoI tried it, with NVlink. The speedup during inference is negligible. You'll probably benefit more during training.
- kwerk 3y agoI use 2x 3090, no nvlink though. I’d read it doesn’t help that much but not well read on any improvement.
- chpatrick 3y agoCould you share what you use to run it on a single 3090? I'd love to try it!
- coder543 3y agoollama has been by far the easiest way for me, either on Linux directly (as I do now) or WSL2.
- bekantan 3y ago> The output quality is not "ruined" at all. That was my experience as well - 3-bit version is pretty good. I also tried 2-bit version, which was disappointing. However, there is a new 2-bit approach in the works[1] (merged yesterday) which performs surprisingly well for Mixtral 8x7B Instruct with 2.10 bits per weight (12.3 GB model size). [1] https://github.com/ggerganov/llama.cpp/pull/4773 https://github.com/ggerganov/llama.cpp/pull/4773
- coder543 3y agoThat is a very interesting discussion. Weird to me that the quantization code wasn’t required to be in the same PR. Ika is also already talking about a slightly higher 2.31bpw quantization, apparently.
- mark_l_watson 3y agoI could only run 2-bit q2 mode on my 32G M2 Pro. I was a little disappointed, but I look forward to try the new approach you linked. I just use Mistral’s and also a 3rd party hosting service for now. After trying the various options for running locally, I have settled on just using Ollama - really convenient and easy, and the serve APIs let me use various LLMs in several different (mostly Lisp) programming languages. With excellent resources from Hugging Face, tool providers, etc., I hope that the user facing interface for running LLMs is simplified even further: enter your hardware specs and get available models filtered by what runs on a user’s setup. Really, we are close to being there. Off topic: I hope I don’t sound too lazy, but I am retired (in the last 12 years before retirement I managed a deep learning team at Capital One, worked for a while at Google and three other AI companies) and I only allocate about 2 hours a day to experiment with LLMs so I like to be efficient with my time.