3 ms·

As other commenters have mentioned, the performance of this set up is probably not really great since there's not enough VRAM and lots of bits have to be moved

by hnfong 2y ago

As other commenters have mentioned, the performance of this set up is probably not really great since there's not enough VRAM and lots of bits have to be moved between CPU and GPU RAM. That said, there are sub-256GB quants of DeepSeek-R1 out there (not the distilled versions). See https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic I can't quantify the difference between these and the full FP8 versions of DSR1, but I've been playing with these ~Q2 quants and they're surprisingly capable in their own right. Another model that deserves mention is DeepSeek v2.5 (which has "fewer" params than V3/R1) - but still needs aggressive quantization before it can run on "consumer" devices (with less than ~100GB VRAM), and this is recently done by a kind soul: https://www.reddit.com/r/LocalLLaMA/comments/1irwx6q/deepseekv25_dynamic_quants_anyone/ https://www.reddit.com/r/LocalLLaMA/comments/1irwx6q/deepsee... DeepSeek v2.5 is arguably better than Llama 3 70B, so it should be of interest to anyone looking to run local inference. I really think more people should know about this.

SlavikCA 2y ago
I tried that Unsloth R1 quantization on my dual Xeon Gold 5218 with 384 GB DDR4-2666 (about half of memory channels used, so not most optimal). Type IQ2_XXS / 183GB, 16k context: CPU only: 3 t/s (tokens per second) for PP (prompt processing) and 1.44 t/s for response. CPU + NVIDIA RTX 70GB VRAM: 4.74 t/s for PP and 1.87 t/s for response. I wish Unsloth produce similar quantization for DeepSeek V3, - it will be more useful, as it doesn't need reasoning tokens, so even with same t/s it will faster overall.
idonotknowwhy 2y ago
Thanks a lot for the v2.5! I'll give that a whirl. Hopefully it's as coherent as v3.5 when quantized so small. > I can't quantify the difference between these and the full FP8 versions of DSR1, but I've been playing with these ~Q2 quants and they're surprisingly capable in their own right. I run the Q2_K_XL and it's perfectly good for me. Where it lacks vs FP8 is in creative writing. If you prompt it with for a story a few times, then compare with FP8, you'll see what I mean. For coding, the 1.58bit clearly makes more errors than the Q2XXS and Q2_K_XL
pinoy420 2y ago
[dead]
colorant 2y ago
Currently >8 token/s; there is a demo in this post: https://www.linkedin.com/posts/jasondai_run-671b-deepseek-r1-q4km-model-from-unsloth-activity-7303194182729244673-l8Ut https://www.linkedin.com/posts/jasondai_run-671b-deepseek-r1...

by hnfong 2y ago

SlavikCA 2y ago

I tried that Unsloth R1 quantization on my dual Xeon Gold 5218 with 384 GB DDR4-2666 (about half of memory channels used, so not most optimal). Type IQ2_XXS / 183GB, 16k context: CPU only: 3 t/s (tokens per second) for PP (prompt processing) and 1.44 t/s for response. CPU + NVIDIA RTX 70GB VRAM: 4.74 t/s for PP and 1.87 t/s for response. I wish Unsloth produce similar quantization for DeepSeek V3, - it will be more useful, as it doesn't need reasoning tokens, so even with same t/s it will faster overall.

idonotknowwhy 2y ago

Thanks a lot for the v2.5! I'll give that a whirl. Hopefully it's as coherent as v3.5 when quantized so small. > I can't quantify the difference between these and the full FP8 versions of DSR1, but I've been playing with these ~Q2 quants and they're surprisingly capable in their own right. I run the Q2_K_XL and it's perfectly good for me. Where it lacks vs FP8 is in creative writing. If you prompt it with for a story a few times, then compare with FP8, you'll see what I mean. For coding, the 1.58bit clearly makes more errors than the Q2XXS and Q2_K_XL

pinoy420 2y ago

[dead]

colorant 2y ago

Currently >8 token/s; there is a demo in this post: https://www.linkedin.com/posts/jasondai_run-671b-deepseek-r1-q4km-model-from-unsloth-activity-7303194182729244673-l8Ut https://www.linkedin.com/posts/jasondai_run-671b-deepseek-r1...