3 ms·
How to run 1.58bit DeepSeek R1 with Open WebUI
- rahimnathwani 2y agoI'm curious how many tokens per second they got, the model won't fit in RAM. That will cause a lot of disk access before every token.
- danielhanchen 2y agoOh so the trick is llama.cpp has VRAM, RAM and disk offloading! For good speeds, it's best to have (VRAM + RAM) >= 140GB will get you probs 5 tokens / s max. For reasonable speeds, (VRAM + RAM) >= 80GB will get you probs 2-3 tokens / s. If you have less, then llama.cpp will resort to mmap / disk offloading, so you'll probs get < 1 tokens / s. There is a PR where we offload non MoE layers to disk if you don't have enough RAM / VRAM, and it should make stuff faster!
- im3w1l 2y agoIf building a new personal computer where one design goal is running these and future models (inference only) at decent speeds what should be prioritized? Getting huge ram (256gb?)? Fast ram? Slow big vram gpu vs faster gpu with less vram? Some super fast ssd?
- danielhanchen 2y agoThat's actually a wonderful question! In terms of trends, I see the following: 1. GPUs are most likely at their limit in terms of FLOPs - float4 / FP4 is most likely the "final" low precision data-type. NVIDIA might provide 1.58bit support or FP2, but unlikely. If there was FP2, it might make it 1.5x faster. 2. Shrinking transistors might still have some room to go, but don't expect 2x or 4x faster - the majority of speedups was in tensor cores and low bit representation. 3. We might get more interesting transformer archs since DeepSeek showcased their unique arch for R1 / V3. Due to these, I would first wait and see - ie I would actually wait until the next OSS model release say Llama 3, Gemma 3, etc and see what the large model labs are focusing on, then maybe I would wait for RTX 50 Super or a cheaper version or even RTX 60x series. Larger VRAM is always better. SSD is good, but not that important - RAM is more important.
- ilaksh 2y agoAre there any evals or benchmarks of this model?
- danielhanchen 2y agoThere's no extensive benchmarks, but I did do a Flappy Bird pass@3 test just to show that a 1.58bit model does in fact work well! The goal was to showcase that MoEs quantized down to 1.58bit without any further training does in fact work!
- ilaksh 2y agoRight and congrats on all of that. But I need to know the evals to know whether this actually makes sense versus alternatives.
- instagib 2y agohttps://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic Tons of information here too.
- danielhanchen 2y agoOh yep! :) I'm the main author of it! :) If you have any questions on it - ask away!