3 ms·
We made DeepSeek R1 run on a local device via offloading and 1.58bit quantization :) https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseek
by danielhanchen 1y ago
We made DeepSeek R1 run on a local device via offloading and 1.58bit quantization :) https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic
I'm working on the new one!
- screaminghawk 1y agoI use this a lot! Thanks for your work and looking forward to the next one
- danielhanchen 1y agoThank you!! New versions should be much better!
- CamperBob2 1y agoYour 1.58-bit dynamic quant model is a religious experience, even at one or two tokens per second (which is what I get on my 128 MB Raptor Lake+4090). It's like owning your own genie... just ridiculously smart. Thanks for the work you've put into it!
- danielhanchen 1y agoOh thank you! :) Glad they were useful!
- nxobject 1y agoLikewise - for me, it feels how I imagined getting a microcomputer in the 70s was like. (Including the hit to the wallet… an Apple II cost the 2024 equivalent of ~$5k, too.)
- danielhanchen 1y ago:) The good ol days!
- behnamoh 1y ago> 1.58bit quantization of course we can run any model if quantize it enough. but I think the OP was talking about the unquantized version.
- danielhanchen 1y agoOh you can still run them unquantized! See https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-tune https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-t... where we show you can offload all MoE layers to system RAM, and leave non MoE layers on the GPU - the speed is still pretty good! You can do it via `-ot ".ffn_.*_exps.=CPU"`
- behnamoh 1y agoThanks, I'll try it! I guess "mixing" GPU+CPU would hurt the perf tho.