4 ms·
Run DeepSeek R1 Dynamic 1.58-bit
- deleted 2y ago[deleted]
- homarp 2y ago"The 1.58bit quantization should fit in 160GB of VRAM for fast inference" instruction for llama.cpp: https://huggingface.co/unsloth/DeepSeek-R1-GGUF#instructions-to-run-this-model-in-llamacpp https://huggingface.co/unsloth/DeepSeek-R1-GGUF#instructions...
- danielhanchen 2y agoOh thanks for sharing this! The fork of llama.cpp for how to do the dynamic quant is here: https://github.com/unslothai/llama.cpp https://github.com/unslothai/llama.cpp. I also found min_p = 0.05 can help reduce chances of some bad tokens coming up for 1.58bit (I found it to happen around 1/8000 tokens of the time)
- homarp 2y agodiscussed here https://news.ycombinator.com/item?id=42850222 https://news.ycombinator.com/item?id=42850222