3 ms·
Oh yes 192GB machines should be able these quants (131GB for 1.58bit, 158GB for 1.73bit, 183GB for 2.22bit) well :)
by danielhanchen 2y ago
Oh yes 192GB machines should be able these quants (131GB for 1.58bit, 158GB for 1.73bit, 183GB for 2.22bit) well :)
- bradfox2 2y agoGreat release Daniel. Applaud the consistency you have shown. Can you release slightly bigger quant versions? Would enjoy something that runs well on 8x32 v100 and 8x80 A100.
- danielhanchen 2y agoThanks! Oh I did release 4bit quants, 5bit, 6bit etc all at https://huggingface.co/unsloth/DeepSeek-R1-GGUF https://huggingface.co/unsloth/DeepSeek-R1-GGUF if that helps - they're not dynamic though but it should function fine :)