4 ms·
Is QAT a different quantisation format to Q4_0? Can you try "gemma-3-27b-it-qat" for a model: https://lmstudio.ai/model/gemma-3-27b-it-qat https://lmstudio.ai/
by kristianp 2y ago
Is QAT a different quantisation format to Q4_0? Can you try "gemma-3-27b-it-qat" for a model: https://lmstudio.ai/model/gemma-3-27b-it-qat https://lmstudio.ai/model/gemma-3-27b-it-qat
- pantulis 2y agoThanks for your suggestion! I think it was just the filename. I tried the model you suggested by opening it in LLMStudio and keep getting 8.3 tps.
- pantulis 2y agoNow this is raising my curiosity, is there anything else I could try and tweak to achieve better tps? Can it be related to being GGUF instead of MLX?