3 ms·
Any gemma-2-9b or 27b 4 bit GGUF's on HuggingFace yet? Thanks!
by luke-stanley 2y ago
Any gemma-2-9b or 27b 4 bit GGUF's on HuggingFace yet? Thanks!
- XzAeRosho 2y agoIt's on HuggingFace already: https://huggingface.co/google/gemma-2-9b https://huggingface.co/google/gemma-2-9b
- luke-stanley 2y agoI know the safe tensors are there, but I said GGUF 4-bit quantised, which is kinda the standard for useful local applications, a typical balanced sweet spot of performance and quality. It's makes it much easier to use, works in more places, be it personal devices or a server etc.
- chown 2y agoIf you are still looking for it, I just made it available on an app[1] that I am working on with Gemma2 support. https://msty.app https://msty.app
- luke-stanley 2y agoAre you saying you put a 4-bit GGUF on HuggingFace?
- luke-stanley 2y agoActually for the 9B model, this has 4-bit quantised weights (and others): https://huggingface.co/bartowski/gemma-2-9b-it-GGUF https://huggingface.co/bartowski/gemma-2-9b-it-GGUF Still no 27B 4-bit GGUF quants on HF yet! I'm monitoring this search: https://huggingface.co/models?library=gguf&sort=trending&search=gemma-2+27b https://huggingface.co/models?library=gguf&sort=trending&sea...
- SubiculumCode 2y agohttps://huggingface.co/bartowski/gemma-2-27b-it-GGUF https://huggingface.co/bartowski/gemma-2-27b-it-GGUF
- thot_experiment 2y agoI'm curious about the quantization quality claims in the table there. Is this a Gemma 2 specific thing (more subtlety in the weights somehow?). In my testing and testing I've seen elsewhere at least for llama3 8B (and some less rigorous testing with other models) q_8 -> q4_K_M are basically indistinguishable from one another?
- janwas 2y agoYes, PPL and certain benchmarks do not detect differences from quantization. But recent work gives cause for concern, e.g., https://arxiv.org/pdf/2310.01382 https://arxiv.org/pdf/2310.01382, https://arxiv.org/pdf/2405.18137 https://arxiv.org/pdf/2405.18137.
- luke-stanley 2y agoThe first paper is good to critique the performance of quantised models, it points out that 40-50% 'compression' typically results in only slight loss for RAG tasks relying on in-context learning, but for factual tasks replying on stored knowledge, performance very quickly dropped off. They looked at Vicuna, one of the earlier models, so I wonder how applicable it is to recent models like the Phi 3 range. I don't think deliberate clever adversarial attacks like those of the 2nd paper are a sensible worry for most, but it is fun. Thanks for the links @janwas.