3 ms·
What quantization formats are these? All the OSS ones from GGML apply a uniform quantization
by vlovich123 1y ago
What quantization formats are these? All the OSS ones from GGML apply a uniform quantization
- int_19h 1y agoGGML hasn't been a thing for some time, and GGUF (its successor) has features such as "importance matrix" quantization that is all about quantizing adaptively. Then there's all the stuff that Unsloth does, e.g.: https://unsloth.ai/blog/dynamic-v2 https://unsloth.ai/blog/dynamic-v2
- smcleod 1y agoNo they don't. GGML is non-uniform. Each layer gets its own level of quantisation.
- smcleod 1y agoHave a watch of https://youtu.be/vW30o4U9BFE https://youtu.be/vW30o4U9BFE