4 ms·
There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average
by smcleod 1y ago
There is no reason to run models for inference at static fp16, modern quantisation formats dynamically assign precision to the layers that need them, an average of 6bpw is practical imperceptible from full precision, 8bpw if you really want to squeeze every tiny last drop out of it (although it's unlikely it will be detectable). That is a huge memory saving.
- nhecker 1y ago> dynamically assign precision to the layers that need them Well now I'm curious; how is a layer judged on its relative need for precision? I guess I still have a lot of learning to do w.r.t. how quantization is done. I was under the impression it was done once, statically, and produced a new giant GGUF blob or whatever format your weights are in. Does that assumption still hold true for the approach you're describing?
- irthomasthomas 1y agoLast I checked they ran some sort of evals before and after quantisation and measured the effect. E.g Exllama-v2 measures the loss while reciting Wikipedia articles.
- clownpenis_fart 1y ago[dead]
- smcleod 1y agoWithin the GGUF (and some other formats) you'll see each layer gets its own quantisation, for example embeddings layers are usually more sensitive to quantisation and as such are often kept at Q8 or FP16. If you run GGUF-dump or click on the GGUF icon on a model in huggingface you'll see.
- smcleod 1y agoHave a watch of https://youtu.be/vW30o4U9BFE https://youtu.be/vW30o4U9BFE
- vlovich123 1y agoWhat quantization formats are these? All the OSS ones from GGML apply a uniform quantization
- int_19h 1y agoGGML hasn't been a thing for some time, and GGUF (its successor) has features such as "importance matrix" quantization that is all about quantizing adaptively. Then there's all the stuff that Unsloth does, e.g.: https://unsloth.ai/blog/dynamic-v2 https://unsloth.ai/blog/dynamic-v2
- smcleod 1y agoNo they don't. GGML is non-uniform. Each layer gets its own level of quantisation.
- smcleod 1y agoHave a watch of https://youtu.be/vW30o4U9BFE https://youtu.be/vW30o4U9BFE