3 ms·
The model is natively quantized (i.e. it was trained that way in the first place, so this is not a post-training quantization which degrades performance).
by kouteiheika 4mo ago
The model is natively quantized (i.e. it was trained that way in the first place, so this is not a post-training quantization which degrades performance).
- theanonymousone 4mo agoBut the huggingface link mentions BF16, F16, and I32?
- kouteiheika 4mo agoNot every weight is quantized. For example, those weights which don't take much space or are highly important are left in higher precision. State-of-art quantization of weights is never done uniformly (i.e. to all weights and in the same way).
- zackangelo 4mo agoI don't believe safetensors has a native int4 dtype, so they packed 4 int4s into a bf16 in this checkpoint.
- knollimar 4mo agoIsn't it not completely quantized? I thought there were some dense parts but most is int4?
- wgd 4mo agoOften in MoE models the experts are quantized while the shared portions, being a much smaller part of the network with greater impact, are kept at higher or full precision. Not familiar with the Kimi QAT approach specifically but it's likely they do this.