2 ms·
Llama.cpp can quantize without special training, but I'm not sure if any special model architecture support is needed to read it in the first place. If it can b
by MrDrMcCoy 2mo ago
Llama.cpp can quantize without special training, but I'm not sure if any special model architecture support is needed to read it in the first place. If it can be converted to gguf at all and you know what tensors to target, it can get the full ternary bonsai treatment today.
- binary132 2mo agoQAT is an optimizing quantization algorithm, not naive quant.
- MrDrMcCoy 2mo agoRight, but the way they phrased it suggested that without QAT it could not be quanted at all.
- dragonwriter 2mo agoIsn’t QAT a training approach (roughly, simulating quantization in the forward pass during training so that quantization of the level targeted in training has close-to-optimal behavior), not a quantization algorithm? Hence, the name?
- binary132 2mo agosounds like a repeatable method of optimizing quantization to me, friend
- NitpickLawyer 2mo agoSure, but that's for "personal" serving. I meant for 3rd party providers. Usually we get a good indication on what it costs to host this, as the prices settle on open router. That's why I said it's tougher to serve than kimi k3 on launch. As a provider you'd do fp8 if the model creator didn't do QAT on q4, or until someone does a good calibrated nvfp4. And that's usually nvda :)
- MrDrMcCoy 2mo agoThat makes sense, but your specific phrasing precluded the possibility of non-QAT quantization.
- NitpickLawyer 2mo agoShould have worded that better, my bad.