It's entirely the QAT. The whole distillation process is quantization-aware from the start, so the ternary weights are learned rather than fitted after the fact

It's entirely the QAT. The whole distillation process is quantization-aware from the start, so the ternary weights are learned rather than fitted after the fact. The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.