4 ms·
distilling K3 into DS4 flash will likely only be a good idea for specialists. the difference in model capacity is otherwise too large. we used to distill GLM 5
by teravor 3mo ago
distilling K3 into DS4 flash will likely only be a good idea for specialists. the difference in model capacity is otherwise too large.
we used to distill GLM 5.2 into Qwen 27B specialists with great success.
developing the workflows is tricky though. we had the advantage of a straightforward mapping function in mind: English -> SQL which was relatively easy to pipeline training for.
- Demiurge 3mo agoWhy would you do this?
- teravor 3mo agoa specialist model sufficiently post-trained can outperform a frontier model while being dirt cheap. what we did was distill GLM 5.2 into a 27B model on SQL and then post-train it with RL afterward. the result outperformed even Fable on that one task. the distillation step is just good sense in this workflow, to bootstrap a smaller model to the utmost you can before actually doing RL.