3 ms·
I don't think you know what distill means
by Bolwin 24d ago
I don't think you know what distill means
- airstrafer 24d agoI guess I don't. Does post-training from another (larger) model not fall under the umbrella of distillation? I'd imagine it leads to the same spiky-ness issues...?
- FergusArgyll 24d agoDistilling you don't have the actual model weights of the teacher. All you have are the teachers answers to a lot of questions. You then teach your own smaller model to answer more similarly to the big teacher model. Fine tuning you have the actual model weights of the original model, you then train that model to answer in a different (or better) way.
- Bolwin 23d agoDistillation requires you to have the actual logits of each token from the teacher model, which in practice means having the model itself. What you're describing is just synthetic data. Note Anthropic misused the term in their post about Chinese model distillation, deliberately I assume.
- FergusArgyll 23d agoThere are multiple kinds of distillation https://arxiv.org/abs/2106.03310 https://arxiv.org/abs/2106.03310 https://arxiv.org/abs/2207.12106 https://arxiv.org/abs/2207.12106