4 ms·
> That said, fine tuning small models Mostly referred to as model distillation, but I give the author the benefit of the doubt that they didn't mean that.
by lyu07282 1y ago
> That said, fine tuning small models
Mostly referred to as model distillation, but I give the author the benefit of the doubt that they didn't mean that.
- sota_pop 1y agoMy understanding of model distillation is quite different in that it trains another (typically smaller) model using the error between the new model’s output and that of the existing - effectively capturing the existing model’s embedded knowledge and encoding it (ideally more densely) into the new.
- lyu07282 1y agoWhat what I was referring to is similar in concept, but I've seen both described in papers as distillation. What I meant was you take the output of a large model like GPT4 and use that as training data to fine-tune a smaller model.
- sota_pop 1y agoYes, that does sound very similar. To my knowledge, isn’t that (effectively) how the latest DeepSeek breakthroughs were made? (i.e. by leveraging chatgpt outputs to provide feedback for training the likes of R1)