2 ms·
Large model trains small model, I suspect you end up with a better training session than purely unsupervised. You can think of it like transfer learning > Know
by verdverm 4y ago
Large model trains small model, I suspect you end up with a better training session than purely unsupervised. You can think of it like transfer learning
> Knowledge distillation has been successfully used
to transfer knowledge from larger, more competent
teacher models into smaller student models affordable for practical applications
It sounds like a bit of Chain of Thought going on too
> We propose a new paradigm, Distilling step-bystep, that leverages the ability of LLMs to reason
about their predictions to train smaller models in
a data-efficient way.
If we anthropomorphize a bit, you can think of this as a person reading all the things and learning on their own vs learning with a mentor, which results in much better results for humans. So maybe this is an interesting, expected result. It will also be interesting to see how this can be combined with DeepMind's RETRO ideas.
https://web.mit.edu/5.95/readings/bloom-two-sigma.pdf https://web.mit.edu/5.95/readings/bloom-two-sigma.pdf