4 ms·
That assumes that training on the output of another language model somehow gives you the ability to improve your model and to catch up somehow
by ithkuil 4y ago
That assumes that training on the output of another language model somehow gives you the ability to improve your model and to catch up somehow
- iandanforth 4y agoIt does. In general this is known as teacher-student training or knowledge distillation. It works better if you have access to the activations of the model but you can work with just outputs as well.
- ithkuil 4y agoDoes it work even if you don't control the inputs?
- satvikpendem 4y agoWell, it does, that's how we got Alpaca from LLaMA.