3 ms·
No, distilling step-by-step https://arxiv.org/abs/2305.02301 https://arxiv.org/abs/2305.02301 distills LLM to task specific model. That works, and I know of mul
by sanxiyn 3y ago
No, distilling step-by-step https://arxiv.org/abs/2305.02301 https://arxiv.org/abs/2305.02301 distills LLM to task specific model. That works, and I know of multiple successes. But it doesn't relate to choice of optimizing training FLOP vs training and inference FLOP, since the resulting distilled model is not LLM.