3 ms·
You aren’t the only one. I wonder how distillation of LLMs is done, didn’t research it and hoped this might be a solution. Well, it might be a solution but one
by Ringz 1y ago
You aren’t the only one. I wonder how distillation of LLMs is done, didn’t research it and hoped this might be a solution. Well, it might be a solution but one that come with more hallucinations.
- qeternity 1y agoIt's the same process as regular training, but instead of a single token for your cost function, it's top-k logits.