3 ms·
Distillation is not a thing unless you actually have the model weights. What people misleadingly call distillation is just training on chat logs, which has alw
by zozbot234 4mo ago
Distillation is not a thing unless you actually have the model weights. What people misleadingly call distillation is just training on chat logs, which has always been routine practice in the industry. There's a reason why every model today talks like early releases of ChatGPT.
- ericpauley 4mo agoIf Anthropic is calling it distillation [1] then that would argue for it being correct (or at least canonical) terminology. [1] https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks https://www.anthropic.com/news/detecting-and-preventing-dist...
- dannyw 4mo agoNo, a company choosing to use some terminology doesn’t make it correct nor canonical in any sense; especially when they have a vested interest in not being neutral or credible. If Google starts calling ads “Best Links” that doesn’t make it correct nor canonical; the correct term is still ads. Traditionally, distillation is when you get the actual logits of a model response (not exposed via API for years) and then use that to train a model.
- senordevnyc 4mo agoIf most people call it that, including the big labs, then maybe…you’re just out of date?
- ACCount37 4mo agoYou can logit distill (full token probabilities) or one hot distill (chat logs), or even align hidden states. All are distillation methods.