4 ms·
Where is the actual evidence of distillation? I keep seeing this repeated ad nauseam but I must have somehow missed the evidence.
by colingauvin 2mo ago
Where is the actual evidence of distillation? I keep seeing this repeated ad nauseam but I must have somehow missed the evidence.
- voxic11 2mo agoDistillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531 https://arxiv.org/pdf/1503.02531 Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit https://github.com/arcee-ai/DistillKit
- maleldil 2mo agoI took GP as asking for evidence that the reduced-cost Sol is actually a distillation of the previous-cost Sol. AFAIK, providers distilling or quantising models and offering them as the same model have not been proven.
- sebzim4500 2mo agoI doubt he was claiming that. He's probably saying that the ability of Chinese companies to be able to distill frontier US models has put downwards pressure on the price of all models.
- colingauvin 2mo agoI'm saying that it is unclear that without distillation this wouldn't still be happening. There is a massive narrative that no one but OpenAI, Anthropic, and Google can make a model without distilling. But there's basically no evidence of that.
- byzantinegene 2mo agoI think the (non-definitive) evidence is that the chinese models are always just slightly behind the publically released US models, and never ahead.
- sebzim4500 2mo agoI think it's more that distilling is enormously cheaper than training from scratch to achieve the same results
- miki123211 2mo agoAlternatively, modern AI is good enough at optimizing its own kernels that it just keeps pushing costs down. Unlike the semi-decentralized inference provider community, OpenAI has both the talent and the compute to throw at the problem of making their models much more efficient to run. GPU kernel optimization is just the kind of well-bounded problem with clear success criteria that AI loves.
- jmalicki 2mo agoThat distillation exists isn't the question. It's about evidence this is an active force in competition in LLMs.
- mindwok 2mo agoI think the biggest actual piece of evidence is how hard the major players are trying to stop it
- porridgeraisin 2mo agoTo be precise, the distillation mentioned in this paper is not the distillation used by other model companies. In the one mentioned in the paper, your teacher and student model typically have similar architectures - and you typically need access to the full logits. What happens here instead is motivated by the fact that these companies don't have access to the training data and compute that anthropic/openai have. The distillation they do basically amounts to using traces from ant/oai models trained on much more data with a lot more compute (in many cases including the hidden intermediate tokens! turns out there were many ways to coax it out) and then either directly training on it or using it in many ways in post training pipelines. It falls under imitation learning, IMO.
- NicoJuicy 2mo agoHere's an example: https://github.com/microsoft/Build25-LAB329 https://github.com/microsoft/Build25-LAB329
- InsideOutSanta 2mo agoMusk confirmed in federal court that xAI does it: https://techcrunch.com/2026/04/30/elon-musk-testifies-that-xai-trained-grok-on-openai-models/ https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x... It's also how providers build their smaller models out of their larger ones; they publicly talk about the process.
- retinaros 2mo agothere is no evidence. it shortcuts post training by a huge margin this is true. but that is all.
- JacobAsmuth 2mo agohttps://techcrunch.com/2026/04/30/elon-musk-testifies-that-xai-trained-grok-on-openai-models/ https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x... Make sure to stay updated!
- retinaros 2mo agoNothing against what I said in there. Distillation is far from enough to get to the frontier. Its at best a ramp (the most efficient one used by everyone ) that shortcut and saves millions of rl runs before a model moves.
- dbbk 2mo agoYou said there isn't evidence of distillation. Yes there is
- retinaros 2mo agoI said there is no evidence distillation is all you need as per the initial comment "The fact that AI models can be so easily distilled and replicated is such a stroke of luck." it is far from trivial to reproduce the capabilities of anthropic models for instance
- wolvoleo 2mo agoAnd probably saves a lot of energy which would be good for the planet
- JacobAsmuth 2mo ago"No evidence" -> Large lab saying that they use distillation when training their models
- JacobAsmuth 2mo ago[flagged]
- flexagoon 2mo agoWhy does Kimi insist its name is Claude?
- Paradigma11 2mo agoPlease elaborate the mechanisms by which a LLM would know what model it is.
- deleted 2mo ago[deleted]
- datadrivenangel 2mo agoask it what type of model it is or what it's name is... it's weird that Kimi will say it's claude...
- tshaddox 2mo agoOkay, but I would also ask “why does Claude say that its name is Claude?”
- Gigachad 2mo agoBecause it's in the system prompt
- fwn 2mo agoAFAIK, you cannot run Claude without one of their mandatory system prompts at all. If we wanted to compare model responses, we would give all models system prompts with model names, thereby fixing the Kimi misattribution. The reason Kimi often states its name as Claude is likely because we can actually run it without the mandatory system prompt, smoothing over awkward competitor mentions.
- InvertedRhodium 2mo agoThe training data likely references Claude significantly more often than Kimi, given the popularity of the models. There will simply be more examples of “Claude” being the response to that question.
- v64 2mo agoThe evidence is Anthropic's own reporting [1]. You may doubt that they're telling the truth, but that's what they're reporting. [1] https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks https://www.anthropic.com/news/detecting-and-preventing-dist...
- Gigachad 2mo ago"Distillation attacks", and it's just the exact same thing they did to the rest of the web and all human works.
- brookst 2mo agoExplain “exact same thing”? Like it load the Wikipedia page for Fort Worth, Texas, does it make Wikipedia spin up an editor to go write the article? I’m really not sure how you’re getting “exact” here.
- Gigachad 2mo agoAnthropic scraped the whole web, scanned every book, pirated every bit of media to feed in to their training. Distillers are doing essentially the same thing scraping all the knowledge from the LLM to create a training set for a new one. They are crying about theft after committing the largest theft in human history.
- colingauvin 2mo ago
- hadlock 2mo agoIt turns out you can train a 1b model at almost 1000 tokens/s on a m5 max laptop. As a personal experiment, I've been asking Sol for synthetic training data and synthetic agentic training data (model distillation in it's purest form), plus modified opencode, codex transcripts etc for training data, and nobody's even paying me to do it. If I'm doing it has a hobby, you can bet industrial users are doing it.
- endymi0n 2mo agoBeen using a lot of Kimi K3 lately and the answers have been… „load-bearing“ to the point of hilariousness. It‘s obvious from where they distilled, even if sceptics rightly point out it can‘t have been the only source of their secret sauce, as it‘s been better than the current Opus 4.x at the time of release.
- actualwitch 2mo agoQwen 3.8 27b also enjoys their genuinely load-bearing seams.