6 ms·
Isn't it widely speculated that these are distilled from current frontier models? Distillation is far less compute intensive than primary training. That said, i
by Gareth321 6mo ago
Isn't it widely speculated that these are distilled from current frontier models? Distillation is far less compute intensive than primary training. That said, if distillation produces something almost as good for a fraction of the cost, Jensen's point may stand.
- zozbot234 6mo agoYou can't really distill a model without access to the internal weights. You could train on chat logs, but that's absolutely not the same thing, it doesn't even come close to comprehensively "extracting" the model's capabilities. And everyone does that in the industry anyway ever since ChatGPT was first released, some versions of Opus even claimed to be DeepSeek if you prompted them in Chinese.
- ls612 6mo agoCalling it distillation does however make normies go along with it when they inevitably add all the Chinese labs to the entities list to pad Dario and Sam’s pockets.
- deleted 6mo ago[deleted]
- itemize123 5mo agoactually distillation is without weights - you basically just need a black box teacher model.
- Gareth321 5mo agoWeights are not required for distillation. I'm not sure how you came to that belief. Distillation is training a student model to minimic a teacher model output. Anthropic, for example, posted a 2026 disclosure (https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks https://www.anthropic.com/news/detecting-and-preventing-dist...) which singles out DeepSeek's distillation activity. They detected over 16M actions over 24,000 fraudulent accounts. That's just what they detected.