4 ms·
Is distillation something we have to live with or are there ways to prevent it?
by mattrighetti 2mo ago
Is distillation something we have to live with or are there ways to prevent it?
- sosodev 2mo agoRealistically you can't prevent distillation. OpenAI / Anthropic are slowly moving towards hiding the steps in-between input and output (hidden thinking), but that only helps so much. Imagine you put a file into Claude and say "do X to this" and it returns it to you without showing any of its internal reasoning. That's harder to distill, but the simple mapping of input to output still creates very valuable training data. It is reflective of all the training the model did to learn how to do that transformation.
- make3 2mo agoYou can also get it to think in the output tokens pretty easily, eg "Here's a math problem, I want your reasoning first, then the answer" which is what I assume they're doing.
- Cytobit 2mo agoYou make it sound like a bad thing.
- epolanski 2mo agoIf it was genuinely useful, we would've long reached the point where you train a model on a previous one's output in an ever improving loop. But this doesn't actually work.
- HarHarVeryFunny 2mo agoYou can prevent it by outputting a reasoning "summary" instead of the actual reasoning trace. Which Anthropic already do.
- sowbug 2mo agoIf you build a device that can help build devices, you shouldn't be too surprised when people use it to build devices.
- recursive 2mo agoTo the extent that we have to live with AI and LLM, I believe we have to live with distilling also.