3 ms·

I see your 'fine point' but I don't think it holds - 'distillation' is a perfectly reasonable term to describe the process of creating outputs from one model to

I see your 'fine point' but I don't think it holds - 'distillation' is a perfectly reasonable term to describe the process of creating outputs from one model to that expose key training element, to use in another model. I think where the definition may be be invalid, is in the creation of 'unrelated data sets for training' models, for unrelated issues. Creating training sets that mach a models core training, is definitely distillation, it does not have to expose the reasoning traces. Synthesizing data for some arbitrary thing ... I'm not sure that would be the same thing. It's hard to draw the line. But the Chinese models are absolutely distilling - and would not be competitive without this distillation. At the same time, there's a lot of real innovation and regular building going on at the same time over there.

HarHarVeryFunny 3mo ago
No - you can't distill if what you are given doesn't have the thing in it that you want to distill out of it. I don't know why it's so important to you to use the word "distillation", but it's the wrong word to use. BTW OpenAI on twitter also said that Kimi 3 "cannot be explained away by distillation or anything like that". The timeline of how long it takes to train a model and when Fable was released don't even line up. This is just Anthropic as usual trying to manipulate the US government into helping them shut down competition.
- bluegatty 3mo ago
  Distillation is absolutely - and uncontroversially - a valid term for what is happening here. This isn't really a debate, I'm not making a fine point - just check with all of the various defintions of the term. Moreover - the 'reasoning traces' are not required for distillation at all. Finally - it's entirely possible for them to have used Fable for later stage fine tuning. It's fair to be skeptical of Anthropic (and everyone else) - but this is 'distilling'.
  - HarHarVeryFunny 3mo ago
    Words have meaning - you cant just redefine them because you want to. Are reasoning traces required for distillation? Well they are if what you are trying to distill is reasoning, such as coding expertise. Do you need reasoning traces for "LLM as judge"? No, but it would be highly perverse to call that distillation when there is a more accurate name for it - LLM as judge. If you want to call use of Anthropic's redacted model outputs in any fashion that violates their terms of service (using them them to help develop anything that competes with Anthropic) as "distillation" then I can't stop you, but it reduces their claims to a joke. Finally, as noted, OpenAI (who are just as anti-Chinese as Anthropic) said that Kimi 3 can't be explained via distillation (even true distillation!!), or even "anything like it". But random internet guy, you, disagrees. OK.
    - throw10920 3mo ago
      > Words have meaning - you cant just redefine them because you want to. You are redefining words. The consensus among people who work in this space is that "distilling" is what's actually going on here. > who are just as anti-Chinese as Anthropic Conflating criticism of IP theft with being "anti-Chinese" is a standard PRC influence playbook technique. And furthermore, OpenAI's market strategy is to win through regulatory capture. They are financially incentivized for Anthropic to be distilled by PRC labs and to be undercut by open models. Their claim about Kimi not being explainable due to distillation is not a factual claim - it's marketing from a company owned by Sam Altman. Although, it does conclusively disprove your claim about the meaning of distillation, because you cannot say that "Kimi can't be explained by distilling" unless the consensus definition of "distillation" is such that it could be done on the summarized reasoning traces that Anthropic models expose.
      - HarHarVeryFunny 3mo ago
        > you cannot say that "Kimi can't be explained by distilling" unless the consensus definition of "distillation" is such that it could be done on the summarized reasoning traces that Anthropic models expose You can interpret it as you choose, but a much more obvious reason he [OpenAI's Dean Ball] might say it can't be distilled is because it can't be distilled. You can't distill alcohol out of orange juice.
  - throw10920 3mo ago
    The GP (HarHarVeryFunny) is not operating in good faith. They've repeatedly lied about my own words to me, and are making up definitions that people who actually work at a frontier lab would disagree with.

HarHarVeryFunny 3mo ago

No - you can't distill if what you are given doesn't have the thing in it that you want to distill out of it. I don't know why it's so important to you to use the word "distillation", but it's the wrong word to use. BTW OpenAI on twitter also said that Kimi 3 "cannot be explained away by distillation or anything like that". The timeline of how long it takes to train a model and when Fable was released don't even line up. This is just Anthropic as usual trying to manipulate the US government into helping them shut down competition.

bluegatty 3mo ago
Distillation is absolutely - and uncontroversially - a valid term for what is happening here. This isn't really a debate, I'm not making a fine point - just check with all of the various defintions of the term. Moreover - the 'reasoning traces' are not required for distillation at all. Finally - it's entirely possible for them to have used Fable for later stage fine tuning. It's fair to be skeptical of Anthropic (and everyone else) - but this is 'distilling'.
- HarHarVeryFunny 3mo ago
  Words have meaning - you cant just redefine them because you want to. Are reasoning traces required for distillation? Well they are if what you are trying to distill is reasoning, such as coding expertise. Do you need reasoning traces for "LLM as judge"? No, but it would be highly perverse to call that distillation when there is a more accurate name for it - LLM as judge. If you want to call use of Anthropic's redacted model outputs in any fashion that violates their terms of service (using them them to help develop anything that competes with Anthropic) as "distillation" then I can't stop you, but it reduces their claims to a joke. Finally, as noted, OpenAI (who are just as anti-Chinese as Anthropic) said that Kimi 3 can't be explained via distillation (even true distillation!!), or even "anything like it". But random internet guy, you, disagrees. OK.
  - throw10920 3mo ago
    > Words have meaning - you cant just redefine them because you want to. You are redefining words. The consensus among people who work in this space is that "distilling" is what's actually going on here. > who are just as anti-Chinese as Anthropic Conflating criticism of IP theft with being "anti-Chinese" is a standard PRC influence playbook technique. And furthermore, OpenAI's market strategy is to win through regulatory capture. They are financially incentivized for Anthropic to be distilled by PRC labs and to be undercut by open models. Their claim about Kimi not being explainable due to distillation is not a factual claim - it's marketing from a company owned by Sam Altman. Although, it does conclusively disprove your claim about the meaning of distillation, because you cannot say that "Kimi can't be explained by distilling" unless the consensus definition of "distillation" is such that it could be done on the summarized reasoning traces that Anthropic models expose.
    - HarHarVeryFunny 3mo ago
      > you cannot say that "Kimi can't be explained by distilling" unless the consensus definition of "distillation" is such that it could be done on the summarized reasoning traces that Anthropic models expose You can interpret it as you choose, but a much more obvious reason he [OpenAI's Dean Ball] might say it can't be distilled is because it can't be distilled. You can't distill alcohol out of orange juice.
- throw10920 3mo ago
  The GP (HarHarVeryFunny) is not operating in good faith. They've repeatedly lied about my own words to me, and are making up definitions that people who actually work at a frontier lab would disagree with.