4 ms·

The article is pretty light on details, and misses (or I missed it if they mentioned it) an important distinction. There are two main types of distillation: -

The article is pretty light on details, and misses (or I missed it if they mentioned it) an important distinction. There are two main types of distillation: - completion based methods, where you take a big model, give it some queries, and use the answers to post-train a smaller model. This is what deepseek did with qwen models, where they took ~800k traces made by R1 and used sft on smaller qwen2.5 models. What the sky team found in their experiments is that you can use as few as 1-2k traces to reach similar results. Much cheaper. - logit/internal representations based methods, where you need access to the raw model, and for each pair q -> response you train the small model on the entire distribution of the logits at the same time. This is a method suited for model creators, where they can take a pair of big + small model of the same architecture, and "distill" it in the smaller one. This is likely how they train their -flash -mini -pico and so on. The first method can be used via API access. The second one can't. You need access to things that API providers won't give you.

m12k 1y ago
From the article: "Considering that the distillation requires access to the innards of the teacher model, it’s not possible for a third party to sneakily distill data from a closed-source model like OpenAI’s o1, as DeepSeek was thought to have done. That said, a student model could still learn quite a bit from a teacher model just through prompting the teacher with certain questions and using the answers to train its own models — an almost Socratic approach to distillation."
- dr_dshiv 1y ago
  Like PHI — textbooks are all you need. You can create entirely synthetic yet high quality training data with a strong model (the generated textbooks) and make very small models like PHI.
- NitpickLawyer 1y ago
  Right, my bad then I read it in a hurry. They do mention the distinction.
- pyman 1y ago
  This is exactly what the DeepSeek team did, and now Anthropic is repackaging it a year later, calling it “subliminal learning” or using the teacher and student analogy to take credit for work done by Chinese researchers. https://malted.ai/deepseek-and-the-future-of-distillation/ https://malted.ai/deepseek-and-the-future-of-distillation/ While Anthropic and OpenAI are still trying to make sense of what China's top computer scientists pulled off a year ago, something that shook the core of Nvidia's business, China is now showcasing the world's first commercial unhackable cryptography system using QKD and post-quantum cryptography to secure all phone calls between Beijing and Hefei.
  - rcxdude 1y ago
    >While Anthropic and OpenAI are still trying to make sense of what China's top computer scientists pulled off a year ago The whole reason they're accusing them of distilling their models is that this was a well-known technique that's relatively easy compared to creating or improving on one in the first place. Deepseek was impressive for how lean it was (and it shook the markets because it demonstrated obviously what the savvier observers already had figured, that the big AI companies in the US didn't have a huge moat), but they certainly did not come up with this concept.
    - pyman 1y ago
      OpenAI raised $40 billion and Anthropic raised $10 billion, claiming they needed the money to buy more expensive Nvidia servers to train bigger models. Then Chinese experts basically said, no you don't. And they proved it.
      - ben_w 1y ago
        More like the Egg of Columbus or the Red Queen. You need to run as hard as you can just to stay where you are, and once you've got the answer it's very much easier to reproduce the result. This is of course also what annoys a certain fraction of commenters in every discussion about LLMs (and in art, diffusion models): they're overwhelmingly learning from the examples made by others, not investigating things for themselves. While many scientists will have had an example like Katie Mack's viral tweet* with someone who doesn't know what "research" even is in the first place and also mistakes "first thing I read" for such research, the fact many humans also do this doesn't make the point wrong when it's about AI. * https://paw.princeton.edu/article/katie-mack-09-taming-troll https://paw.princeton.edu/article/katie-mack-09-taming-troll

m12k 1y ago

From the article: "Considering that the distillation requires access to the innards of the teacher model, it’s not possible for a third party to sneakily distill data from a closed-source model like OpenAI’s o1, as DeepSeek was thought to have done. That said, a student model could still learn quite a bit from a teacher model just through prompting the teacher with certain questions and using the answers to train its own models — an almost Socratic approach to distillation."

dr_dshiv 1y ago
Like PHI — textbooks are all you need. You can create entirely synthetic yet high quality training data with a strong model (the generated textbooks) and make very small models like PHI.
NitpickLawyer 1y ago
Right, my bad then I read it in a hurry. They do mention the distinction.
pyman 1y ago
This is exactly what the DeepSeek team did, and now Anthropic is repackaging it a year later, calling it “subliminal learning” or using the teacher and student analogy to take credit for work done by Chinese researchers. https://malted.ai/deepseek-and-the-future-of-distillation/ https://malted.ai/deepseek-and-the-future-of-distillation/ While Anthropic and OpenAI are still trying to make sense of what China's top computer scientists pulled off a year ago, something that shook the core of Nvidia's business, China is now showcasing the world's first commercial unhackable cryptography system using QKD and post-quantum cryptography to secure all phone calls between Beijing and Hefei.
- rcxdude 1y ago
  >While Anthropic and OpenAI are still trying to make sense of what China's top computer scientists pulled off a year ago The whole reason they're accusing them of distilling their models is that this was a well-known technique that's relatively easy compared to creating or improving on one in the first place. Deepseek was impressive for how lean it was (and it shook the markets because it demonstrated obviously what the savvier observers already had figured, that the big AI companies in the US didn't have a huge moat), but they certainly did not come up with this concept.
  - pyman 1y ago
    OpenAI raised $40 billion and Anthropic raised $10 billion, claiming they needed the money to buy more expensive Nvidia servers to train bigger models. Then Chinese experts basically said, no you don't. And they proved it.
    - ben_w 1y ago
      More like the Egg of Columbus or the Red Queen. You need to run as hard as you can just to stay where you are, and once you've got the answer it's very much easier to reproduce the result. This is of course also what annoys a certain fraction of commenters in every discussion about LLMs (and in art, diffusion models): they're overwhelmingly learning from the examples made by others, not investigating things for themselves. While many scientists will have had an example like Katie Mack's viral tweet* with someone who doesn't know what "research" even is in the first place and also mistakes "first thing I read" for such research, the fact many humans also do this doesn't make the point wrong when it's about AI. * https://paw.princeton.edu/article/katie-mack-09-taming-troll https://paw.princeton.edu/article/katie-mack-09-taming-troll