First, I don’t believe there has been a shred of evidence that DeepSeek distilled OpenAI’s model. As to distillation: people use the term kind of imprecisely.

First, I don’t believe there has been a shred of evidence that DeepSeek distilled OpenAI’s model. As to distillation: people use the term kind of imprecisely. The most powerful form of distillation is one in which you train a smaller “student” model on a large amount of predictions from a larger, more powerful “teacher” model that uses the same tokenization scheme. You train the smaller model to output not just the same tokens as the teacher, but rather on the full probability distribution predicted by that model for each token. It’s a very dense transfer of knowledge. This is best done using the same training data that the teacher model was trained on, since you are asking the student to learn the teacher’s distribution. The more common version is to take an existing model and fine-tune it on a small number of text outputs from a larger model. No probability distributions over tokens - just the text itself. This doesn’t significantly alter the smaller model’s knowledge or capabilities, but rather, causes it to imitate the other model’s style. But it can unlock capabilities that were in the small model to begin with, but were not as accessible in its base form. For this type of distillation, it has been shown that a very small number of training examples are required if they are carefully selected.