3 ms·

Dumb questions, from someone not in the field... What is a distilled model? Why doesn't Google do this (to make their models smaller)? Seems like you could m

Dumb questions, from someone not in the field... What is a distilled model? Why doesn't Google do this (to make their models smaller)? Seems like you could make a competitor to Gemini?

HenryNdubuaku 5mo ago
No question is stupid! 1. Distilled means taking the intelligence of a big model and compacting into a tiny model. 2. Google already does so with FunctionGemma, but Needle argues that better performance could be achieved with 10x smaller model using our technologies.
tintor 5mo ago
Model distillation is lossy compression of big model to produce a smaller model. Smaller model requires less space on disk, less video memory, and less compute (cheaper hardware). Downside is that distilled model performs worse on the same benchmarks compared to original model.
jmalicki 5mo ago
There are two answers already and neither is entirely adequate. In normal LLM training, you take a set of documents and have it learn to predict the future, then have some private RLHF/RLVR etc. data that it learns to produce good chat outputs from. In distillation, you take a set of prompts you are interested in, and record the big LLM's outputs, then train your small model to produce the same output as the big LLM. This has a few advantages - you can get performance much more quickly on your documents/prompts of interest, with a much cheaper training budget, and you don't have to worry about acquiring very expensive RLHF/RLVR training data. A lot of the very good Chinese LLMs got very good very quickly through distillation from frontier models, which is why Anthropic/Google/OpenAI are blocking it so aggressively.
- NitpickLawyer 5mo ago
  For completeness sake I'll add a bit more. The concept of distillation is not new in ML, and there are nuances to it. Traditionally you would have access to the bigger model, and for LLMs specifically you can train the small model on the entire distribution of output logits at the same time. So this would train the small model to output scores for each token in a similar fashion to the large model. There's "more to learn" from the entire distribution, rather than just from the chosen token. But since you don't have access to this from the API providers, the next best thing is to use the outputs themselves and train on those. That's more like a "poor man's distillation". It's still good, and as you mentioned worked fairly well for models catching up. But a lab that develops both the big model and the small model could make it better. (or you could choose to distill from an existing open model).
- PhunkyPhil 5mo ago
  > In distillation, you take a set of prompts you are interested in, and record the big LLM's outputs, then train your small model to produce the same output as the big LLM. Why use the bigger LLM outputs for this and not human outputs? If we assume that human responses to prompts are better than sota models (in some cases they are) then why use the big model at all?
  - jmalicki 5mo ago
    Have you seen what annotations cost? It can be on the order of $50/annotation for a reasonable document, some agentic annotations can cost over $1000 each, whereas a model response might cost $0.10, or maybe $20 for an agentic session. Plus all of that takes a ton of effort to collect. You can set up model distillation as a weekend batch job.

Dumb questions, from someone not in the field... What is a distilled model? Why doesn't Google do this (to make their models smaller)? Seems like you could make a competitor to Gemini?

HenryNdubuaku 5mo ago

No question is stupid! 1. Distilled means taking the intelligence of a big model and compacting into a tiny model. 2. Google already does so with FunctionGemma, but Needle argues that better performance could be achieved with 10x smaller model using our technologies.

tintor 5mo ago

Model distillation is lossy compression of big model to produce a smaller model. Smaller model requires less space on disk, less video memory, and less compute (cheaper hardware). Downside is that distilled model performs worse on the same benchmarks compared to original model.

jmalicki 5mo ago

There are two answers already and neither is entirely adequate. In normal LLM training, you take a set of documents and have it learn to predict the future, then have some private RLHF/RLVR etc. data that it learns to produce good chat outputs from. In distillation, you take a set of prompts you are interested in, and record the big LLM's outputs, then train your small model to produce the same output as the big LLM. This has a few advantages - you can get performance much more quickly on your documents/prompts of interest, with a much cheaper training budget, and you don't have to worry about acquiring very expensive RLHF/RLVR training data. A lot of the very good Chinese LLMs got very good very quickly through distillation from frontier models, which is why Anthropic/Google/OpenAI are blocking it so aggressively.

NitpickLawyer 5mo ago
For completeness sake I'll add a bit more. The concept of distillation is not new in ML, and there are nuances to it. Traditionally you would have access to the bigger model, and for LLMs specifically you can train the small model on the entire distribution of output logits at the same time. So this would train the small model to output scores for each token in a similar fashion to the large model. There's "more to learn" from the entire distribution, rather than just from the chosen token. But since you don't have access to this from the API providers, the next best thing is to use the outputs themselves and train on those. That's more like a "poor man's distillation". It's still good, and as you mentioned worked fairly well for models catching up. But a lab that develops both the big model and the small model could make it better. (or you could choose to distill from an existing open model).
PhunkyPhil 5mo ago
> In distillation, you take a set of prompts you are interested in, and record the big LLM's outputs, then train your small model to produce the same output as the big LLM. Why use the bigger LLM outputs for this and not human outputs? If we assume that human responses to prompts are better than sota models (in some cases they are) then why use the big model at all?
- jmalicki 5mo ago
  Have you seen what annotations cost? It can be on the order of $50/annotation for a reasonable document, some agentic annotations can cost over $1000 each, whereas a model response might cost $0.10, or maybe $20 for an agentic session. Plus all of that takes a ton of effort to collect. You can set up model distillation as a weekend batch job.