4 ms·

How are smaller models distilled from large models, I know of LoRA, quantization like technique; but does distilling also mean generating new datasets for conve

How are smaller models distilled from large models, I know of LoRA, quantization like technique; but does distilling also mean generating new datasets for conversing with smaller models entirely from the big models for many simpler tasks?

tintor 2y ago
Smaller models can be trained to match log probs of the larger model. Larger model can be used to generate synthethic data for the smaller model.
Havoc 2y ago
I believe these were separately trained not distilled. Though the 405B was used for synth data for the other two