3 ms·
We have a partial understanding of why distillation works—it is explained by The Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635 https://arxiv.org/a
by ZeljkoS 2y ago
We have a partial understanding of why distillation works—it is explained by The Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635). But if I am understanding correctly, that doesn't mean you can train a smaller network from scratch. You need a lot of randomness in the initial large network, for some neurons to have "winning" states. Then you can distill those winning subsystems to a smaller network.
Note that similar process happens with human brain, it is called Synaptic pruning (https://en.wikipedia.org/wiki/Synaptic_pruning https://en.wikipedia.org/wiki/Synaptic_pruning). Relevant quote from Wikipedia (https://en.wikipedia.org/wiki/Neuron#Connectivity https://en.wikipedia.org/wiki/Neuron#Connectivity):
"It has been estimated that the brain of a three-year-old child has about 10^15 synapses (1 quadrillion). This number declines with age, stabilizing by adulthood. Estimates vary for an adult, ranging from 10^14 to 5x10^14 synapses (100 to 500 trillion)."
- 3abiton 2y agoSo more 'mature' models might arise in the near future with less params and better benchmarks?
- raducu 2y ago"Better", but not better than the model they were distilled from, at least that's how I understand it.
- salemba 2y agoI think this is how the "child brain" works too. The better the parents and the environement are, the better the child evolution is :)
- cristiancavalli 2y agoNot at all — how many people were geniuses and their parents not? I can name several and I’m sure with a quick search you can too.
- iFreilicht 2y agoHow is that relevant? A few examples do not disprove anything. It's pretty common knowledge that the more successful/rich etc. your parents were, the more likely you'll be successful/rich etc. This does not directly prove the theory your parent comment posits, being that better circumstances during a child's development improve the development of that child's brain. That would require success being a good predictor of brain development, which I'm somewhat uncertain about.
- _w1tm 2y agoThey might also be more biased and less able to adapt to new technology. Interesting times.
- coder543 2y agoThat's been happening consistently for over a year now. Small models today are better than big models from a year or two ago.
- Arthur_ODC 2y agoSo, can a distilled 8B model (say, the Deepseek-R1-Distil-Llama-8B or whatever) be "trained up" to a higher parameter 16B Parameter model after distillation from a superior model, or is it forever stuck at the 8B parameters that can just be fine tuned?