3 ms·
The frontier labs distill their own base models all day long. It’s not just something done by nefarious Chinese copycats. The knowledge embodied by the internal
by semiquaver 5mo ago
The frontier labs distill their own base models all day long. It’s not just something done by nefarious Chinese copycats. The knowledge embodied by the internal base models that we never see is much more powerful and useful than the much sparser raw training data
- supern0va 5mo agoI think you replied to the wrong parent.
- coldtea 5mo ago>It’s not just something done by nefarious Chinese copycats And even that would be rich as a accusation from SOTAs that depend on explicitly disregarding millions of training data intellectual property..
- manmal 5mo agoBut how? The training data is the unadulterated content those models are based on? I genuinely don’t understand, no snark.
- wtallis 5mo agoRaw training data is raw. A really big model trained on it has already done a first-pass of finding patterns and squeezing out redundancy. Re-ingesting the full training set to train a smaller model is probably more expensive, for marginal quality improvement over distilling from the large model.
- adgjlsfhk1 5mo agoDistilling from a larger model is not only probably cheaper than from data, it's also likely higher quality. There's pretty strong support for the proposition that NNs learn a smoothed and regularized version of the data. The NNs are likely higher quality than most of the data they are training from.
- semiquaver 5mo agoThis guy or gal or person gets it.
- flossly 5mo ago> nefarious Chinese copycats LLMs are themselves copy cats. I say thanks for open sourcing and thereby promoting affordable innovation, instead of "nefarious". :)
- deleted 5mo ago[deleted]