3 ms·
Distilling knowledge is data compression.
by jjk166 3y ago
Distilling knowledge is data compression.
- w10-1 3y agoYou're conflating memorization with generalization, no?
- jjk166 3y agoMemorization is storing data. Generalization is developing the heuristics by which you compress stored data. To distill knowledge is to apply heuristics to lossily-compress a large amount of data to a much smaller amount of data from which you nevertheless can recover enough information to be useful in the future.
- staunton 3y ago> Generalization is developing the heuristics by which you compress stored data such that > you nevertheless can recover enough information to be useful in the future. I disagree (in case you meant to imply it) that compression implies generalization.
- jjk166 3y agoI did not mean to imply compression implies generalization, if anything the reverse. Compression is the act of cutting, generalization is the whetstone by which you may sharpen a blade, which is the compression heuristic. A more general heuristic is to compression what a sharper blade is to cutting.