4 ms·
How does this work? I know that LLMs predict the log-likelihood of the next token in a sequence.. so I guess you initialize the LLM from scratch, say that each
by sterlind 3y ago
How does this work? I know that LLMs predict the log-likelihood of the next token in a sequence.. so I guess you initialize the LLM from scratch, say that each token of the input text is the Nth most likely possibility, and record what N is for that token? And then you do some combination of run-length encoding and Huffman trees or something to record the Ns?
Since this is not open-source, but seems simple to do, perhaps I'll make my own.
- Scene_Cast2 3y agoA slightly more elegant method would be to throw an arithmetic coder on the output vector (normalized and represented as an interval) and record the interval represented by the real token to be compressed. This would allow for better compression ratios than your approach, I think (due to arbitrary probability support). My potential issue with NNs for compression is that sometimes they predict near zero for some probabilities that actually occur in real data - in which case the number of bytes needed to encode would blow up. But that can be somewhat guarded against (perhaps by limiting the lower bound of output probabilities).