4 ms·
I'd kind of like to see some sample LLM entries in the list (as in, at the bottom of the list). Not because they'd ever be competitive, but because it would und
by sfink 2y ago
I'd kind of like to see some sample LLM entries in the list (as in, at the bottom of the list). Not because they'd ever be competitive, but because it would underscore how they burn massive amounts of space for the model with little concern for how much of it is "necessary" for predicting text.
(I'm not sure how you would optimally use an LLM for a lossless compression task. I guess you could convert the input to a stream of single-bit "yes, it guessed right" tokens and multi-bit "no, the actual token is X" tokens, and then compress that stream with a more traditional entropy encoder? That way the LLM decompressor would always be working off a valid prefix. But there are probably more clever ways.)
- Sesse__ 2y ago> I'm not sure how you would optimally use an LLM for a lossless compression task. You have the model output probabilities for each token, and then use arithmetic encoding using that model. (This uses optimally few bits per token, under the assumption that your model is right.) This is how all the winning models work, AFAIK, except they frequently work per-bit instead of per-token.
- sfink 2y agoOh duh, that makes a lot more sense. Thanks!
- jonas21 2y agoThe top entry is based on a Transformer architecture. It's basically an LLM, but trained on the input instead.