3 ms·
there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are
by vrighter 2mo ago
there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too.
And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).
- tgv 2mo agoAnything can only represent past measurements. Statistics is not an exception. But they don't make a prediction about the future. That comes from a model you have, and it often is implicit: "the linear trend from the last 12 months will hold in the next month" or whatever. So compression isn't by definition prediction. The other way around doesn't have to hold either, but in the case of LLMs it does.
- vrighter 2mo agoa prediction is not necessarily correct. What compressors actually store is how wrong the prediction was, in fact, a guess takes exactly 0 bits to encode. It builds a st atistical model at runtime based on past encountered data. Exactly the same as llms, with the exception that llms remain fixed at runtime, instead of adapting to the data it is currently processing, so they can't adapt at runtime if the model does not suit the data