4 ms·

Prediction is compression, but I am not sure if it is true the other way around. It's obvious that an accurate predictor enables encoding only the data that th

by Lerc 2mo ago

Prediction is compression, but I am not sure if it is true the other way around. It's obvious that an accurate predictor enables encoding only the data that the predictor gets wrong. But a compressor can encode patterns that defy prediction by looking at the data as a whole. It doesn't have to look at everything in sequence as it arrives. Applying transformations prior to entropy encoding often isn't just 'rearranging into an easier to compresss format' the transformation can be doing the job of peeking into the future. That makes the encoding a whole lot easier, but it is much harder to call it prediction.

tgv 2mo ago
Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".
- vrighter 2mo ago
  there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too. And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).
  - tgv 2mo ago
    Anything can only represent past measurements. Statistics is not an exception. But they don't make a prediction about the future. That comes from a model you have, and it often is implicit: "the linear trend from the last 12 months will hold in the next month" or whatever. So compression isn't by definition prediction. The other way around doesn't have to hold either, but in the case of LLMs it does.
    - vrighter 2mo ago
      a prediction is not necessarily correct. What compressors actually store is how wrong the prediction was, in fact, a guess takes exactly 0 bits to encode. It builds a st atistical model at runtime based on past encountered data. Exactly the same as llms, with the exception that llms remain fixed at runtime, instead of adapting to the data it is currently processing, so they can't adapt at runtime if the model does not suit the data
- deleted 2mo ago
  [deleted]
Xmd5a 2mo ago
https://arxiv.org/abs/2311.08768 https://arxiv.org/abs/2311.08768 >Three Conjectures on Unexpectedeness >Unexpectedness is a central concept in Simplicity Theory, a theory of cognition relating various inferential processes to the computation of Kolmogorov complexities, rather than probabilities. Its predictive power has been confirmed by several experiments with human subjects, yet its theoretical basis remains largely unexplored: why does it work? This paper lays the groundwork for three theoretical conjectures. First, unexpectedness can be seen as a generalization of Bayes' rule. Second, the frequentist core of unexpectedness can be connected to the function of tracking ergodic properties of the world. Third, unexpectedness can be seen as constituent of various measures of divergence between the entropy of the world (environment) and the variety of the observer (system).

by Lerc 2mo ago

tgv 2mo ago

Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".

vrighter 2mo ago
there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too. And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).
- tgv 2mo ago
  Anything can only represent past measurements. Statistics is not an exception. But they don't make a prediction about the future. That comes from a model you have, and it often is implicit: "the linear trend from the last 12 months will hold in the next month" or whatever. So compression isn't by definition prediction. The other way around doesn't have to hold either, but in the case of LLMs it does.
  - vrighter 2mo ago
    a prediction is not necessarily correct. What compressors actually store is how wrong the prediction was, in fact, a guess takes exactly 0 bits to encode. It builds a st atistical model at runtime based on past encountered data. Exactly the same as llms, with the exception that llms remain fixed at runtime, instead of adapting to the data it is currently processing, so they can't adapt at runtime if the model does not suit the data
deleted 2mo ago
[deleted]

Xmd5a 2mo ago

https://arxiv.org/abs/2311.08768 https://arxiv.org/abs/2311.08768 >Three Conjectures on Unexpectedeness >Unexpectedness is a central concept in Simplicity Theory, a theory of cognition relating various inferential processes to the computation of Kolmogorov complexities, rather than probabilities. Its predictive power has been confirmed by several experiments with human subjects, yet its theoretical basis remains largely unexplored: why does it work? This paper lays the groundwork for three theoretical conjectures. First, unexpectedness can be seen as a generalization of Bayes' rule. Second, the frequentist core of unexpectedness can be connected to the function of tracking ergodic properties of the world. Third, unexpectedness can be seen as constituent of various measures of divergence between the entropy of the world (environment) and the variety of the observer (system).