3 ms·

I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that yo

I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot. Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.

Legend2440 2mo ago
>Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot. You can, actually! Any compressor can be losslessly converted into a generator, and vice versa. Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.
- pjankiewicz 2mo ago
  I agree that technically they are doing the same thing but in practice LLMs are better compressors than PNGs (learned this while I was researching this topic). That was quite surprising to me.
- vrighter 2mo ago
  Actually, it's trivial. I did it for fun once when I was learning about the PPM algorithm. It took about 15 minutes to reverse the whole thing.
davmre 2mo ago
Any compressor actually can be used, trivially, as an autoregressive language model. Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate. You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
- cubefox 2mo ago
  The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.
  - vatsachak 2mo ago
    https://bellard.org/ts_zip/ https://bellard.org/ts_zip/
    - cubefox 2mo ago
      This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.
      - vatsachak 2mo ago
        LLMs are both the best compression and prediction algorithm for English text.
        deleted 2mo ago
        [deleted]
        cubefox 2mo ago
        Probably not if you account for algorithm size.
        vatsachak 2mo ago
        True that
        davmre 2mo ago
        That depends how much text you're compressing. For example, LLM pretraining datasets are on the order of tens or hundreds of terabytes. If an LLM-based code for that data is ~twice as efficient as gzip, you could afford to transmit the weights of even a very large LLM and still come out ahead. In other words: LLMs actually are excellent compressors of their training sets in the formal information-theoretic sense.
aaroninsf 2mo ago
That sounds like boostrapping the weights involved in early layers, to obviate the need for those layers to learn (optimize) for the distribution in the training set. Makes me wonder idly, - is this conceptually akin in some sense to a "universal grammar," and if so - with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training, - does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics
- pjankiewicz 2mo ago
  That was my thinking as well mainly to increase the speed of training. But it may turn out that the simple statistics that you can capture like this may account for 1% of the training and are likely to be captured as the first thing during the training. But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc. This paper proposes something similar where you can inject memory without training https://arxiv.org/abs/2605.16893 https://arxiv.org/abs/2605.16893

Legend2440 2mo ago

>Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot. You can, actually! Any compressor can be losslessly converted into a generator, and vice versa. Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.

pjankiewicz 2mo ago
I agree that technically they are doing the same thing but in practice LLMs are better compressors than PNGs (learned this while I was researching this topic). That was quite surprising to me.
vrighter 2mo ago
Actually, it's trivial. I did it for fun once when I was learning about the PPM algorithm. It took about 15 minutes to reverse the whole thing.

davmre 2mo ago

Any compressor actually can be used, trivially, as an autoregressive language model. Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate. You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.

cubefox 2mo ago
The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.
- vatsachak 2mo ago
  https://bellard.org/ts_zip/ https://bellard.org/ts_zip/
  - cubefox 2mo ago
    This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.
    - vatsachak 2mo ago
      LLMs are both the best compression and prediction algorithm for English text.
      - deleted 2mo ago
        [deleted]
      - cubefox 2mo ago
        Probably not if you account for algorithm size.
        vatsachak 2mo ago
        True that
        davmre 2mo ago
        That depends how much text you're compressing. For example, LLM pretraining datasets are on the order of tens or hundreds of terabytes. If an LLM-based code for that data is ~twice as efficient as gzip, you could afford to transmit the weights of even a very large LLM and still come out ahead. In other words: LLMs actually are excellent compressors of their training sets in the formal information-theoretic sense.

aaroninsf 2mo ago

That sounds like boostrapping the weights involved in early layers, to obviate the need for those layers to learn (optimize) for the distribution in the training set. Makes me wonder idly, - is this conceptually akin in some sense to a "universal grammar," and if so - with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training, - does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics

pjankiewicz 2mo ago
That was my thinking as well mainly to increase the speed of training. But it may turn out that the simple statistics that you can capture like this may account for 1% of the training and are likely to be captured as the first thing during the training. But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc. This paper proposes something similar where you can inject memory without training https://arxiv.org/abs/2605.16893 https://arxiv.org/abs/2605.16893