4 ms·
This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise ve
by cubefox 2mo ago
This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.
- vatsachak 2mo agoLLMs are both the best compression and prediction algorithm for English text.
- deleted 2mo ago[deleted]
- cubefox 2mo agoProbably not if you account for algorithm size.
- vatsachak 2mo agoTrue that
- davmre 2mo agoThat depends how much text you're compressing. For example, LLM pretraining datasets are on the order of tens or hundreds of terabytes. If an LLM-based code for that data is ~twice as efficient as gzip, you could afford to transmit the weights of even a very large LLM and still come out ahead. In other words: LLMs actually are excellent compressors of their training sets in the formal information-theoretic sense.