4 ms·

I would say compression and intelligence are closely related, not equal. For example, zstd is a pretty good compressor, but does any one think it should be call

by ks2048 3y ago

I would say compression and intelligence are closely related, not equal. For example, zstd is a pretty good compressor, but does any one think it should be called a pretty good “AI”?

dragonwriter 3y ago
Wasn't there a paper recently about a fairly simple wrapper around a common conpression algorithm beating LLMs on classification tasks?
- jmholla 3y ago
  I thought that was about LLMs being trained on compressed data. But I might be thinking about a different paper.
  - dragonwriter 3y ago
    Gzip + kNN for text classification: https://aclanthology.org/2023.findings-acl.426.pdf https://aclanthology.org/2023.findings-acl.426.pdf
    - junipertea 3y ago
      Not LLM, just BERT, also did not actually outperform it. source: https://kenschutte.com/gzip-knn-paper/ https://kenschutte.com/gzip-knn-paper/
willis936 3y ago
Someone in sales with a product that uses zstd.
londons_explore 3y ago
Read the paper... Zstd, while it might be better than gzip or bzip, is still a very poor compressor compared to an ideal compressor (which hasn't yet been discovered). That is why zstd acts like a rather bad AI. Note that if you wanted to use zstd as an AI, you would patch out of the source code checksum checks, and you would then feed it a file to decompress (The cat sat on the mat), followed by a few bytes of random noise. A great compressor would output: The cat sat on the mat. It was comfortable, so he then lay down to sleep. A medium compressor would output: The cat sat on the mat. bat cat cat mat sat bat. A terrible compressor would output: The cat sat on the mat. D7s"/r %we See how each is using knowledge at different levels to generate a completion. Notice also how that few bytes generates different amounts of output depending on the compressors level of world understanding, and therefore compression ratio.
- astrange 3y ago
  > A terrible compressor would output: The cat sat on the mat. Dsr %we3 9T23 }£{D:rg!@ !jv£dP$ LLMs are sort of unable to do this because they use a fixed tokenizer instead of raw bytes. That means they won't output binary garbage even early on + saves a lot of memory, but it may hurt learning things like capitalization, rhyming, etc we think are obvious.
  - londons_explore 3y ago
    Even if you trained an LLM with a simplified tokenizer that simply had 256 tokens for each of 256 possible ascii characters, you would see the same result.
- canjobear 3y ago
  Concretely gzip will output something like "theudcanvas. ;cm,zumhmcyoetter toauuo long a one aay,;wvbu.mvns. x the dtls and enso.;k.like bla.njv" https://github.com/Futrell/ziplm https://github.com/Futrell/ziplm
twitch_checksum 3y ago
Well, Zstandard can be used as a very fast classification engine, which is a task traditionally done by AI : https://twitter.com/abhi9u/status/1683141215871705088 https://twitter.com/abhi9u/status/1683141215871705088 https://github.com/cyrilou242/ftcc https://github.com/cyrilou242/ftcc

by ks2048 3y ago

I would say compression and intelligence are closely related, not equal. For example, zstd is a pretty good compressor, but does any one think it should be called a pretty good “AI”?

dragonwriter 3y ago

Wasn't there a paper recently about a fairly simple wrapper around a common conpression algorithm beating LLMs on classification tasks?

jmholla 3y ago
I thought that was about LLMs being trained on compressed data. But I might be thinking about a different paper.
- dragonwriter 3y ago
  Gzip + kNN for text classification: https://aclanthology.org/2023.findings-acl.426.pdf https://aclanthology.org/2023.findings-acl.426.pdf
  - junipertea 3y ago
    Not LLM, just BERT, also did not actually outperform it. source: https://kenschutte.com/gzip-knn-paper/ https://kenschutte.com/gzip-knn-paper/

willis936 3y ago

Someone in sales with a product that uses zstd.

londons_explore 3y ago

Read the paper... Zstd, while it might be better than gzip or bzip, is still a very poor compressor compared to an ideal compressor (which hasn't yet been discovered). That is why zstd acts like a rather bad AI. Note that if you wanted to use zstd as an AI, you would patch out of the source code checksum checks, and you would then feed it a file to decompress (The cat sat on the mat), followed by a few bytes of random noise. A great compressor would output: The cat sat on the mat. It was comfortable, so he then lay down to sleep. A medium compressor would output: The cat sat on the mat. bat cat cat mat sat bat. A terrible compressor would output: The cat sat on the mat. D7s"/r %we See how each is using knowledge at different levels to generate a completion. Notice also how that few bytes generates different amounts of output depending on the compressors level of world understanding, and therefore compression ratio.

astrange 3y ago
> A terrible compressor would output: The cat sat on the mat. Dsr %we3 9T23 }£{D:rg!@ !jv£dP$ LLMs are sort of unable to do this because they use a fixed tokenizer instead of raw bytes. That means they won't output binary garbage even early on + saves a lot of memory, but it may hurt learning things like capitalization, rhyming, etc we think are obvious.
- londons_explore 3y ago
  Even if you trained an LLM with a simplified tokenizer that simply had 256 tokens for each of 256 possible ascii characters, you would see the same result.
canjobear 3y ago
Concretely gzip will output something like "theudcanvas. ;cm,zumhmcyoetter toauuo long a one aay,;wvbu.mvns. x the dtls and enso.;k.like bla.njv" https://github.com/Futrell/ziplm https://github.com/Futrell/ziplm

twitch_checksum 3y ago

Well, Zstandard can be used as a very fast classification engine, which is a task traditionally done by AI : https://twitter.com/abhi9u/status/1683141215871705088 https://twitter.com/abhi9u/status/1683141215871705088 https://github.com/cyrilou242/ftcc https://github.com/cyrilou242/ftcc