6 ms·
Reminder that compression = intelligence: https://mattmahoney.net/dc/rationale.html https://mattmahoney.net/dc/rationale.html ...which leads to the theory of
by optimalsolver 3y ago
Reminder that compression = intelligence:
https://mattmahoney.net/dc/rationale.html https://mattmahoney.net/dc/rationale.html
...which leads to the theory of mind known as "compressionism":
https://ceur-ws.org/Vol-1419/paper0045.pdf https://ceur-ws.org/Vol-1419/paper0045.pdf
- ks2048 3y agoI would say compression and intelligence are closely related, not equal. For example, zstd is a pretty good compressor, but does any one think it should be called a pretty good “AI”?
- dragonwriter 3y agoWasn't there a paper recently about a fairly simple wrapper around a common conpression algorithm beating LLMs on classification tasks?
- jmholla 3y agoI thought that was about LLMs being trained on compressed data. But I might be thinking about a different paper.
- dragonwriter 3y agoGzip + kNN for text classification: https://aclanthology.org/2023.findings-acl.426.pdf https://aclanthology.org/2023.findings-acl.426.pdf
- junipertea 3y agoNot LLM, just BERT, also did not actually outperform it. source: https://kenschutte.com/gzip-knn-paper/ https://kenschutte.com/gzip-knn-paper/
- willis936 3y agoSomeone in sales with a product that uses zstd.
- londons_explore 3y agoRead the paper... Zstd, while it might be better than gzip or bzip, is still a very poor compressor compared to an ideal compressor (which hasn't yet been discovered). That is why zstd acts like a rather bad AI. Note that if you wanted to use zstd as an AI, you would patch out of the source code checksum checks, and you would then feed it a file to decompress (The cat sat on the mat), followed by a few bytes of random noise. A great compressor would output: The cat sat on the mat. It was comfortable, so he then lay down to sleep. A medium compressor would output: The cat sat on the mat. bat cat cat mat sat bat. A terrible compressor would output: The cat sat on the mat. D7s"/r %we See how each is using knowledge at different levels to generate a completion. Notice also how that few bytes generates different amounts of output depending on the compressors level of world understanding, and therefore compression ratio.
- astrange 3y ago> A terrible compressor would output: The cat sat on the mat. Dsr %we3 9T23 }£{D:rg!@ !jv£dP$ LLMs are sort of unable to do this because they use a fixed tokenizer instead of raw bytes. That means they won't output binary garbage even early on + saves a lot of memory, but it may hurt learning things like capitalization, rhyming, etc we think are obvious.
- londons_explore 3y agoEven if you trained an LLM with a simplified tokenizer that simply had 256 tokens for each of 256 possible ascii characters, you would see the same result.
- canjobear 3y agoConcretely gzip will output something like "theudcanvas. ;cm,zumhmcyoetter toauuo long a one aay,;wvbu.mvns. x the dtls and enso.;k.like bla.njv" https://github.com/Futrell/ziplm https://github.com/Futrell/ziplm
- twitch_checksum 3y agoWell, Zstandard can be used as a very fast classification engine, which is a task traditionally done by AI : https://twitter.com/abhi9u/status/1683141215871705088 https://twitter.com/abhi9u/status/1683141215871705088 https://github.com/cyrilou242/ftcc https://github.com/cyrilou242/ftcc
- tysam_and 3y agoNo. Compression == information, not intelligence. This is a common mistake that people make, and is a popular expression making the rounds right now, but it is incorrect. Intelligence _may arise_ from information. Information _necessarily arises_ from compression. It is very similar, to me, to the concept of humors (https://en.wikipedia.org/wiki/Humorism https://en.wikipedia.org/wiki/Humorism). Empirically, it was observed that these things had some correlation, and they occurred together, but they are not directly causative. Similarly with the theory of spontaneous generation (https://en.wikipedia.org/wiki/Spontaneous_generation https://en.wikipedia.org/wiki/Spontaneous_generation), which was another theory spawned (heh, as it were), from casual causal correlation (again, as it were). This can be shown rather easily when you show that the time-dynamics of an exemplar system with high information diverge from the time dynamics of a system with high intelligence, i.e. there is a structural component to the information embedded in the system such that the temporal aspects of applied information are somehow necessary for intelligence. I hope to release some work at least tangentially related to this within the next few years (though of course it is a bit more high-flying and depends on some other in-flight work). If we are to attempt to move towards AGI, I really think we need to stick with the mathematical basics. Neural networks, although they've been made out to be really complicated, are actually quite simple in terms of the concepts powering them, I believe. That's something I'm currently working on putting together and trying to distill to its essence. That will also be at least 1-2 years out, and likely before any temporally-related work, all going well. In my experience, this all is just a personal belief from a great deal of time and thought with the minutia of neural networks. That said, I could perhaps be biased with some notion of simplicity, since I've worked with them long enough for some concepts to feel comfortably second-nature.
- bbstats 3y agoOTOH true intelligence can still poorly compress things.
- tysam_and 3y agoAny ERM-based method will have this flaw (if it is a flaw -- I'd possibly consider it a tradeoff) -- it is the nature of compression itself. I would make the argument, I think, that I believe that this disrupts the 'information' layer of the hierarchy, not the 'intelligence', though that is just my personal 2 cents of course.
- burtonator 3y agoThere must be a relationship here between Shannon's estimation that english is 1 bit of entropy per character and is highly redundant and 'easy' to predict. A highly advanced AI could compress the text and predict the next sequences easily. This seems like a direct connection like electricity and magnetism. And maybe that's why English needs to be about 1 bit because we're not very intelligent.
- YeGoblynQueenne 3y ago>> Reminder that compression = intelligence: If I conclude from this that you mean that gzip is intelligent, will I be accused of bad faith?
- bbstats 3y agoI would say compression has some congruency with intelligence.
- deleted 3y ago[deleted]
- canjobear 3y agoCompression is equivalent to learning but it's not clear to me that that gets you all the way to intelligent action. In reinforcement learning terms, it can get you asymptotically to a perfect model of the environment, but it's not clear to me that it would tell you how to find the highest-value action in that environment.