7 ms·

AI is currently just glorified compression

77 points by kklisura 3y ago

glimshe 3y ago
"Humans are just glorified monkeys"...
kklisura 3y ago
Relevant paper: https://arxiv.org/abs/2311.13110 https://arxiv.org/abs/2311.13110
- westurner 3y ago
  "White-Box Transformers via Sparse Rate Reduction" (2023) ; https://arxiv.org/abs/2311.13110 https://arxiv.org/abs/2311.13110 https://scholar.google.com/scholar?cites=15364532811271216525&as_sdt=5,43&sciodt=0,43&hl=en https://scholar.google.com/scholar?cites=1536453281127121652... : > Abstract: In this paper, we contend that the objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a mixture of low-dimensional Gaussian distributions supported on incoherent subspaces. The quality of the final representation can be measured by a unified objective function called sparse rate reduction. From this perspective, popular deep networks such as transformers can be naturally viewed as realizing iterative schemes to optimize this objective incrementally. Particularly, we show that the standard transformer block can be derived from alternating optimization on complementary parts of this objective: the multi-head self-attention operator can be viewed as a gradient descent step to compress the token sets by minimizing their lossy coding rate, and the subsequent multi-layer perceptron can be viewed as attempting to sparsify the representation of the tokens. This leads to a family of white-box transformer-like deep network architectures which are mathematically fully interpretable. Despite their simplicity, experiments show that these networks indeed learn to optimize the designed objective: they compress and sparsify representations of large-scale real-world vision datasets such as ImageNet, and achieve performance very close to thoroughly engineered transformers such as ViT. Code is at [ https://github.com/Ma-Lab-Berkeley/CRATE https://github.com/Ma-Lab-Berkeley/CRATE. "Bad numbers in the “gzip beats BERT” paper?" (2023) https://news.ycombinator.com/context?id=36766633 https://news.ycombinator.com/context?id=36766633 "78% MNIST accuracy using GZIP in under 10 lines of code" (2023) https://news.ycombinator.com/item?id=37583593 https://news.ycombinator.com/item?id=37583593
kklisura 3y ago
Nitter: https://nitter.net/ChombaBupe/status/1727720756371403131 https://nitter.net/ChombaBupe/status/1727720756371403131
ShamelessC 3y ago
tl;dr Tweet removes any potentially boring nuance from the research in favor of sensationalism and upvotes.
- whatasaas 3y ago
  Yelp. Reductionism and perfectionism to attack something. It’s just a {blank} or it can only detect cancer 98% of the time. I’d like to coin the the hot tub Time Machine argument but that’s probably already a thing.
IMTDb 3y ago
We also use to say that compression is the science of prediction. Scientific models are also tools used to predict the results of experiments. So if AI is able - through « compression » - to build models that help us predict experiments and understand the world better; it fully deserves the « intelligence » suffix.
- pas 3y ago
  intelligence is adaptive problem solving ability. one way of that is to observe the problem, build a model of it, come up with possible strategies, test them through simulation, and pick the best one. current GPT with good prompts is a very nice puzzle piece of this, but not there .. yet.
rodlette 3y ago
This is covered in "Information Theory, Inference, and Learning Algorithms" by David MacKay ( https://www.inference.org.uk/itprnn/book.pdf https://www.inference.org.uk/itprnn/book.pdf ): > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning. * In compression, gzip is predicting the next character. The model's prior is "contiguous characters will likely recur". This prior holds well for English text, but not for h264 data. * In ML, learning a model is compressing the training data into a model + parameters. It's not a damning indictment that current AI is just compression. What's damning is our belief that compression is a simpler/weaker problem.
- retrac 3y ago
  Compression is abstraction.
- ultrarunner 3y ago
  I haven't read MacKay's work, so maybe this is naive, but I think the belief is that compression is a deterministic endeavor, whereas AGI may not be. The ability to recall information is incredibly useful; the ability to do so in convenient and flexible ways doubly so. The level of complexity being proportional to the GPT's ability to interpret means that compression is not necessarily a simple problem, like you say. However, if intelligence involves nondeterministic traits, i.e. something in the vein of creation of the present, AGI could be a significantly different problem to solve than compression. I think there's at least an intuition that this is the case, which explains the belief that compression is a simpler/weaker problem. As an aside, I'm currently unsure of my position on this.
  - earthboundkid 3y ago
    LLMs are deterministic in principle. It’s like JPEG where it’s a lossy compression plus some deliberate injection of noise to add variety.
Bnichs 3y ago
"Jet engines are just glorified oil lamps." Not that I think the current AI is as life-changing as is purported, but this comparison is terrible. Almost all complex software is made up of a bunch of other previous simpler technologies.
- johnmw 3y ago
  I think the point was that the current transformer model paradigm will never be able to reach AGI, no matter how far you take it. It needs something fundamentally more to be able to do that. But maybe as you say, that something will be built on top of transformer technology.
  - deleted 3y ago
    [deleted]
2OEH8eoCRo0 3y ago
Once again, so does the human brain. You aren't remembering a high bandwidth stream of raw sensory input but a distilled essence of that.
- __loam 3y ago
  It's fun to see people endlessly make this point while providing no mechanisms for how it works. We might say "it surely must be this way" but we haven't shown it affirmatively in any sense. Until a mechanism is shown, I think it's really irresponsible to make statements like this. You really have no clue what it's doing.
  - crazygringo 3y ago
    There's no requirement that a mechanism be known in order to show that something is true. There are lots of other ways of deriving and inferring facts. There's nothing irresponsible about it whatsoever.
    - __loam 3y ago
      It does feel like if the "fact" being shown is that our basic cognition is "like" a computing system, the burden of proof is a lot higher. When we make statements about how people are "just" doing compression in their brains, that's incredibly dehumanizing. Be careful with this line of thinking, it doesn't lead to nice places.
  - cwillu 3y ago
    We know memories are essentially recreated when remembered, we know that eye-witness testimony is spectacularly prone to hallucinatory-details-in-the-llm-sense, we know that focusing attention on one aspect of a video will cause participants to simply not see remarkably obvious and noteworthy details of that video. No, I think it's safe to say that it's been shown pretty affirmatively. Wings were inspired by birds, and although they mechanically function in a manner entirely unlike a bird's wing, the aerodynamics of aircraft wings fully explain why birds can fly.
Racing0461 3y ago
compression is just glorified flow of electrons. Still usefull non the less.
booleandilemma 3y ago
As usual, when compression is brought up in the context of AI, it seems relevant to mention the Hutter Prize: https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize
simbolit 3y ago
think about video compression for a "zoom" call. You could just send a lot of bitmap files. Or, you could save bandwidth and zip them before send. Or you could integrate some image-specific compression into the file format, as in motion-jpeg. And in mpeg2/h264/h265 you supercharge this with temporal compression, not just adjacent pixels, not just blocks within the frame, but also pixels and blocks from adjacent frames are used to predict each pixel. And now think about not sending video, just a single initial jpeg, some face shape data, and now a continuous stream of position data of a dozen or so points on your face and the facial movements are reconstructed. (Nvidia research project from last year) And now think about no longer sending images at all, just coordinates for points in latent space (aka very abstract image descriptions) and a super-fast diffusion model "generates" the right frames on the fly. Where does "compression" end, where does "AI" start? Ultimately, the marketing people decide. Don't get hung up on the term "AI". "AI" is a bullshit hype buzzword. Calling it such serves no scientific-practical purpose, it is solely meant to impress venture capital.
aaron695 3y ago
[dead]
xeckr 3y ago
Ok. Let me know when WinRAR learns to speak.
jasfi 3y ago
I once read that intelligence is compression (or similar). An abstract way of thinking about it, but if true, then AI is on the right track.
- hindsightbias 3y ago
  If language models and CNNs are already creating better compression algorithms than humans, aren't they then smarter than us?
  - tovej 3y ago
    In this analogy, the language model IS a man-made compression algo, so definitely not
bane 3y ago
This wasn't obvious to me a while ago, but I've come around to it. To me the important thing is that it's not only compression, it's lossy compression. Models (any kind) are basically an imperfect approximation of some system, and that definition is precisely what's identified here. You can demonstrate it by assuming some sequential process that gets an "AI" like an LLM to generate as much content as it can[1], then train a new generation model on this content. Then use the new model to generate as much content as it can, train a third generation model, and so on. LLM->generate->LLM'->generate'->LLM''->generate''->...->LLM'...' Since each generation may not produce content with every possible token it has been trained on, it stands to probability that eventually some tokens, especially low probability tokens, will simply be left out of the generated content. The next model will therefore lose that token and it will never be part of the content it generates, and so on. After enough generations of this process, you eventually end up with a model with a single, or no, tokens it can use to generate text. The problem is that the models are trained on so much information that they're effectively models of the textual content of our civilization. If we hand off the reigns to these systems, they may work well for a while, and may even seem to be produce novel ideas (but which are secretly just old ideas in combinations we haven't seen before), but they'll also produce content which will inevitably be used to train later models. As the content these models produces starts to dominate the ratio of human-text:model-text, there will be less of a reason to preserve the human text and we'll end up with the scenario above. Things may work well for a while, perhaps a long time, but even if we don't end up with models trained on one token, we will end up with fewer and fewer ideas-as-text represented within them. Civilization will stagnate, the singularity will not commence, and things will slowly regress. Of course this all presumes research stops, and somehow LLMs become religion and we turn society over to them. Which of course isn't what's going to happen. Is it?
- tymscar 3y ago
  I agree, however in that scenario, are people fully stopping their content generation? Because if not, the pool will grown larger and the problem you described would still be there but to a lesser extent.
furyofantares 3y ago
I love how I've seen Ilya talk about it. If we could find the shortest program to reproduce the training set, that would be optimal compression. But it's an intractable problem, we can't even come close, there's just no way to come at it. But with deep learning we can instead find a circuit that approaches reproducing the training set. This is lossy compression. There's nothing "just glorified" about it though; the result is astounding. A more appropriate takeaway might be that sufficient compression is mind-bendingly more powerful than intuition might otherwise guess. Does calling it "just glorified" guide any intuition that in order to compress amazon reviews a neural net is gonna have weights that correspond to sentiment? Does it tell you that such compression also ends up producing an artifact that can be put to work in a generative framework? And that it'll be a very useful generative framework because such compression required weights that correspond to all sorts of useful ideas, a compression of something more meaningful than just text? Calling it "just glorified X" is clickbait. It's compression alright, and it's either 1) also whole lot more, or 2) compression is a whole lot more wild and impressive than you thought, or both.
- pas 3y ago
  but currently it doesn't seem very useful. it hallucinates too much, there's no reflexivity, no critical reasoning. sure, nice prompts can give it some structure, and with a lot of pre- and postprocessing (ie. RAG and other ways of forcing it to pick from a list) it's good for information retrieval.
- theGnuMe 3y ago
  The "sentiment" weights are just the supervised classification part on top of the "compression" piece. An autoencoder is closer to pure compression. SVD is compression as well. SVD is also useful to solve some equations.
3cats-in-a-coat 3y ago
So is our brain. “Model” by definition is an approximation of something else (or the thing we’re modeling… it’s a very overloaded term all right). But if you think you can use this “realization” to dismiss AI or claim what it can or cannot do… you’re missing the forest for the trees.
TeMPOraL 3y ago
Yes. AI is just compression. So is understanding, fundamentally. That's literally what it's about.
- lossolo 3y ago
  If that would be the case then LLMs would know how to do math but they don't.
  - mattstir 3y ago
    Popular chat-bot LLMs weren't trained to do math, they were trained to predict the next token in a conversational setting. The fact that they can do any math at all is a miracle considering that they have absolutely no thought process; they're just predicting the next token, one after the other, until some threshold is met and they stop producing output. Framing it another way, it's sort of like if you asked your calculator to have a conversation with you and it actually had a fairly decent go of it. Sure, it wasn't grammatically correct a lot of the time and it struggled quite a bit. But it wasn't at all designed to speak conversationally, so the fact that it could respond at all should be rather impressive.
  - TeMPOraL 3y ago
    They can do math - they can write code to be executed. It's not much different than me or you doing any maths more complex than reciting the times table.
  - lern_too_spel 3y ago
    They can write proofs, so they can do math. If you mean they're bad at arithmetic, a big part of that is improper tokenization. If you do digit level tokenization, even small transformers can learn arithmetic and generalize to longer inputs.
Arnt 3y ago
I chatted briefly to overtone.ai a few weeks ago, heading back to the hotel from a conference. What they do is train an existing LLM model to detect things about the text (overtones, I suppose you might say). What's interesting in this context is that they train the AI using an English corpus, but once trained the AI is able to detect the same traits in other languages. This sounds quite different from compression.
- thargor90 3y ago
  Nobody is arguing that the usecases are the same. In the end you can't even chat with gzip (although you could with it's predictor). The thing is, that building the predictor is almost the same thing for compression and LLM. Of course the goals and taken tradeoffs are different. The paper shows this analogy. ChatGPT et al use structured prediction to simulate intelligence. Building the predictor is fancy lossy compression. Questions arise if lossy compression of things without copyright is legal or not. If I mp3 a lossless recording we currently think it is not legal. With LLMs this is not entirely clear yet.
  - Arnt 3y ago
    As it happens I know zip. I ported that to linux back in 0.95 days, I think the code was called info-zip in those days. Chatting with its predictor is a fanciful description to say the least. I've also read the transformers paper mentioned in the tweet. Of course an LLM is on some level similar to a compression system, and on some level it's also just high and low voltages on some integrated circuits. Saying "just" glorified compression isn't something I'll believe without good arguments, though.
jesuslop 3y ago
Liked this view always (look for Hutter prize rationale), but I think it needs to be accomodated in a general perception/action loop that optimizes a lower level fitness/utility/reward (for instance an inner sense of pleasure/pain).
dgudkov 3y ago
Is generalization also a form of compression?
RecycledEle 3y ago
It seems this X post is from someone trying to claim LLMs are copyright violations. The argument that LLMs are only a new compression algorithm is nonsense.
thealistra 3y ago
Isn’t childbirth a glorified decompression of parents’ DNA? This is a tweet level of discourse

77 points by kklisura 3y ago

glimshe 3y ago

"Humans are just glorified monkeys"...

kklisura 3y ago

Relevant paper: https://arxiv.org/abs/2311.13110 https://arxiv.org/abs/2311.13110

westurner 3y ago
"White-Box Transformers via Sparse Rate Reduction" (2023) ; https://arxiv.org/abs/2311.13110 https://arxiv.org/abs/2311.13110 https://scholar.google.com/scholar?cites=15364532811271216525&as_sdt=5,43&sciodt=0,43&hl=en https://scholar.google.com/scholar?cites=1536453281127121652... : > Abstract: In this paper, we contend that the objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a mixture of low-dimensional Gaussian distributions supported on incoherent subspaces. The quality of the final representation can be measured by a unified objective function called sparse rate reduction. From this perspective, popular deep networks such as transformers can be naturally viewed as realizing iterative schemes to optimize this objective incrementally. Particularly, we show that the standard transformer block can be derived from alternating optimization on complementary parts of this objective: the multi-head self-attention operator can be viewed as a gradient descent step to compress the token sets by minimizing their lossy coding rate, and the subsequent multi-layer perceptron can be viewed as attempting to sparsify the representation of the tokens. This leads to a family of white-box transformer-like deep network architectures which are mathematically fully interpretable. Despite their simplicity, experiments show that these networks indeed learn to optimize the designed objective: they compress and sparsify representations of large-scale real-world vision datasets such as ImageNet, and achieve performance very close to thoroughly engineered transformers such as ViT. Code is at [ https://github.com/Ma-Lab-Berkeley/CRATE https://github.com/Ma-Lab-Berkeley/CRATE. "Bad numbers in the “gzip beats BERT” paper?" (2023) https://news.ycombinator.com/context?id=36766633 https://news.ycombinator.com/context?id=36766633 "78% MNIST accuracy using GZIP in under 10 lines of code" (2023) https://news.ycombinator.com/item?id=37583593 https://news.ycombinator.com/item?id=37583593

kklisura 3y ago

Nitter: https://nitter.net/ChombaBupe/status/1727720756371403131 https://nitter.net/ChombaBupe/status/1727720756371403131

ShamelessC 3y ago

tl;dr Tweet removes any potentially boring nuance from the research in favor of sensationalism and upvotes.

whatasaas 3y ago
Yelp. Reductionism and perfectionism to attack something. It’s just a {blank} or it can only detect cancer 98% of the time. I’d like to coin the the hot tub Time Machine argument but that’s probably already a thing.

IMTDb 3y ago

We also use to say that compression is the science of prediction. Scientific models are also tools used to predict the results of experiments. So if AI is able - through « compression » - to build models that help us predict experiments and understand the world better; it fully deserves the « intelligence » suffix.

pas 3y ago
intelligence is adaptive problem solving ability. one way of that is to observe the problem, build a model of it, come up with possible strategies, test them through simulation, and pick the best one. current GPT with good prompts is a very nice puzzle piece of this, but not there .. yet.

rodlette 3y ago

This is covered in "Information Theory, Inference, and Learning Algorithms" by David MacKay ( https://www.inference.org.uk/itprnn/book.pdf https://www.inference.org.uk/itprnn/book.pdf ): > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning. * In compression, gzip is predicting the next character. The model's prior is "contiguous characters will likely recur". This prior holds well for English text, but not for h264 data. * In ML, learning a model is compressing the training data into a model + parameters. It's not a damning indictment that current AI is just compression. What's damning is our belief that compression is a simpler/weaker problem.

retrac 3y ago
Compression is abstraction.
ultrarunner 3y ago
I haven't read MacKay's work, so maybe this is naive, but I think the belief is that compression is a deterministic endeavor, whereas AGI may not be. The ability to recall information is incredibly useful; the ability to do so in convenient and flexible ways doubly so. The level of complexity being proportional to the GPT's ability to interpret means that compression is not necessarily a simple problem, like you say. However, if intelligence involves nondeterministic traits, i.e. something in the vein of creation of the present, AGI could be a significantly different problem to solve than compression. I think there's at least an intuition that this is the case, which explains the belief that compression is a simpler/weaker problem. As an aside, I'm currently unsure of my position on this.
- earthboundkid 3y ago
  LLMs are deterministic in principle. It’s like JPEG where it’s a lossy compression plus some deliberate injection of noise to add variety.

Bnichs 3y ago

"Jet engines are just glorified oil lamps." Not that I think the current AI is as life-changing as is purported, but this comparison is terrible. Almost all complex software is made up of a bunch of other previous simpler technologies.

johnmw 3y ago
I think the point was that the current transformer model paradigm will never be able to reach AGI, no matter how far you take it. It needs something fundamentally more to be able to do that. But maybe as you say, that something will be built on top of transformer technology.
- deleted 3y ago
  [deleted]

2OEH8eoCRo0 3y ago

Once again, so does the human brain. You aren't remembering a high bandwidth stream of raw sensory input but a distilled essence of that.

__loam 3y ago
It's fun to see people endlessly make this point while providing no mechanisms for how it works. We might say "it surely must be this way" but we haven't shown it affirmatively in any sense. Until a mechanism is shown, I think it's really irresponsible to make statements like this. You really have no clue what it's doing.
- crazygringo 3y ago
  There's no requirement that a mechanism be known in order to show that something is true. There are lots of other ways of deriving and inferring facts. There's nothing irresponsible about it whatsoever.
  - __loam 3y ago
    It does feel like if the "fact" being shown is that our basic cognition is "like" a computing system, the burden of proof is a lot higher. When we make statements about how people are "just" doing compression in their brains, that's incredibly dehumanizing. Be careful with this line of thinking, it doesn't lead to nice places.
- cwillu 3y ago
  We know memories are essentially recreated when remembered, we know that eye-witness testimony is spectacularly prone to hallucinatory-details-in-the-llm-sense, we know that focusing attention on one aspect of a video will cause participants to simply not see remarkably obvious and noteworthy details of that video. No, I think it's safe to say that it's been shown pretty affirmatively. Wings were inspired by birds, and although they mechanically function in a manner entirely unlike a bird's wing, the aerodynamics of aircraft wings fully explain why birds can fly.

Racing0461 3y ago

compression is just glorified flow of electrons. Still usefull non the less.

booleandilemma 3y ago

As usual, when compression is brought up in the context of AI, it seems relevant to mention the Hutter Prize: https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize

simbolit 3y ago

think about video compression for a "zoom" call. You could just send a lot of bitmap files. Or, you could save bandwidth and zip them before send. Or you could integrate some image-specific compression into the file format, as in motion-jpeg. And in mpeg2/h264/h265 you supercharge this with temporal compression, not just adjacent pixels, not just blocks within the frame, but also pixels and blocks from adjacent frames are used to predict each pixel. And now think about not sending video, just a single initial jpeg, some face shape data, and now a continuous stream of position data of a dozen or so points on your face and the facial movements are reconstructed. (Nvidia research project from last year) And now think about no longer sending images at all, just coordinates for points in latent space (aka very abstract image descriptions) and a super-fast diffusion model "generates" the right frames on the fly. Where does "compression" end, where does "AI" start? Ultimately, the marketing people decide. Don't get hung up on the term "AI". "AI" is a bullshit hype buzzword. Calling it such serves no scientific-practical purpose, it is solely meant to impress venture capital.

aaron695 3y ago

[dead]

xeckr 3y ago

Ok. Let me know when WinRAR learns to speak.

jasfi 3y ago

I once read that intelligence is compression (or similar). An abstract way of thinking about it, but if true, then AI is on the right track.

hindsightbias 3y ago
If language models and CNNs are already creating better compression algorithms than humans, aren't they then smarter than us?
- tovej 3y ago
  In this analogy, the language model IS a man-made compression algo, so definitely not

bane 3y ago

This wasn't obvious to me a while ago, but I've come around to it. To me the important thing is that it's not only compression, it's lossy compression. Models (any kind) are basically an imperfect approximation of some system, and that definition is precisely what's identified here. You can demonstrate it by assuming some sequential process that gets an "AI" like an LLM to generate as much content as it can[1], then train a new generation model on this content. Then use the new model to generate as much content as it can, train a third generation model, and so on. LLM->generate->LLM'->generate'->LLM''->generate''->...->LLM'...' Since each generation may not produce content with every possible token it has been trained on, it stands to probability that eventually some tokens, especially low probability tokens, will simply be left out of the generated content. The next model will therefore lose that token and it will never be part of the content it generates, and so on. After enough generations of this process, you eventually end up with a model with a single, or no, tokens it can use to generate text. The problem is that the models are trained on so much information that they're effectively models of the textual content of our civilization. If we hand off the reigns to these systems, they may work well for a while, and may even seem to be produce novel ideas (but which are secretly just old ideas in combinations we haven't seen before), but they'll also produce content which will inevitably be used to train later models. As the content these models produces starts to dominate the ratio of human-text:model-text, there will be less of a reason to preserve the human text and we'll end up with the scenario above. Things may work well for a while, perhaps a long time, but even if we don't end up with models trained on one token, we will end up with fewer and fewer ideas-as-text represented within them. Civilization will stagnate, the singularity will not commence, and things will slowly regress. Of course this all presumes research stops, and somehow LLMs become religion and we turn society over to them. Which of course isn't what's going to happen. Is it?

tymscar 3y ago
I agree, however in that scenario, are people fully stopping their content generation? Because if not, the pool will grown larger and the problem you described would still be there but to a lesser extent.

furyofantares 3y ago

I love how I've seen Ilya talk about it. If we could find the shortest program to reproduce the training set, that would be optimal compression. But it's an intractable problem, we can't even come close, there's just no way to come at it. But with deep learning we can instead find a circuit that approaches reproducing the training set. This is lossy compression. There's nothing "just glorified" about it though; the result is astounding. A more appropriate takeaway might be that sufficient compression is mind-bendingly more powerful than intuition might otherwise guess. Does calling it "just glorified" guide any intuition that in order to compress amazon reviews a neural net is gonna have weights that correspond to sentiment? Does it tell you that such compression also ends up producing an artifact that can be put to work in a generative framework? And that it'll be a very useful generative framework because such compression required weights that correspond to all sorts of useful ideas, a compression of something more meaningful than just text? Calling it "just glorified X" is clickbait. It's compression alright, and it's either 1) also whole lot more, or 2) compression is a whole lot more wild and impressive than you thought, or both.

pas 3y ago
but currently it doesn't seem very useful. it hallucinates too much, there's no reflexivity, no critical reasoning. sure, nice prompts can give it some structure, and with a lot of pre- and postprocessing (ie. RAG and other ways of forcing it to pick from a list) it's good for information retrieval.
theGnuMe 3y ago
The "sentiment" weights are just the supervised classification part on top of the "compression" piece. An autoencoder is closer to pure compression. SVD is compression as well. SVD is also useful to solve some equations.

3cats-in-a-coat 3y ago

So is our brain. “Model” by definition is an approximation of something else (or the thing we’re modeling… it’s a very overloaded term all right). But if you think you can use this “realization” to dismiss AI or claim what it can or cannot do… you’re missing the forest for the trees.

TeMPOraL 3y ago

Yes. AI is just compression. So is understanding, fundamentally. That's literally what it's about.

lossolo 3y ago
If that would be the case then LLMs would know how to do math but they don't.
- mattstir 3y ago
  Popular chat-bot LLMs weren't trained to do math, they were trained to predict the next token in a conversational setting. The fact that they can do any math at all is a miracle considering that they have absolutely no thought process; they're just predicting the next token, one after the other, until some threshold is met and they stop producing output. Framing it another way, it's sort of like if you asked your calculator to have a conversation with you and it actually had a fairly decent go of it. Sure, it wasn't grammatically correct a lot of the time and it struggled quite a bit. But it wasn't at all designed to speak conversationally, so the fact that it could respond at all should be rather impressive.
- TeMPOraL 3y ago
  They can do math - they can write code to be executed. It's not much different than me or you doing any maths more complex than reciting the times table.
- lern_too_spel 3y ago
  They can write proofs, so they can do math. If you mean they're bad at arithmetic, a big part of that is improper tokenization. If you do digit level tokenization, even small transformers can learn arithmetic and generalize to longer inputs.

Arnt 3y ago

I chatted briefly to overtone.ai a few weeks ago, heading back to the hotel from a conference. What they do is train an existing LLM model to detect things about the text (overtones, I suppose you might say). What's interesting in this context is that they train the AI using an English corpus, but once trained the AI is able to detect the same traits in other languages. This sounds quite different from compression.

thargor90 3y ago
Nobody is arguing that the usecases are the same. In the end you can't even chat with gzip (although you could with it's predictor). The thing is, that building the predictor is almost the same thing for compression and LLM. Of course the goals and taken tradeoffs are different. The paper shows this analogy. ChatGPT et al use structured prediction to simulate intelligence. Building the predictor is fancy lossy compression. Questions arise if lossy compression of things without copyright is legal or not. If I mp3 a lossless recording we currently think it is not legal. With LLMs this is not entirely clear yet.
- Arnt 3y ago
  As it happens I know zip. I ported that to linux back in 0.95 days, I think the code was called info-zip in those days. Chatting with its predictor is a fanciful description to say the least. I've also read the transformers paper mentioned in the tweet. Of course an LLM is on some level similar to a compression system, and on some level it's also just high and low voltages on some integrated circuits. Saying "just" glorified compression isn't something I'll believe without good arguments, though.

jesuslop 3y ago

Liked this view always (look for Hutter prize rationale), but I think it needs to be accomodated in a general perception/action loop that optimizes a lower level fitness/utility/reward (for instance an inner sense of pleasure/pain).

dgudkov 3y ago

Is generalization also a form of compression?

RecycledEle 3y ago

It seems this X post is from someone trying to claim LLMs are copyright violations. The argument that LLMs are only a new compression algorithm is nonsense.

thealistra 3y ago

Isn’t childbirth a glorified decompression of parents’ DNA? This is a tweet level of discourse