3 ms·

Intelligence is compression? What do you mean? Intuitively that doesn't seem right. >What I don't understand is why LLMs haven't been able to do this yet LLMs

Intelligence is compression? What do you mean? Intuitively that doesn't seem right. >What I don't understand is why LLMs haven't been able to do this yet LLMs are just trained on what humans have said. Why is it surprising that it's still not possible to reconstruct the intelligence that wrote all that by working backwards? Think of your own work experience. When you look at a piece of code, say, are you always able to discern why the person did what they did, just from the code, with no additional context?

teekert 1mo ago
I guess they mean that intelligence is being able to hold models (compressed versions of reality) internally and use them to make predictions with a probability better than chance. That last part is the definition of information.
- fluoridation 1mo ago
  I find that highly questionable as a general description of what intelligence does. That's more like a description of a general knowledge base. When I think of someone intelligent, I think of someone who's able to draw unexpected connections between seemingly unrelated facts. In the broadest possible terms, I'd call it the ability to make abstractions and analogies. This is not just compression, but the ability to mentally operate on webs of meaning.
  - rcxdude 1mo ago
    Doing those things also contributes to compression. I do recommend reading up on it, it's perhaps a little overstated for what people intuitively consider the two concepts but it's been quite well explored and has held up pretty well in practice.
  - teekert 1mo ago
    Unexpected connections between seemingly unrelated "models" :) Is a fact stored on your brain like digits on a harddrive? No, it's a pathway that lights up and branches when information enters it. It is dynamic, a compressed form you could say, right? The model holds information, but not all information, but enough to be useful (in decision making). Arguably it's the same, but the model is probably a "compressed" version of the whole fact that took place in reality. And you can entertain the models internally and sharpen them. Alone or with others.

teekert 1mo ago

I guess they mean that intelligence is being able to hold models (compressed versions of reality) internally and use them to make predictions with a probability better than chance. That last part is the definition of information.

fluoridation 1mo ago
I find that highly questionable as a general description of what intelligence does. That's more like a description of a general knowledge base. When I think of someone intelligent, I think of someone who's able to draw unexpected connections between seemingly unrelated facts. In the broadest possible terms, I'd call it the ability to make abstractions and analogies. This is not just compression, but the ability to mentally operate on webs of meaning.
- rcxdude 1mo ago
  Doing those things also contributes to compression. I do recommend reading up on it, it's perhaps a little overstated for what people intuitively consider the two concepts but it's been quite well explored and has held up pretty well in practice.
- teekert 1mo ago
  Unexpected connections between seemingly unrelated "models" :) Is a fact stored on your brain like digits on a harddrive? No, it's a pathway that lights up and branches when information enters it. It is dynamic, a compressed form you could say, right? The model holds information, but not all information, but enough to be useful (in decision making). Arguably it's the same, but the model is probably a "compressed" version of the whole fact that took place in reality. And you can entertain the models internally and sharpen them. Alone or with others.