4 ms·
Perhaps we should judge the performance of an LLM by how well it can compress arbitrary information. A higher IQ would mean more compression, after all.
by amelius 1y ago
Perhaps we should judge the performance of an LLM by how well it can compress arbitrary information. A higher IQ would mean more compression, after all.
- a_shovel 1y agoWhich LLMs perform better or worse will be determined entirely by the scoring formula used and how it penalizes errors. It is not in the nature of an LLM to be capable of lossless compression.
- optimalsolver 1y agoLossily or losslessly?
- amelius 1y agoLosslessly, because that's easier to test.
- optimalsolver 1y agoWhen you're recalling a memory, do you remember the position of every blade of grass, or the exact angle of the Sun? Humans, the only extant example of a general intelligence, don't do lossless compression at all. I don't think you get to AGI by trying to compress noise.
- amelius 1y agoLossless compression requires the recognition of patterns in the data. And a smart way to use those patterns to perform the actual compression.
- Legend2440 1y agoWe already do this. Prediction is formally equivalent to compression, so loss is just a measure of how well you can compress the training dataset.
- amelius 1y agoIn a sense, yes. But that's compression of an input/output map, not arbitrary information. Also, it is not lossless.