3 ms·
The lossy compression of language is why we should find it unsurprising that LLMs tend to perform better at code, than at human language tasks or reasoning. Whi
by lukifer 2mo ago
The lossy compression of language is why we should find it unsurprising that LLMs tend to perform better at code, than at human language tasks or reasoning. While there can be subtle semantic differences in real codebases (using "null" to mean "unknown" in one context, versus "intentionally blank" in another), there is a much tighter coupling of semantics to meaning (low ambiguity) compared to "love" in English (let alone any inexpressible je ne sais quoi).
With apologies if this is common knowledge at this point, 3Blue1Brown has been doing an excellent series on compression, and its relationship to intelligence (or more controversially, that they are one and the same): https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=l6DKRf-fAAM
But that also throws in sharp relief, that there is vastly more to the human experience than intelligence alone: qualia, desire, gut instinct, intuition, emergent creativity. (Whether the "God of the Gaps" for the delta between capabilities of human vs AIs is fixed, or diminishing, or even shrinking to zero, remains an open experiment we're all living through.)
- cwmoore 2mo agoI have come across a concept that given a file of compressed text files, adding a new text to it expands it more or less depending on how different the new text is from the compressed. The compression series sounds interesting! I think the deltas are growing at different rates.
- Hunpeter 2mo ago> given a file of compressed text files, adding a new text to it expands it more or less depending on how different the new text is from the compressed. The 3b1b videos mention how this concept may be used to find similarities between different languages. Researchers have been able to get results that closely resemble how languages are usually grouped into families.
- nullbio 2mo agoI think LLMs perform better at code than human language tasks because there's no clear way to eval human language tasks in a non-ambiguous or concrete manner. Any sort of eval that happens around language is transformation tasks, which have deterministic properties or goals. The fuzzy side of human communication can only be modeled probabilistically because there are no clear boundaries. Human communication is more like a felt mutual agreement where the correct interpretation is generally determined by popularity.
- scotty79 2mo ago> we should find it unsurprising that LLMs tend to perform better at code, than at human language tasks or reasoning Math is pure reasoning and stock LLM is already better at it than 99.9% of humans. And as for human language, agentic LLM can produce a perfectly human text. The fact that AI texts have tells is just because default settings are the same for millions of people using given LLM and almost everybody just pretty much on-shots the text instead of doing it agentically with anti-slop detection and rephrasing in the loop.
- vrighter 2mo agoI also think they are one and the same. Because a network is a markov chain, with a huge, lossily compressed, state transition table. (that the chains weren't designed by a human is irrelevant to what they are). given that, you can very mechanically convert it to a statistical compressor by just attaching an entropy coder. This gets very good compression.