3 ms·
That's something of a myth. Or, rather, English happens to have so many short words that only a few permutations are possible if you fix the first and last lett
by Sharlin 3y ago
That's something of a myth. Or, rather, English happens to have so many short words that only a few permutations are possible if you fix the first and last letters. In something like German it would absolutely not work as well. And even in English, it becomes more difficult if you don't cherry-pick fairly simple permutations that preserve some important structure of the word, like "deffinert" or "porblem".
But on the other hand, yes, natural language has low entropy and lots of redundancy because it's meant to have error correction. From most sentences you can drop several words, not just scramble them, and the meaning can be inferred from the context.
- godelski 3y agoIt's not a myth, it is a hallucination that works in English. Which used because we're all speaking that language here. Like I said above, it is a different bias, but we have different biases than LLMs. Every language is going to have different biases. Some classes of heteronyms don't even exist in certain languages so a similar thing to The Chaos doesn't work as well there either. Maybe Chomsky would say there are more clear examples since he believes in a universal grammar but I'm also not a linguist. Just wanted an example. I wouldn't call this cherry picking as it both works in a few latin languages as well as strongly works in English. But even in English there is a bit of care you need, like double letters help to be stuck together. That's not cherry picking though as the example isn't such a unique thing that it invalidates the main thesis (what cherry picking is about).
- Sharlin 3y agoIt’s not a hallucination. It’s simply error correction, just like optical disks or ECC RAM are resistant to scrambled bits thanks to redundant, low-entropy coding – the fact that the bits are correlated.