3 ms·
A better example is the difference between the contents of a normal English book vs a randomly generated string of letters. The former has a lot of structural
by yters 7y ago
A better example is the difference between the contents of a normal English book vs a randomly generated string of letters. The former has a lot of structural properties the latter lacks, plus shares a large amount of mutual information with a source independent from the physical properties of the book.
- ganzuul 7y agoWhen we cast our randomness as a string we impose a lot of structure. Proportionally, getting to English from there might involve a lot less new structure than in the previous step. Often I wonder if I'm just spouting gibberish, saying things like this, but it does make sense to me.
- yters 7y agoNo, the information cost grows exponentially going from random letters to English sentences. We can measure this with crossentropy to get a probability of generating a well formed English sentence from a uniform distribution over the letters. To generate just a few hundred words we end up needing more trials than are available in the lifespan of our universe.