a purely random sequence of letters has greater Shannon information than a string of words that make up a sentence, because the words and the sentence have some

a purely random sequence of letters has greater Shannon information than a string of words that make up a sentence, because the words and the sentence have some higher-order patterns in them (like statistics of letters that typically follow each other, such as a “u” following a “q”), which can be used to compress the message. A random sequence has no such patterns and thus cannot be compressed. Thinking in those terms naturally leads to the kind of “counting arguments” that Bengio makes. These seem to take each gene as a bit of information, and ask whether there are enough such bits to specify all the bits in the brain, usually taken as the number of connections. Obviously the answer is there are not enough such bits. (There aren’t even enough bits if you take individual bases of the genome as your units of information).