It's worth emphasizing that information and entropy are subjective, in the Bayesian sense. The more you know and understand something, the more you can encode i

It's worth emphasizing that information and entropy are subjective, in the Bayesian sense. The more you know and understand something, the more you can encode it in a small number of bits. Consider the sequence: 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97 Most people would have a hard time remembering it because it looks random... But of course, it really is "All the primes between 30 and 100". That's much easier to remember, because the brain (at least, the average brain on HN) already has concepts for "prime number", "30" and "100". Compression has a strong connection to AI. In fact, Occam's razor is perhaps AI's most fundamental theorem: if you have derived many programs which can perform the same task, which one is the smartest? A: The shortest one. I would also recommend MacKay's book. It's awesome.