4 ms·
Well, the entropy in information theory is randomness too, in the sense that it's _unpredictable_, right? If you've got 1000 numbers, but you can predict them
by jrochkind1 12y ago
Well, the entropy in information theory is randomness too, in the sense that it's _unpredictable_, right?
If you've got 1000 numbers, but you can predict them all with "n[i] = i * 2", then there's not that much information present, in the information theoretical entropy sense -- all you need is the formula `n[i] = i * 2` to describe all 1000 numbers, actually writing all 1000 (or 100,000) out just wastes space without providing any more information beyond `n[i] = i * 2 for i=1..1000` or whatever.
But if the 1000 numbers are truly random, you need all 1000 numbers to describe the set of 1000 numbers in it. There is more information present, there is more entropy, precisely because of the randomness or unpredictability.
Or to put it another way, if you can compress something losslessly, then it makes sense to think the uncompressed version doesn't have more "information" in it than the compressed version, even though it's more bytes, right? And what can't be compressed losslessly at all? Something entirely random with no predictability.
So that's my understanding of the relationship -- information theoretic entropy is, in fact, randomness too. Random here meaning 'unpredictable', not describable by any formula or shortcut or compression -- the normal way we use the term 'random' in day-to-day software engineering, in fact, and the meaning that's crucial for cryptography for instance. (Then, as now, military budgets drove computer science research -- Shannon was a cryptographer on a DoD contract) (It doesn't neccesarily mean the 'random' data is _senseless_, just that it's entirely irregular and unpredictable).
So, okay, now I'm grasping and I'm not sure what I'm talking about, but in this sense, as the entropy in the universe increases, the 'information' in the universe increases in the sense that there's no shortcut for describing the universe's state, it's increasingly just randomly distributed stuff?
I don't know if that last paragraph makes sense, or if you can really talk at all about 'the amount of information in the universe increasing' -- I vaguely recall reading stuff on that before, the relationship between physical and information entropy, but I don't remember enough to speak on it really.
But my understanding is that Shannon was inspired by the thermodynamic entropy in inventing information theory, realizing that the amount of randomness present in a sequence of data, the amount of unpredictability in it, was usefully understood as 'how much' information it contained. The "information" in "information theory" is not the plain-use meaning of the word 'information', although it has a relationship to it. But it means, well, precisely this, entropy or unpredictability. And that meaning has proven useful for reasoning about computation. But it's a low-level mathematical/computational sort of 'information', not information like answers to trivia questions.
- pjungwir 12y agoThat is really an awesome explanation, thank you! And thanks to the other commenters as well. I think my interpretation of the 2nd law as "static" (like on a TV) misleads me because static seems so uniform, and really increasing entropy is the opposite of uniform. It is n[i] = rand() rather than n[i] = C. So your explanation helps a lot! EDIT: So a followup question: Does information theory have anything like the 2nd law of thermodynamics? In the digital world we are used to lossless transfer, but children know that when you play "telephone" you lose information (or "gain" it in the sense the new message can't be predicted). Or maybe the analog of the 2nd law is the pigeonhole principle---I'm not sure. But if there is so much symmetry between the two kinds of entropy, is there a 2nd law for information?
- jrochkind1 12y agoThanks! I have no idea the answer to your followup question, heh.
- lovemenot 12y agoMy naive understanding of your follow up question is that energy is a special case of information and the Second Law is a special case of Shannon's Information Theory. The system in the second law is a special case of Shannon's channel. Apparent lossless digital transfer is a red herring though. Noise is inherent to any channel.
- orbifold 12y agoI think it is unfair to give Shannon too much credit, Boltzmann and Gibbs came up with the exact definition Shannon gave S = -k \Sigma p_i ln(p_i), including a suitable information theoretic interpretation. As physicists they only thought about ignorance of the exact particles position and momentum in systems of 10^24 particles, but the essential idea is the same. Moreover von Neumann had generalized that notion to quantum mechanics S = - tr(\rho ln(\rho)), which gives it a form that is even more abstract than the information theoretic formulation. It was only ignorance on Shannons part that he was not familiar with the work of physicists and mathematicians before him on the subject.