4 ms·

That is really an awesome explanation, thank you! And thanks to the other commenters as well. I think my interpretation of the 2nd law as "static" (like on a T

That is really an awesome explanation, thank you! And thanks to the other commenters as well. I think my interpretation of the 2nd law as "static" (like on a TV) misleads me because static seems so uniform, and really increasing entropy is the opposite of uniform. It is n[i] = rand() rather than n[i] = C. So your explanation helps a lot! EDIT: So a followup question: Does information theory have anything like the 2nd law of thermodynamics? In the digital world we are used to lossless transfer, but children know that when you play "telephone" you lose information (or "gain" it in the sense the new message can't be predicted). Or maybe the analog of the 2nd law is the pigeonhole principle---I'm not sure. But if there is so much symmetry between the two kinds of entropy, is there a 2nd law for information?

jrochkind1 12y ago
Thanks! I have no idea the answer to your followup question, heh.
lovemenot 12y ago
My naive understanding of your follow up question is that energy is a special case of information and the Second Law is a special case of Shannon's Information Theory. The system in the second law is a special case of Shannon's channel. Apparent lossless digital transfer is a red herring though. Noise is inherent to any channel.
- orbifold 12y ago
  I think it is unfair to give Shannon too much credit, Boltzmann and Gibbs came up with the exact definition Shannon gave S = -k \Sigma p_i ln(p_i), including a suitable information theoretic interpretation. As physicists they only thought about ignorance of the exact particles position and momentum in systems of 10^24 particles, but the essential idea is the same. Moreover von Neumann had generalized that notion to quantum mechanics S = - tr(\rho ln(\rho)), which gives it a form that is even more abstract than the information theoretic formulation. It was only ignorance on Shannons part that he was not familiar with the work of physicists and mathematicians before him on the subject.

jrochkind1 12y ago

Thanks! I have no idea the answer to your followup question, heh.

lovemenot 12y ago

My naive understanding of your follow up question is that energy is a special case of information and the Second Law is a special case of Shannon's Information Theory. The system in the second law is a special case of Shannon's channel. Apparent lossless digital transfer is a red herring though. Noise is inherent to any channel.

orbifold 12y ago
I think it is unfair to give Shannon too much credit, Boltzmann and Gibbs came up with the exact definition Shannon gave S = -k \Sigma p_i ln(p_i), including a suitable information theoretic interpretation. As physicists they only thought about ignorance of the exact particles position and momentum in systems of 10^24 particles, but the essential idea is the same. Moreover von Neumann had generalized that notion to quantum mechanics S = - tr(\rho ln(\rho)), which gives it a form that is even more abstract than the information theoretic formulation. It was only ignorance on Shannons part that he was not familiar with the work of physicists and mathematicians before him on the subject.