4 ms·
How about: entropy := amount of disorder information := knowledge of the system (knowledge of disorder) uncertainty := unknown disorder (entropy - informatio
by hackinthebochs 12y ago
How about:
entropy := amount of disorder
information := knowledge of the system (knowledge of disorder)
uncertainty := unknown disorder (entropy - information)
And so entropy and information are opposites: as you gain more information about a system the uncertainty is reduced. If the entropy of a system increase, the amount of information required to describe it increases. If you have an amount of information equal to the entropy, your uncertainty is zero and the system is completely described. This seems to square with our usage of the terms in the context of thermodynamics and information theory.
- tjradcliffe 12y agoAnother way to look at this is the process of reading a stream of bytes. A high-entropy stream means that based on what you have read so far you have a very small chance of predicting the next byte you read. By the same token (as it were) in those circumstances the next byte you read will contain a great deal of information about its value that you did not previously have. In a low-entropy stream the bytes coming in might be: 0, 1, 2, 3... and by the time you get to byte N you can be pretty sure of its value, so the next byte contains very little information you don't already have. Both information and entropy are measures of the novelty of the next byte in the stream, but information is measured from the perspective of what you get when you read it and entropy is measured from the perspective of what you have before you read it.