3 ms·

No, you're right, it's just a terminological morass. Entropy and information are different names for the same concept, one that in its full generality is simply

by pash 12y ago

No, you're right, it's just a terminological morass. Entropy and information are different names for the same concept, one that in its full generality is simply a measure of the uncertainty about the value of a random variable. Entropy is a bad name, one that will probably continue to confuse people for generations. Information is a decent name, but it's a bit backwards: information is the resolution of uncertainty. Really, the concepts implied by the names information and uncertainty are opposite sides of the same coin: for each unit of uncertainty, you gain one unit of information when you observe a random variable's outcome. The problem is that in the conventional definition (with the negative sign in front of the sum), a more positive quantity denotes more uncertainty. Perhaps the conventional quantity should have been called uncertainty and the name information should have been given to the inverse quantity, i.e., to the sum without the negative sign. As it is, people usually use the word information as a synonym for entropy or uncertainty, but when they focus on resolving uncertainty they sometimes use it to mean something like the opposite. In the end, so long as you are confident in you understanding of the concept, it doesn't much matter because it's easy to figure out what everybody means.

hackinthebochs 12y ago
How about: entropy := amount of disorder information := knowledge of the system (knowledge of disorder) uncertainty := unknown disorder (entropy - information) And so entropy and information are opposites: as you gain more information about a system the uncertainty is reduced. If the entropy of a system increase, the amount of information required to describe it increases. If you have an amount of information equal to the entropy, your uncertainty is zero and the system is completely described. This seems to square with our usage of the terms in the context of thermodynamics and information theory.
- tjradcliffe 12y ago
  Another way to look at this is the process of reading a stream of bytes. A high-entropy stream means that based on what you have read so far you have a very small chance of predicting the next byte you read. By the same token (as it were) in those circumstances the next byte you read will contain a great deal of information about its value that you did not previously have. In a low-entropy stream the bytes coming in might be: 0, 1, 2, 3... and by the time you get to byte N you can be pretty sure of its value, so the next byte contains very little information you don't already have. Both information and entropy are measures of the novelty of the next byte in the stream, but information is measured from the perspective of what you get when you read it and entropy is measured from the perspective of what you have before you read it.
derefr 12y ago
If you understand quantum mechanics, it's pretty easy to map the process of quantum decoherence to the increase of entropy. It can be more clearly said, then, that quantum coherence is a measure of negentropy: in classical terms, uncertainty, but in quantum terms, the amount of computational power the system has—the amount of certainty it can be used to create. If you think of our universe as one big quantum computer, it started with some amount of negentropy, and is slowly "burning through" it by resolving quantum predecessor states to their ultimate physical conclusions. When all the negentropy is gone, the system will be in a (quantum-)predictable state. (It might be a classically predictable state, too, now that I think about it; if the universe only contains atomic hydrogen spread evenly across the universe at an exactly even temperature, then I think you get "for free" both a reading of the position and momentum of all those atoms, no?) Anyway, that's all to say: you can measure the power of a quantum-coherent state in "bits of negentropy"; when that state collapses/decoheres, you will know N more bits of information than you knew before it collapsed. The negentropy (potential to answer questions) in the system has been converted into entropy (answers to questions). And that's the strangest thing about all this—that it falls out that "ability to increase certainty" is the ultimate conserved resource in our universe, the ultimate clamp on thermodynamics. A certain clump of particles, as a closed system, can only be used to increase your certainty by N bits; after that, the system is useless for answering any further questions about anything, ever again. And, on the other hand, think about things like electron spin: as long as no decoherence is occurring—no certainty is being created—thermodynamics doesn't care if something is acting on something else with a force, or anything like that. There's not really such a thing as "energy" in the universe, no requirement that every ongoing force have a "casting cost" in negentropy. There's only decoherence. Heat-death-universe hydrogen atoms are perpetual motion machines, for free, only and precisely because their perpetual motion will never compute a thing.
- pash 12y ago
  In the Bayesian and inductive-logical view of probability theory, you must define a probability distribution with respect to an information set. That set formalizes your observations of (and assumptions about) the state of the universe, and the entropy of the resulting distribution quantifies the quality of your knowledge about the universe. From that perspective, the second law of thermodynamics simply says that the quality of your knowledge about the universe with respect to an information set assembled at time t degrades as you move from t out into the future. Put like that, it seems rather obvious, I think. Quantum theory further tells us that it's not possible to assemble an information set that is complete, in either the sense that (a) we can get a perfect picture of the universe as it is right now, or that (b) the quality of our information won't degrade over time. (The two senses are equivalent, and you can probably see why.) So Laplace's demon can't exist. Which isn't terribly surprising to the modern mind, either, but I can imagine how scientists of a previous era might have taken the news badly.

by pash 12y ago

hackinthebochs 12y ago

How about: entropy := amount of disorder information := knowledge of the system (knowledge of disorder) uncertainty := unknown disorder (entropy - information) And so entropy and information are opposites: as you gain more information about a system the uncertainty is reduced. If the entropy of a system increase, the amount of information required to describe it increases. If you have an amount of information equal to the entropy, your uncertainty is zero and the system is completely described. This seems to square with our usage of the terms in the context of thermodynamics and information theory.

tjradcliffe 12y ago
Another way to look at this is the process of reading a stream of bytes. A high-entropy stream means that based on what you have read so far you have a very small chance of predicting the next byte you read. By the same token (as it were) in those circumstances the next byte you read will contain a great deal of information about its value that you did not previously have. In a low-entropy stream the bytes coming in might be: 0, 1, 2, 3... and by the time you get to byte N you can be pretty sure of its value, so the next byte contains very little information you don't already have. Both information and entropy are measures of the novelty of the next byte in the stream, but information is measured from the perspective of what you get when you read it and entropy is measured from the perspective of what you have before you read it.

derefr 12y ago

If you understand quantum mechanics, it's pretty easy to map the process of quantum decoherence to the increase of entropy. It can be more clearly said, then, that quantum coherence is a measure of negentropy: in classical terms, uncertainty, but in quantum terms, the amount of computational power the system has—the amount of certainty it can be used to create. If you think of our universe as one big quantum computer, it started with some amount of negentropy, and is slowly "burning through" it by resolving quantum predecessor states to their ultimate physical conclusions. When all the negentropy is gone, the system will be in a (quantum-)predictable state. (It might be a classically predictable state, too, now that I think about it; if the universe only contains atomic hydrogen spread evenly across the universe at an exactly even temperature, then I think you get "for free" both a reading of the position and momentum of all those atoms, no?) Anyway, that's all to say: you can measure the power of a quantum-coherent state in "bits of negentropy"; when that state collapses/decoheres, you will know N more bits of information than you knew before it collapsed. The negentropy (potential to answer questions) in the system has been converted into entropy (answers to questions). And that's the strangest thing about all this—that it falls out that "ability to increase certainty" is the ultimate conserved resource in our universe, the ultimate clamp on thermodynamics. A certain clump of particles, as a closed system, can only be used to increase your certainty by N bits; after that, the system is useless for answering any further questions about anything, ever again. And, on the other hand, think about things like electron spin: as long as no decoherence is occurring—no certainty is being created—thermodynamics doesn't care if something is acting on something else with a force, or anything like that. There's not really such a thing as "energy" in the universe, no requirement that every ongoing force have a "casting cost" in negentropy. There's only decoherence. Heat-death-universe hydrogen atoms are perpetual motion machines, for free, only and precisely because their perpetual motion will never compute a thing.

pash 12y ago
In the Bayesian and inductive-logical view of probability theory, you must define a probability distribution with respect to an information set. That set formalizes your observations of (and assumptions about) the state of the universe, and the entropy of the resulting distribution quantifies the quality of your knowledge about the universe. From that perspective, the second law of thermodynamics simply says that the quality of your knowledge about the universe with respect to an information set assembled at time t degrades as you move from t out into the future. Put like that, it seems rather obvious, I think. Quantum theory further tells us that it's not possible to assemble an information set that is complete, in either the sense that (a) we can get a perfect picture of the universe as it is right now, or that (b) the quality of our information won't degrade over time. (The two senses are equivalent, and you can probably see why.) So Laplace's demon can't exist. Which isn't terribly surprising to the modern mind, either, but I can imagine how scientists of a previous era might have taken the news badly.