4 ms·

Shannon / information entropy absolutely is fundamental to thermodynamics. It clears up a lot of confusion when you realize that the entropy isn't in the system

Shannon / information entropy absolutely is fundamental to thermodynamics. It clears up a lot of confusion when you realize that the entropy isn't in the system itself, it's in your head (your lack of information about a system's exact state, knowing only summary statistics like its mass, volume, and temperature).

antpls 7y ago
I studied Shannon entropy like any CS student back in University, so I'm not an expert in all interpretations of Shannon entropy, but I don't think Shannon entropy tells you anything about what or how much information you are lacking about the system. It may be used to this end, but it's not what Shannon entropy directly gives you. There is no way to reduce the entropy of a totally random system by knowing more about it, unless your hypothesis is that true randomness doesn't exist in the universe
- jbay808 7y ago
  To connect them, the entropy of a system (such as an ideal gas) can be computed as the logarithm of the number of equally probable microstates in your probability distribution for that system. (You need to define a sensible measure, because the state variables are continuous, but that's not important). You can express this in terms of the message length or number of bits of information you would need to gain about that container of ideal gas in order to narrow it down to a particular microstate. It's no different than the entropy of, say, a password. > There is no way to reduce the entropy of a totally random system by knowing more about it There absolutely is! For example, Maxwell's demon. Maxwell's demon will fail to reduce system entropy if it doesn't know the microstate to begin with, but otherwise it can use its knowledge to reduce the entropy of a gas by separating its parts. > unless your hypothesis is that true randomness doesn't exist in the universe I do also subscribe to this hypothesis, that like entropy, randomness also only exists in our heads as a reflection of our lack of information about a system, and isn't a property of any system itself. But I don't think I need to rely on this assumption.
  - dr_dshiv 7y ago
    With the entropy of a password, abcdefg has less entropy than kfbeksn because it is more likely to occur. Further, the calculation of entropy is based on an alphabet of symbols, shared by sender and receiver. I don't see how that connects to the ideal gas example. Care to share?
    - jbay808 7y ago
      I'm on mobile so it's hard to write a lot of detail. But here's probably the most straightforward way to clarify it. Gasses are the same in principle but harder to visualize concretely, so let's think instead of a crystal lattice with impurities -- for example a doped silicon wafer. I hand you a 28 gram disc of 0.1% n-doped silicon. It's got about 6e23 atoms in it, of which 0.1% are phosphorus. That's a good statistical description of the crystal, but there are still a large number of possible configurations of phosphorus and silicon atoms that match that specification, but only one of them describes what you are holding in your hand. For whatever reason, you want to know the exact atomic arrangement of this particular wafer. Fortunately, I've imaged this disc with an advanced electron microscope that has identified the identity of each atom in the crystal lattice, whether silicon or phosphorus. How large is the data file that I have to send you? How much can I compress my electron microscope data? Just like "abcdefg is less likely than kfbeksn", we have some prior information we can use to do compression. For example, dopant atoms are unlikely to cluster together, so Si-Si-P-P-P-Si-Si is less likely than P-Si-Si-P-Si-Si-P. That prior information is no harder to incorporate into the entropy calculation than it is in the case of passwords. But such subtleties aside, we can start by assuming that any configuration is equally likely, compute the total number of configurations, and take the log base 2. That's going to be the minimum file size of the electron microscope scan. (I guesstimated 6e12 Tb but don't quote me on that). To do a proper job, we'd also need to measure and transmit the state vector of the lattice vibrations, but I'm ignoring that for now. Ok, so that's for a crystal, which I chose because the electron microscope technology is easy to concretely visualize, and the state can be expressed as a string of Si's and P's. But an ideal gas is similar in principle, except its state variable consists of a string of positions and velocity values. We will have to discretize to some resolution to get discrete states, but that's fine. There is some datafile I could send you that would (quantum uncertainty aside) let you know the exact state of the gas, and with fine manipulators you could use that data to sort the gas molecules into different containers by molecular energy and reduce its entropy to zero. The entropy of the data you'd need to do that would be at least as high as the entropy of the gas, and upon clearing that data from your memory, the second law is satisfied.
cygx 7y ago
I would argue that's exactly the wrong way to think about thermodynamic entropy. Consider this: Energy isn't just in your head (you can use it to do work, and you have to know where it is precisely if you want to do General Relativity). Temperature isn't just in your head (you can measure it with a thermometer, and it tends to manifest as an average energy). The laws of thermodynamics then connect energy and temperature to entropy, which implies that the latter also can't just be in your head unless you add some mubo-jumbo. Thermodynamics doesn't care about our ignorance of potential system configurations, but that these configurations are available for microscopic system evolution!
- kgwgk 7y ago
  > Thermodynamics doesn't care about our ignorance of potential system configurations, but that these configurations are available for microscopic system evolution! Thermodynamics equations are based on the macroscopic constraints on the system that we choose to measure and manipulate, implicitely admitting our ignorance about the potential microscopic configurations. Thermodynamic entropy, unlike energy, is not an intrinsic property of the physical system; it is a property of our macroscopic description of the system and the experimental processes that we decide to perform on it. See section VI. THE "ANTHROPOMORPHIC" NATURE OF ENTROPY in https://bayes.wustl.edu/etj/articles/gibbs.vs.boltzmann.pdf https://bayes.wustl.edu/etj/articles/gibbs.vs.boltzmann.pdf "Recognition that the "entropy of a physical system" is not meaningful without further qualifications is important in clarifying many questions concerning irreversibility and the second law. For example, I have been asked several times whether, in my opinion, a biological system, say a cat, which converts inanimate food into a highly organized structure and behavior, represents a violation of the second law. The answer I always give is that, until we specify the set of parameters which define the thermodynamic state of the cat, no definite question has been asked!"
- jbay808 7y ago
  Energy and entropy are quite different though; energy certainly is a property of the system itself. Temperature, interestingly, is also in our heads. It isn't actually a measure of average energy, but actually is defined as a function of energy and entropy. (Entropy comes first and temperature is defined in terms of it, not the other way around). T = dQ/dS You can create systems in real life with infinite temperature, or even with negative temperature, but of course they certainly don't have infinite energy. One way to do so is to pump a system into having a population inversion, such that its entropy decreases as its energy increases, because the number of possible high energy microstates is small. In general, only one microstate is ever "available" for a system to evolve into, which is the one that its current microstate will evolve into in accordance with the laws of physics. The trouble is that we don't know its current microstate, so we don't know its future microstate, so it seems to us that there are many available microstates, but that's all due to our own ignorance having only conducted summary statistical measurements and isn't an intrinsic property of the system.

antpls 7y ago

I studied Shannon entropy like any CS student back in University, so I'm not an expert in all interpretations of Shannon entropy, but I don't think Shannon entropy tells you anything about what or how much information you are lacking about the system. It may be used to this end, but it's not what Shannon entropy directly gives you. There is no way to reduce the entropy of a totally random system by knowing more about it, unless your hypothesis is that true randomness doesn't exist in the universe

jbay808 7y ago
To connect them, the entropy of a system (such as an ideal gas) can be computed as the logarithm of the number of equally probable microstates in your probability distribution for that system. (You need to define a sensible measure, because the state variables are continuous, but that's not important). You can express this in terms of the message length or number of bits of information you would need to gain about that container of ideal gas in order to narrow it down to a particular microstate. It's no different than the entropy of, say, a password. > There is no way to reduce the entropy of a totally random system by knowing more about it There absolutely is! For example, Maxwell's demon. Maxwell's demon will fail to reduce system entropy if it doesn't know the microstate to begin with, but otherwise it can use its knowledge to reduce the entropy of a gas by separating its parts. > unless your hypothesis is that true randomness doesn't exist in the universe I do also subscribe to this hypothesis, that like entropy, randomness also only exists in our heads as a reflection of our lack of information about a system, and isn't a property of any system itself. But I don't think I need to rely on this assumption.
- dr_dshiv 7y ago
  With the entropy of a password, abcdefg has less entropy than kfbeksn because it is more likely to occur. Further, the calculation of entropy is based on an alphabet of symbols, shared by sender and receiver. I don't see how that connects to the ideal gas example. Care to share?
  - jbay808 7y ago
    I'm on mobile so it's hard to write a lot of detail. But here's probably the most straightforward way to clarify it. Gasses are the same in principle but harder to visualize concretely, so let's think instead of a crystal lattice with impurities -- for example a doped silicon wafer. I hand you a 28 gram disc of 0.1% n-doped silicon. It's got about 6e23 atoms in it, of which 0.1% are phosphorus. That's a good statistical description of the crystal, but there are still a large number of possible configurations of phosphorus and silicon atoms that match that specification, but only one of them describes what you are holding in your hand. For whatever reason, you want to know the exact atomic arrangement of this particular wafer. Fortunately, I've imaged this disc with an advanced electron microscope that has identified the identity of each atom in the crystal lattice, whether silicon or phosphorus. How large is the data file that I have to send you? How much can I compress my electron microscope data? Just like "abcdefg is less likely than kfbeksn", we have some prior information we can use to do compression. For example, dopant atoms are unlikely to cluster together, so Si-Si-P-P-P-Si-Si is less likely than P-Si-Si-P-Si-Si-P. That prior information is no harder to incorporate into the entropy calculation than it is in the case of passwords. But such subtleties aside, we can start by assuming that any configuration is equally likely, compute the total number of configurations, and take the log base 2. That's going to be the minimum file size of the electron microscope scan. (I guesstimated 6e12 Tb but don't quote me on that). To do a proper job, we'd also need to measure and transmit the state vector of the lattice vibrations, but I'm ignoring that for now. Ok, so that's for a crystal, which I chose because the electron microscope technology is easy to concretely visualize, and the state can be expressed as a string of Si's and P's. But an ideal gas is similar in principle, except its state variable consists of a string of positions and velocity values. We will have to discretize to some resolution to get discrete states, but that's fine. There is some datafile I could send you that would (quantum uncertainty aside) let you know the exact state of the gas, and with fine manipulators you could use that data to sort the gas molecules into different containers by molecular energy and reduce its entropy to zero. The entropy of the data you'd need to do that would be at least as high as the entropy of the gas, and upon clearing that data from your memory, the second law is satisfied.

cygx 7y ago

I would argue that's exactly the wrong way to think about thermodynamic entropy. Consider this: Energy isn't just in your head (you can use it to do work, and you have to know where it is precisely if you want to do General Relativity). Temperature isn't just in your head (you can measure it with a thermometer, and it tends to manifest as an average energy). The laws of thermodynamics then connect energy and temperature to entropy, which implies that the latter also can't just be in your head unless you add some mubo-jumbo. Thermodynamics doesn't care about our ignorance of potential system configurations, but that these configurations are available for microscopic system evolution!

kgwgk 7y ago
> Thermodynamics doesn't care about our ignorance of potential system configurations, but that these configurations are available for microscopic system evolution! Thermodynamics equations are based on the macroscopic constraints on the system that we choose to measure and manipulate, implicitely admitting our ignorance about the potential microscopic configurations. Thermodynamic entropy, unlike energy, is not an intrinsic property of the physical system; it is a property of our macroscopic description of the system and the experimental processes that we decide to perform on it. See section VI. THE "ANTHROPOMORPHIC" NATURE OF ENTROPY in https://bayes.wustl.edu/etj/articles/gibbs.vs.boltzmann.pdf https://bayes.wustl.edu/etj/articles/gibbs.vs.boltzmann.pdf "Recognition that the "entropy of a physical system" is not meaningful without further qualifications is important in clarifying many questions concerning irreversibility and the second law. For example, I have been asked several times whether, in my opinion, a biological system, say a cat, which converts inanimate food into a highly organized structure and behavior, represents a violation of the second law. The answer I always give is that, until we specify the set of parameters which define the thermodynamic state of the cat, no definite question has been asked!"
jbay808 7y ago
Energy and entropy are quite different though; energy certainly is a property of the system itself. Temperature, interestingly, is also in our heads. It isn't actually a measure of average energy, but actually is defined as a function of energy and entropy. (Entropy comes first and temperature is defined in terms of it, not the other way around). T = dQ/dS You can create systems in real life with infinite temperature, or even with negative temperature, but of course they certainly don't have infinite energy. One way to do so is to pump a system into having a population inversion, such that its entropy decreases as its energy increases, because the number of possible high energy microstates is small. In general, only one microstate is ever "available" for a system to evolve into, which is the one that its current microstate will evolve into in accordance with the laws of physics. The trouble is that we don't know its current microstate, so we don't know its future microstate, so it seems to us that there are many available microstates, but that's all due to our own ignorance having only conducted summary statistical measurements and isn't an intrinsic property of the system.