4 ms·

I really liked the approach my stat mech teacher used. In nearly all situations, entropy just ends up being the log of the number of ways a system can be arran

by dekhn 2y ago

I really liked the approach my stat mech teacher used. In nearly all situations, entropy just ends up being the log of the number of ways a system can be arranged (https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula) although I found it easiest to think in terms of pairs of dice rolls.

petsfed 2y ago
And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties. Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.
- enugu 2y ago
  Assuming each of the N microstates for a given macrostate are equally possible with probability p=1/N, the Shannon Entropy is -Σp.log(p) = -N.p.log(p)=-1.log(1/N)=log(N), which is the physics interpretation. In the continuous version, you would get log(V) where V is the volume in phase space occupied by the microstates for a given macrostate. Liouville's theorem that the volume is conserved in phase space implies that any macroscopic process can only move all the microstates from a macrostate A into a macrostate B only if the volume of B is bigger than the volume of A. This implies that the entropy of B should be bigger than the entropy of A which is the Second Law.
  - cubefox 2y ago
    The second law of thermodynamics is time-asymmetric, but the fundamental physical laws are time-symmetric, so from them you can only predict that the entropy of B should be bigger than the entropy of A irrespective of whether B is in the future or the past of A. You need the additional assumption (Past Hypothesis) that the universe started in a low entropy state in order to get the second law of thermodynamics. > If our goal is to predict the future, it suffices to choose a distribution that is uniform in the Liouville measure given to us by classical mechanics (or its quantum analogue). If we want to reconstruct the past, in contrast, we need to conditionalize over trajectories that also started in a low-entropy past state — that the “Past Hypothesis” that is required to get stat mech off the ground in a world governed by time-symmetric fundamental laws. https://www.preposterousuniverse.com/blog/2013/07/09/cosmology-and-the-past-hypothesis/ https://www.preposterousuniverse.com/blog/2013/07/09/cosmolo...
    - kgwgk 2y ago
      The second law of thermodynamics is about systems that are well described by a small set of macroscopic variables. The evolution of an initial macrostate prepared by an experimenter who can control only the macrovariables is reproducible. When a thermodynamical system is prepared in such a reproducible way the preparation is happening in the past, by definition. The second law is about how part of the information that we had about a system - constrained to be in a macrostate - is “lost” when we “forget” the previous state and describe it using just the current macrostate. We know more precisely the past than the future - the previous state is in the past by definition.
- kgwgk 2y ago
  > not very compatible with Shannon's usage in any but the shallowest sense The connection is not so shallow, there are entire books based on it. “The concept of information, intimately connected with that of probability, gives indeed insight on questions of statistical mechanics such as the meaning of irreversibility. This concept was introduced in statistical physics by Brillouin (1956) and Jaynes (1957) soon after its discovery by Shannon in 1948 (Shannon and Weaver, 1949). An immense literature has since then been published, ranging from research articles to textbooks. The variety of topics that belong to this field of science makes it impossible to give here a bibliography, and special searches are necessary for deepening the understanding of one or another aspect. For tutorial introductions, somewhat more detailed than the present one, see R. Balian (1991-92; 2004).” https://arxiv.org/pdf/cond-mat/0501322 https://arxiv.org/pdf/cond-mat/0501322
  - petsfed 2y ago
    I don't dispute that the math is compatible. The problem is the interpretation thereof. When I say "shallowest", I mean the implications of each are very different. Insofar as I'm aware, there is no information-theoretic equivalent to the 2nd or 3rd laws of thermodynamics, so the intuition a student works up from physics about how and why entropy matters just doesn't transfer. Likewise, even if an information science student is well versed in the concept of configuration entropy, that's 15 minutes of one lecture in statistical thermodynamics. There's still the rest of the course to consider.
abetusk 2y ago
Also known as "the number of bits to describe a system". For example, 2^N equally probable states, N bits to describe each state.
Lichtso 2y ago
The "can be arranged" is the tricky part. E.g. you might know from context that some states are impossible (where the probability distribution is zero), even though they combinatorially exist. That changes the entropy to you. That is why information and entropy are different things. Entropy is what you know you do not know. That knowledge of the magnitude of the unknown is what is being quantified. Also, the point where I think the article is wrong (or not concise enough) as it would include the unknown unknowns, which are not entropy IMO: > I claim it’s the amount of information we don’t know about a situation
- slashdave 2y ago
  Exactly. If you want to reuse the term "entropy" in information theory, then fine. Just stop trying to make a physical analogy. It's not rigorous.
akira2501 2y ago
I spend time just staring at the graph on this page. https://en.wikipedia.org/wiki/Thermodynamic_beta https://en.wikipedia.org/wiki/Thermodynamic_beta

by dekhn 2y ago

petsfed 2y ago

And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties. Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.

enugu 2y ago
Assuming each of the N microstates for a given macrostate are equally possible with probability p=1/N, the Shannon Entropy is -Σp.log(p) = -N.p.log(p)=-1.log(1/N)=log(N), which is the physics interpretation. In the continuous version, you would get log(V) where V is the volume in phase space occupied by the microstates for a given macrostate. Liouville's theorem that the volume is conserved in phase space implies that any macroscopic process can only move all the microstates from a macrostate A into a macrostate B only if the volume of B is bigger than the volume of A. This implies that the entropy of B should be bigger than the entropy of A which is the Second Law.
- cubefox 2y ago
  The second law of thermodynamics is time-asymmetric, but the fundamental physical laws are time-symmetric, so from them you can only predict that the entropy of B should be bigger than the entropy of A irrespective of whether B is in the future or the past of A. You need the additional assumption (Past Hypothesis) that the universe started in a low entropy state in order to get the second law of thermodynamics. > If our goal is to predict the future, it suffices to choose a distribution that is uniform in the Liouville measure given to us by classical mechanics (or its quantum analogue). If we want to reconstruct the past, in contrast, we need to conditionalize over trajectories that also started in a low-entropy past state — that the “Past Hypothesis” that is required to get stat mech off the ground in a world governed by time-symmetric fundamental laws. https://www.preposterousuniverse.com/blog/2013/07/09/cosmology-and-the-past-hypothesis/ https://www.preposterousuniverse.com/blog/2013/07/09/cosmolo...
  - kgwgk 2y ago
    The second law of thermodynamics is about systems that are well described by a small set of macroscopic variables. The evolution of an initial macrostate prepared by an experimenter who can control only the macrovariables is reproducible. When a thermodynamical system is prepared in such a reproducible way the preparation is happening in the past, by definition. The second law is about how part of the information that we had about a system - constrained to be in a macrostate - is “lost” when we “forget” the previous state and describe it using just the current macrostate. We know more precisely the past than the future - the previous state is in the past by definition.
kgwgk 2y ago
> not very compatible with Shannon's usage in any but the shallowest sense The connection is not so shallow, there are entire books based on it. “The concept of information, intimately connected with that of probability, gives indeed insight on questions of statistical mechanics such as the meaning of irreversibility. This concept was introduced in statistical physics by Brillouin (1956) and Jaynes (1957) soon after its discovery by Shannon in 1948 (Shannon and Weaver, 1949). An immense literature has since then been published, ranging from research articles to textbooks. The variety of topics that belong to this field of science makes it impossible to give here a bibliography, and special searches are necessary for deepening the understanding of one or another aspect. For tutorial introductions, somewhat more detailed than the present one, see R. Balian (1991-92; 2004).” https://arxiv.org/pdf/cond-mat/0501322 https://arxiv.org/pdf/cond-mat/0501322
- petsfed 2y ago
  I don't dispute that the math is compatible. The problem is the interpretation thereof. When I say "shallowest", I mean the implications of each are very different. Insofar as I'm aware, there is no information-theoretic equivalent to the 2nd or 3rd laws of thermodynamics, so the intuition a student works up from physics about how and why entropy matters just doesn't transfer. Likewise, even if an information science student is well versed in the concept of configuration entropy, that's 15 minutes of one lecture in statistical thermodynamics. There's still the rest of the course to consider.

abetusk 2y ago

Also known as "the number of bits to describe a system". For example, 2^N equally probable states, N bits to describe each state.

Lichtso 2y ago

The "can be arranged" is the tricky part. E.g. you might know from context that some states are impossible (where the probability distribution is zero), even though they combinatorially exist. That changes the entropy to you. That is why information and entropy are different things. Entropy is what you know you do not know. That knowledge of the magnitude of the unknown is what is being quantified. Also, the point where I think the article is wrong (or not concise enough) as it would include the unknown unknowns, which are not entropy IMO: > I claim it’s the amount of information we don’t know about a situation

slashdave 2y ago
Exactly. If you want to reuse the term "entropy" in information theory, then fine. Just stop trying to make a physical analogy. It's not rigorous.

akira2501 2y ago

I spend time just staring at the graph on this page. https://en.wikipedia.org/wiki/Thermodynamic_beta https://en.wikipedia.org/wiki/Thermodynamic_beta