3 ms·

One of my favorite stat mech facts is that you can get to inverse temperature as a Lagrange multiplier using only probability and information theory: Given a s

by carlob 5y ago

One of my favorite stat mech facts is that you can get to inverse temperature as a Lagrange multiplier using only probability and information theory: Given a system that has many states each with energy E_i, what is the highest Shannon entropy distribution that has average energy E? If you define a Lagrange multiplier \beta to enforce the constraint \sum_i p_i E_i = E and maximize the entropy you get the Boltzmann distribution!

contravariant 5y ago
One minor nitpick is that in this derivation your choice of coordinates matters. Luckily (classical) physics provides a canonical set of coordinates which even comes with a measure that is preserved under the equations of motion. This magical property of the phase space can't be emphasized enough if you ask me.
- happytron 5y ago
  Can you expand on this? Is there a nicer coordinate-free way to define entropy?
  - contravariant 5y ago
    The definition of entropy is not coordinate independent. Very briefly the reason for this is that the uniform distribution isn't coordinate independent. There is a related coordinate independent way to measure distances from one distribution to another, which is the Kullback leibler divergence, and looks like: D_KL(P || Q) = \int dP/dQ log(dP/dQ) dQ = E_P[log(dP/dQ)] (This is by far the most general definition and uses the Radon-Nikodym derivative to be applicable to discrete, continuous and even weirder distributions. For reasons that I'll try to explain I also view this as a generalization of the notion of entropy) If you let Q be a uniform distribution this is equivalent to the entropy (up to some sign changes and a constant). However uniform distributions aren't coordinate independent and therefore the notion of entropy isn't. My personal conclusion is that there's no way of doing statistics without picking a(n improper) prior, since you inevitably need to pick some coordinates which ends up doing the same thing. You can then equivalently talk about the KL-divergence of a distribution from this prior or the entropy of a distribution, both end up being more or less the same thing.

by carlob 5y ago

contravariant 5y ago

One minor nitpick is that in this derivation your choice of coordinates matters. Luckily (classical) physics provides a canonical set of coordinates which even comes with a measure that is preserved under the equations of motion. This magical property of the phase space can't be emphasized enough if you ask me.

happytron 5y ago
Can you expand on this? Is there a nicer coordinate-free way to define entropy?
- contravariant 5y ago
  The definition of entropy is not coordinate independent. Very briefly the reason for this is that the uniform distribution isn't coordinate independent. There is a related coordinate independent way to measure distances from one distribution to another, which is the Kullback leibler divergence, and looks like: D_KL(P || Q) = \int dP/dQ log(dP/dQ) dQ = E_P[log(dP/dQ)] (This is by far the most general definition and uses the Radon-Nikodym derivative to be applicable to discrete, continuous and even weirder distributions. For reasons that I'll try to explain I also view this as a generalization of the notion of entropy) If you let Q be a uniform distribution this is equivalent to the entropy (up to some sign changes and a constant). However uniform distributions aren't coordinate independent and therefore the notion of entropy isn't. My personal conclusion is that there's no way of doing statistics without picking a(n improper) prior, since you inevitably need to pick some coordinates which ends up doing the same thing. You can then equivalently talk about the KL-divergence of a distribution from this prior or the entropy of a distribution, both end up being more or less the same thing.