5 ms·
If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a prob
by prof-dr-ir 2y ago
If I would write a book with that title then I would get to the point a bit faster, probably as follows.
Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is:
S = - sum_i p_i log( p_i )
Mathematically we say that entropy is a real-valued function on the space of probability distributions. (Elementary exercises: show that S >= 0 and it is maximized on the uniform distribution.)
That is it. I think there is little need for all the mystery.
- kgwgk 2y agoThat covers one and a half of the twelve points he discusses.
- prof-dr-ir 2y agoCorrect! And it took me just one paragraph, not the 18 pages of meandering (and I think confusing) text that it takes the author of the pdf to introduce the same idea.
- kgwgk 2y agoYou didn’t introduce any idea. You said it’s “just a number” and wrote down a formula without any explanation or justification. I concede that it was much shorter though. Well done!
- bdjsiqoocwk 2y agoHaha you reminded me of that idea in software engineering that "it's easy to make an algorithm faster if you accept that at times it might output the wrong result; in fact you can make infinitely fast"
- rachofsunshine 2y agoThe problem is that this doesn't get at many of the intuitive properties of entropy. A different explanation (based on macro- and micro-states) makes it intuitively obvious why entropy is non-decreasing with time or, with a little more depth, what entropy has to do with temperature.
- prof-dr-ir 2y agoThe above evidently only suffices as a definition, not as an entire course. My point was just that I don't think any other introduction beats this one, especially for a book with the given title. In particular it has always been my starting point whenever I introduce (the entropy of) macro- and micro-states in my statistical physics course.
- mjw_byrne 2y agoThat doesn't strike me as a problem. Definitions are often highly abstract and counterintuitive, with much study required to understand at an intuitive level what motivates them. Rigour and intuition are often competing concerns, and I think definitions should favour the former. The definition of compactness in topology, or indeed just the definition of a topological space, are examples of this - at face value, they're bizarre. You have to muck around a fair bit to understand why they cut so brilliantly to the heart of the thing.
- nabla9 2y agoEveryone who sees that formula can immediately see that it leads to principle of maximum entropy. Just like everyone seeing Maxwell's equations can immediately see that you can derive the the speed of light classically. Oh dear. The joy of explaining the little you know.
- prof-dr-ir 2y agoAs of this moment there are six other top-level comments which each try to define entropy, and frankly they are all wrong, circular, or incomplete. Clearly the very definition of entropy is confusing, and the definition is what my comment provides. I never said that all the other properties of entropy are now immediately visible. Instead I think it is the only universal starting point of any reasonable discussion or course on the subject. And lastly I am frankly getting discouraged by all the dismissive responses. So this will be my last comment for the day, and I will leave you in the careful hands of, say, the six other people who are obviously so extremely knowledgeable about this topic. /s
- mitthrowaway2 2y agoSo the only thing you need to know about entropy is that it's a real-valued number you can associate with a probability distribution? And that's it? I disagree. There are several numbers that can be associated with probability distribution, and entropy is an especially useful one, but to understand why entropy is useful, or why you'd use that function instead of a different one, you'd need to know a few more things than just what you've written here.
- Maxatar 2y agoExactly, saying that's all there is to know about entropy is like saying all you need to know about chess are the rules and all you need to know about programming is the syntax/semantics. Knowing the plain definition or the rules is nothing but a superficial understanding of the subject. Knowing how to use the rules to actually do something meaningful, having a strategy, that's where meaningful knowledge lies.
- FabHK 2y agoIn particular, the expectation (or variance) of a real-valued random variable can also be seen as "a real-valued number you can associate with a probability distribution". Thus, GP's statement is basically: "entropy is like expectation, but different".
- deleted 2y ago[deleted]
- prof-dr-ir 2y agoOf course that is not my statement. See all my other replies to identical misinterpretations of my comment.
- bdjsiqoocwk 2y ago[flagged]
- bubblyworld 2y agoPlease don't post comments just to be a dick.
- bdjsiqoocwk 2y ago[flagged]
- prof-dr-ir 2y agoYou are completely right of course. I am merely a professor in theoretical physics who has been teaching this stuff for a number of years now.
- bdjsiqoocwk 2y ago[flagged]
- deleted 2y ago[deleted]
- senderista 2y agoMany students will want to know where the minus sign comes from. I like to write the formula instead as S = sum_i p_i log( 1 / p_i ), where (1 / p_i) is the "surprise" (i.e., expected number of trials before first success) associated with a given outcome (or symbol), and we average it over all outcomes (i.e., weight it by the probability of the outcome). We take the log of the "surprise" because entropy is an extensive quantity, so we want it to be additive.
- mensetmanusman 2y agoDon’t forget it’s the only measure of the arrow of time.
- kgwgk 2y agoOne could also say that it’s just a consequence of the passage of time (as in getting away from a boundary condition). The decay of radioactive atoms is also a measure of the arrow of time - of course we can say that’s the same thing. CP violation may (or may not) be more relevant regarding the arrow of time.
- kaashif 2y agoThat definition is on page 18, I agree it could've been reached a bit faster but a lot of the preceding material is motivation, puzzles, and examples. This definition isn't the end goal, the physics things are.
- klysm 2y agoThe definition by itself without intuition of application is of little use
- bubblyworld 2y agoThanks for defining it rigorously. I think people are getting offended on John Baez's behalf because his book obviously covers a lot more - like why does this particular number seem to be so useful in so many different contexts? How could you have motivated it a priori? Etcetera, although I suspect you know all this already. But I think you're right that a clear focus on the maths is useful for dispelling misconceptions about entropy.
- kgwgk 2y agoMisconceptions about entropy are misconceptions about physics. You can’t dispell them focusing on the maths and ignoring the physics entirely - especially if you just write an equation without any conceptual discussion, not even mathematical.
- bubblyworld 2y agoI didn't say to only focus on the mathematics. Obviously wherever you apply the concept (and it's applied to much more than physics) there will be other sources of confusion. But just knowing that entropy is a property of a distribution, not a state, already helps clarify your thinking. For instance, you know that the question "what is the entropy of a broken egg?" is actually meaningless, because you haven't specified a distribution (or a set of micro/macro states in the stat mech formulation).
- kgwgk 2y agoOk, I don’t think we disagree. But knowing that entropy is a property of a distribution given by that equation is far from “being it” as a definition of the concept of entropy in physics. Anyway, it seems that - like many others - I just misunderstood the “little need for all the mystery” remark.
- bubblyworld 2y agoRight, I see what you're saying. I agree that there is a lot of subtlety in the way entropy is actually used in practice.