43 ms·

It took me 10 years to understand entropy

398 points by dil8 4y ago

javajosh 4y ago
Nice writeup! BTW statistical thermodynamics has a name for that set of possible microstates, perhaps the most pretentious sounding name in all of physics, the "canonical ensemble".
- jleyank 4y ago
  Nah, Ultraviolet Catastrophe is worse, then there's wavefunction collapse. And there's gotta be something in particle physics that puts these terms to shame. Given that the particle names are drawn from literature and whimsy. I looked up the dictionary definition of pretentious, and so it doesn't really apply, but the hERG channel is a critical ion channel and various drugs block this and cause Really Bad Things. hERG stands for "the human Ether-a-go-go-Related Gene" - a pretty bloody stupid name but whimsey is not restricted to particle physics. Canonical ensemble, along with microcanonical and grand canonical ensembles are all over statistical mechanics. And I suspect there's not one bit of whimsey in their naming. There was not any humour in my stat mech course aside from me trying to make sense of it.
  - javajosh 4y ago
    "Ultraviolet Catastrophe"'s problem is that it sounds way cooler than it is. I mean, that's the title of a cyberpunk novel; more hyperbolic (and disappointing) than pretentious. Eigenthings would be second on my list of pretentiousness (oddly, not gedankenthings; I'm inconsistent I guess). Standard Model names strike me whimsical to the point of being undignified.
    - ravi-delia 4y ago
      I'm still mad about top and bottom when truth and beauty were right there!
      - lvncelot 4y ago
        If I'm not mistaken, the "official" names are just "t" and "b", so top and bottom, just like truth and beauty, are more mnemonics. So referring to them as truth and beauty would be just as correct.
        jleyank 4y ago
        Thought one was their names and the other their attributes?
        lvncelot 4y ago
        Seems like you're right and I misremembered.
    - dr_dshiv 4y ago
      Ultraviolet catastrophe is cool! It’s why we needed quantum physics, otherwise there would be catastrophic infinite energy emerging from random high frequency oscillations!
      - javajosh 4y ago
        Well that's not quite what it is, first of all, and second of all I don't like the practice of calling something a catastrophe when its just predicted by a theory, but didn't happen. I mean, bad theories always predict something horrible should be happening. What's next, do flat earthers get to have an "ocean water catastrophe" because their theory implies all the water drains away? Or a "super luminal" catastrophe because their theory requires infinite unending linear acceleration of the earth (and moon and sun)? I mean, it really is a catastrophe I guess...for the theorist.
  - antognini 4y ago
    My favorite is "gravothermal catastrophe" with "violent relaxation" being a close runner-up.
  - XorNot 4y ago
    There's actually a sort of general principle in medical science that people should avoid whimsical names for things, since in all likelihood someone with a life-altering or fatal condition or their family shouldn't be told that it's due to a mutation in the Sonic Hedgehog protein [1] [1] https://en.wikipedia.org/wiki/Sonic_hedgehog https://en.wikipedia.org/wiki/Sonic_hedgehog
- effie 4y ago
  > the most pretentious sounding name in all of physics Also, "the free will theorem" and "the god particle".
  - hexxagone 4y ago
    the "god particle" was actually the "goddamn particle" https://www.businessinsider.com/why-the-higgs-is-called-the-god-particle-2015-5 https://www.businessinsider.com/why-the-higgs-is-called-the-...
- allisvancity 4y ago
  "well, I never!" —the Grand canonical ensemble
vivekd 4y ago
This reminds me of a great article that I saw on Hacker news that really helped explain the concept of Entrophy to me. Linked here: https://news.ycombinator.com/item?id=24140808 https://news.ycombinator.com/item?id=24140808
- ctxc 4y ago
  Gotta love these interactive websites.
deleted 4y ago
[deleted]
tamaharbor 4y ago
English was not my college professors native language. It took me a while to realize entropy was not the same as enthalpy. Very confusing.
- jleyank 4y ago
  Yeah, and while computational modeling has a good handle on enthalpy in drug design, it's all handwaving and statistics for entropy. And yet it's entropy that breaks predictions and drives medchem. Solve how to properly model the free energy of interaction between a ligand and a protein (or two proteins) with proper solvent treatment and (a) you'll be famous, not Kardashian famous, but famous and (b) a whole lot of people will buy or download your software.
- lambdatronics 4y ago
  Wait until you find out about enstrophy.
RappingBoomer 4y ago
but is it not hubris to think that we really know much about the origin and outcome of the universe? is it wise to make decisions based on this modicum of knowledge that we currently have regarding thermodynamics and the universe? I suspect that the scientists of a trillion years from now will know a lot more than we do know...so, I don't really much that much confidence in current pronouncements regarding the beginning and possible end of the universe.. and yes I do have a degree in science and courses in physics & thermodynamics
- lazide 4y ago
  Does it hurt to try?
  - RappingBoomer 4y ago
    it hurts to make decisions based on our current knowledge that is a child's knowledge
    - lazide 4y ago
      How exactly? And how exactly would we ever get past a ‘child’s knowledge’ without learning and making mistakes?
- mhh__ 4y ago
  The models we have work and are relatively parsimonious. It would be silly to assume they are final but equally silly to shy away from using them for obvious reasons. The modernization of physics also means we have outlines for what theories should look like, so even if our current theories are wrong we can still use the principles of (say) symmetry and information to constrain future work.
- tomcatfish 4y ago
  As the old quote runs... "[W]hen people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together." -- Asimov It isn't wise to say you're ignorant, it's wise to know how ignorant you are. If I see a coin come up HHH and have to bet on the next 2 flips, you can be damn sure I'll bet HH. You can bet HT, TH, or whatever else at equal probability, but I suspect I'll come out the winner more frequently than you.
l33t2328 4y ago
Like the great Von Neuman once quipped, “ Why don't you call it entropy”, von Neumann suggested. “In the first place, a mathematical development very much like yours already exists in Boltzmann's statistical mechanics, and in the second place, no one understands entropy very well, so in any discussion you will be in a position of advantage.”
SnowHill9902 4y ago
Statistical mechanics is one way of representing entropy but you don’t need it. The second law of thermodynamics can be expressed in other much more general terms. Also it requires that the system be isolated not “thermally isolated”. There’s other types of interactions such as gravitational and electromagnetic.
- ravi-delia 4y ago
  I mean, come on. You know and I know that the statistical mechanics definition gets you 99% of the way there in terms of intuition. Obviously if I spin a rotor in my thermally insulated box with a magnet on the outside I can add energy to order things with, I don't think anyone is confused on that point.
  - SnowHill9902 4y ago
    Well, claiming to having understood entropy is no joke. He should be flawless then. Just like with enlightenment he who claims to understand entropy really does not and he who does will not say so.
- chermi 4y ago
  Can you explain what you mean by "not needing stat mech" and thermo entropy being "more general"?
shannifin 4y ago
> "Entropy is not Disorder One of the most popular belief about entropy is that it represents disorder." This is what confused me the most about entropy in high school, the "order / disorder" lingo. Isn't "order" a metaphysical concept, something a conscious entity thinks about a system? How would nature know the difference? It took me some years to understand that that lingo is indeed misleading. (Still definitely not an expert of course.)
- hackernewds 4y ago
  What is a way to articulate it then?
  - andrewflnr 4y ago
    OP is about as good as any explanation I've seen.
    - XorNot 4y ago
      "Compressibility" (in the software sense) would perhaps carry the most meaning. Entropy is ultimately all about the ability to extract information (i.e. work) out of a system.
      - shannifin 4y ago
        Indeed, it was actually James Gleick's book The Information that helped me understand the concept better.
      - andrewflnr 4y ago
        "Compressibility" is really not that great an illustration for physical entropy. Information-theoretic entropy is not quite the same thing as physical entropy, but close enough to confuse you if you're not paying attention.
        XorNot 4y ago
raven105x 4y ago
Entropy: "to describe energy loss in irreversible processes". We have no clue about what is or is not reversible. Complex systems exhibit self-organizing behavior for no reason (that we understand), and we continue to identify more conditions under which this occurs. How does a Nobel Prize get handed out for identifying/quantifying "self-organization" http://pespmc1.vub.ac.be/COMPNATS.html http://pespmc1.vub.ac.be/COMPNATS.html without bringing everything we think we know about entropy under scrutiny? Self-organization does not consume energy any more than entropic decay emits it. Irreversibility is a poor assumption.
- birdyrooster 4y ago
  Welp, I suppose the author has another 10 years to work on their article about how it took them 20 years to actually understand entropy.
- deleted 4y ago
  [deleted]
- Retric 4y ago
  Entropy isn’t measuring a loss of energy, but the loss of the ability for a closed system to do useful work. Order is often used to describe what’s going on but it’s not the kind of order we normally think of. Sufficient cold water in a warm room is just as capable of preforming work as warm water in a cold room.
- ravi-delia 4y ago
  Complex systems are net increases in entropy. The water is flowing downhill, but it takes a really weird organism-shaped path to get there. Self organization is supposedly interesting because we don't know why such a path manifests. Thus far, nothing has given the second law a second's (ha ha) pause. It's not impossible, but considering most of our foundational physics is time-symmetric it makes sense to call entropy irreversible. Even if it could be reversed (and don't hold your breath on that one), it's still the cause of the arrow of time.
- XorNot 4y ago
  > Self-organization does not consume energy any more than entropic decay emits it. This statement is incredibly wrong - this is exactly what both these processes do. We calculate chemistry reaction kinetics by including entropy terms, and optimize reactions by manipulating the entropy on one side of the equation (a classic is getting a liquid phase to precipitate out as you produce it). I mean the reason coal can be turned into electricity is because there's a big increase in entropy going from "solid carbon in a specific location" to "CO2 diffused everywhere".
7speter 4y ago
Of all places, it was a conversation about the Socratic Forms in a Political Theory course I took in college that really brought the weight of the concept home to me. It went something like "Unlike the realm Socratic forms exist in, everything in our universe is subject to entropy; it is in everything's nature to degrade or decay over time." Maybe there's more to that? I'm all ears.
- dr_dshiv 4y ago
  The forms are a low energy state that materially arise from an entropic process. Degradation and decay can lead to more organization and beauty.
nfc 4y ago
I enjoyed the article but have a very minor nitpick. I didn't understand why the author added this sentence. "However, the timescales involved in these calculation are so unreasonably large and abstract that one could wonder if these makes any sense at all." Apart from the fact that we could wonder about anything and everything I think the author does not state what evidence do we have to suspect that large enough timescales would change the laws of physics. It could be the case of course, and it would be great to talk about them if they exist but without further justification I feel that this sentence is an unjustified opinion in what is otherwise a very nice article that helps better understand enthropy.
cheese_van 4y ago
I read with interest most well written articles explaining entropy. I often leave the article mildly satisfied that I understood it. Until the next day when I again have to figure out the difference between "high" and "low" entropy in a particular model, and invariably I mix up the two.
evouga 4y ago
One aspect of entropy that I always find counterintuitive is that unlike mass, charge, etc. it is not a physical quantity. In fact, from the point of view of an experimenter with perfect information about a physical system, the entropy of the system is exactly conserved over time (as made precise by Liouville's Theorem). The Second Law survives in this setting only in the most trivial sense that a constant function does not decrease. It's only when you start making crude measurements---lumping positions into pixels, clouds of particles each with their own kinetic energy into a single scalar called "temperature," etc---that you start to see a nontrivial entropy and Second Law. Different ways of lumping microstates into macrostates will give you different (and inconsistent) notions of entropy.
- jawarner 4y ago
  If I can pick your brain, there’s a related concept — entropy production. How does that relate with these ideas?
  - jbay808 4y ago
    Not the GP, but entropy production is any process that increases your ignorance about a system (and usually, if you're doing it intentionally, everybody else's ignorance as well). To produce entropy you have to grow the number of possible microstates that are consistent with available knowledge of the macrostate. Usually you accomplish that by converting stored energy into heat somehow. A charged capacitor has low entropy compared for the energy it holds; discharge it through a resistor, and you produce a bunch of entropy because that energy can now be distributed in a lot more ways among a lot more degrees of freedom, and nobody can possibly keep track of them.
- Hermel 4y ago
  Not really. Even from a point of view of an experimenter with perfect information, the entropy of the system declines over time as fewer and fewer bits are needed to describe the system. For example, start with a Glas of warm water and an ice cube in it. Over time, the ice will melt and the range of different temperatures of the molecules decline. Consequently, you need fewer and fewer bits to describe the complete state of the system. It takes fewer Bits to encode all the velocities of a million molecules that all move at a similar speed than to encode all the velocities of a million molecules that move at very different speeds. The more similar the state of the molecules becomes, the shorter a text becomes that has to describe the complete state of a system. Therefore, entropy is decreasing even from the point of view of an observer with perfect information.
leereeves 4y ago
Small nit in case the author sees this: the image labelled "Entropy of each configuration of system with two dices where the observed macrostate is their sum" is either incorrect or mislabeled. For example, 2 and 12 each have 1 microstate, and ln 1 = 0, so the entropy of 2 and 12 is 0, but the image says 0.028 (which is the probability of 2 or 12, not the entropy).
sjg007 4y ago
I view entropy as a probability distribution of some set of configurations of something. Entropy is low if there’s only one configuration and high if uniformly distributed. There’s also some observer/interaction effect which is like introduction a conditional probability which would cause crystallization in an otherwise homogeneous system. Essentially a catalyst. I also find it fascinating that when it is super cold outside and you throw a pan of boiling water out the window it turns to snow instantly vs a cup of room temperature water which does not. It probably fits in terms of activation energy as well.
hatware 4y ago
Google freewall? Guess I won't read this article...
- jwilk 4y ago
  What do you mean?
  - hatware 4y ago
    It is required that I log-in to the article using my google or facebook account to read it. Garbage. Your article isn't worth reading if I need to jump through hoops to get there.
    - jwilk 4y ago
      Here's an archived copy: https://archive.today/vRD7i https://archive.today/vRD7i
- JKCalhoun 4y ago
  Seeing "Read the rest of this story with a free account." Nope.
topspin 4y ago
That's pretty good as far as I'm concerned. Took me a couple years to really grasp electrical impedance. Breakthrough for me was a concise book written in 1976 by Rufus P. Turner. Subtle things take a while to get.
chemmail 4y ago
I had an art teacher who was very philosophical. One day he described to the class what entropy was. I took a lot of physics and even astrophysics. Little did i know he had a better conceptual understanding and explanation than i've ever heard before. Too bad i don't remember exactly what he said.
- Koshkin 4y ago
  In art, a high-entropy painting is one that would be hard to tell apart from similar paintings, one example being paintings created by simply splattering paints all over the canvas.
- l33t2328 4y ago
  Not to poke holes in your nostalgia, but how do you know he had a great explanation if you don’t remember it after further study?
  - mhh__ 4y ago
    "Don't try to understand it, feel it" - from tenet, but does sort of apply to entropy as a way of looking at problems. That being said I heard a sports science student try to recall their working definition of entry and it was some mess of locks and keys floating around randomly hitting eachother?
    - denton-scratch 4y ago
      > definition of entry and it was some mess of locks and keys floating around randomly hitting eachother? Locks and keys does indeed sound like "entry". I thought we were discussing "entropy".
  - deleted 4y ago
    [deleted]
hilbert42 4y ago
Of all of physics, entropy is the most depressing part.
stevebmark 4y ago
The author mentions Boltzman brains and that a human body could theoretically spontaneously form out of particles given a long enough time span. Of course, nothing like this can ever happen. It’s the fallacy of thinking infinite time means infinite possibilities.
- nomel 4y ago
  > that a human body could theoretically spontaneously form out of particles given a long enough time span. To be fair, isn't this precisely what happened?
  - dilawar 4y ago
    If you leave out "spontaneously"!
    - _carbyau_ 4y ago
      Specifically if you leave it in! It's a matter of viewpoint or scope. "Spontaneous" can be defined a few different ways: https://www.merriam-webster.com/dictionary/spontaneous https://www.merriam-webster.com/dictionary/spontaneous From the link "2: arising from a momentary impulse" you get the implied meaning from the original comment if you assume standard human experience of a "moment" IE a few seconds or less. However you could argue that human evolution is but a moment in the scope of the universe. And then with definition "5: developing or occurring without apparent external influence, force, cause, or treatment" The only way it wouldn't be spontaneous would be if an external actor (Deity of your choice?) directed human evolution somehow. To say this is debatable is an understatement.. And so we have fun in word play and hopefully appreciation of different viewpoints. :-) edit: rearranged for better flow
  - dj_mc_merlin 4y ago
    "Oh, that was easy," says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing."
- ulamia 4y ago
photochemsyn 4y ago
Here's another head-spinning application of the concept of entropy, in quantum information theory: https://www.cambridge.org/core/books/abs/quantum-information-theory/quantum-information-and-entropy/5BCF610B966AFB2468D0F0F46249DCD0 https://www.cambridge.org/core/books/abs/quantum-information... > "The first fundamental measure that we introduce is the von Neumman entropy. It is the quantum analog of the Shannon entropy, but it captures both classical and quantum uncertainty in a quantum state. The von Neumann entropy gives meaning to a notion of the information qubit. This notion is different from that of the physical qubit, which is the description of a quantum state in an electron or a photon. The information qubit is the fundamental quantum informational unit of measure, determining how much quantum information is in a quantum system." Incidentally chem.libretexts.org, a collection of open-source chemistry textbooks, has a good overview of the physical-chemical applications. The site is kind of a mess but you'd want chapter 18.3: https://chem.libretexts.org/Bookshelves/General_Chemistry/Map:_A_Molecular_Approach_(Tro) https://chem.libretexts.org/Bookshelves/General_Chemistry/Ma...
ogogmad 4y ago
I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). It's worth considering the similarities and differences between entropy and standard deviation. I thought the 2nd law of thermodynamics was saying that with incomplete knowledge, the probability distribution of possible states becomes more and more spread out as time goes on. It's almost a limit to how you can make predictions or simulations of physics when the initial state of the system is not fully known. Equivalently, it's a banal statement about chaos in the sense of chaos theory. The only thing I don't get is how physicists get around the discrete and finite restriction. Maybe the state of the system is not what has entropy. Rather, one can define an arbitrary function f from the system to a finite set S, and then talk about the entropy of f(System at time t), because this is indeed a discrete and finite probability distribution which you can take the entropy of. Hmmm. Maybe I understand entropy.
- hanche 4y ago
  In Shannon’s 1948 paper, part V deals with continuous sources. The key is to realise that you cannot measure a continuous signal exactly, and so you can define a rate of information relative to the fidelity of your measurement. (I only skimmed that part years ago, and never studied it carefully. But it makes perfect sense.)
  - ogogmad 4y ago
    If you mean differential entropy (which Shannon supposedly suggested as a generalisation to continuous random variables), this is not a good generalisation of entropy to continuous random variables. It lacks all the interesting properties of entropy. The "proper" generalisation of entropy to continuous random variables is something called relative entropy, or in some books it's called KL divergence. But this is now a property of how two probability distributions relate to each other, rather than a property of a single probability distribution alone. I'm not an expert in probability theory or physics, but this is what I've learnt from a brief study of these areas.
threatripper 4y ago
I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equally likely. If you randomly flip bits you go from one state to another state but each one is equally likely to occur. There is no special meaning to a specific configuration if you don't give it one. If you look at the average of all bits you start grouping all states together with the equal number of 1s. If you talk about the average there is only one configuration that is all 1s but most configurations have roughly 50% 1s. If you now start flipping bits you will meander through all possible bit-states but the average will most likely be close to 50% 1s most of the time. In physics we usually look at averages such as the average velocity expressed as temperature. Therefore it makes sense to group together all states using the average and then the states with very low or very high averages are few. But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I don't know what Entropy is supposed to mean on the level of individual states/configurations. I don't understand what kind of macroscopic "averaging" function we may use to group up those states. There could be more than one possibility - from that would follow that there is more than one definition of macro-Entropy. Ideally there should be one general definition of how we have to look at those microstates and from that follows our general definition of Entropy. Sadly I didn't study Physics and this topic still continues to confuse me. The usual explanations fail to enlighten me.
- oakwhiz 4y ago
  >There could be more than one possibility This seems to connect with the idea behind Chaitin's incompleteness theorem. Making specific statements about the reducible complexity of something is not always possible.
- pfortuny 4y ago
  It is also because “the system” is intrinsic to the notion: as you say, any configuration of bits is equally likely; this only takes into account the “system of the bits “. The moment your system is “the bits and their mean value” everything changes, as there are systems with a single possible configuration. That is what happens when he starts the first example: “a system of particles INSIDE A VOLUME. The volume is what makes the entropy larger or smaller. The particles in a different volume (or just by themselves) have a different entropy.
fysicsnurd 4y ago
entropy does not increase. The universe has organized itself into people, brains, cities, iPhones
dieselerator 4y ago
That's the problem with randomness. The required Dilbert reference: https://dilbert.com/strip/2001-10-25 https://dilbert.com/strip/2001-10-25
complex_pi 4y ago
In short: the authors make a good summary of these ideas: - Entropy in thermodynamic equilibrium is well understood. The early theory (before statistical mechanics was developed) fits well with our modern understanding. - The analogies made about entropy are not always good and indeed, if you try to match the physics with "entropy is disorder" it does not always work. - In non-equilibrium situations it is, as the author points out, more complex. Regarding the last item, even Stephen Hawking postulated some strange ideas about the universe having to rewind past some point in time, so that the big crush would be the mirror of the big bang.
sytelus 4y ago
Anyone who think they understand entropy is living in a state of sin.
- kgwgk 4y ago
  For those missing the reference, the original quote is also great: "Anyone who attempts to generate random numbers by deterministic means is, of course, living in a state of sin." John von Neumann
deltaonefour 4y ago
>Contrary to popular opinion, uniformly distributed matter is unstable when interactions are dominated by gravity (Jeans instability) and is actually the least likely state, thus with very low entropy. Most probable states, with high-entropy, are those where matter is all lumped together in massive objects. That means over time the system becomes more ordered and starts organizing itself into spheres. I once brought this question up on physics stack exchange and basically the answers were either some form of rolling their eyes at me or dismissing me outright. The people who did answer the question stated that as particles organize themselves into spheres some other part of the universe gets hotter as a result and that the seemingly self organization I see going on with the solar system was just an isolated system. This answer still seemed far fetched to me. It still looks as if some overall self organization is still going on if the universe gets hotter on one side and matter gets organized into solar systems on another side. It took me 3 years to somewhat understand what entropy is. If you have loaded dice that always roll 6s then the dice rolling ALL 6s is the highest entropic state. rolling Random numbers would then be a low entropy state. Entropy is simply a phenomenon of probability. As time moves forward, particles enter high probability configurations. Like rolling dice. As you roll dice more and more... rolling random numbers has a higher probability then rolling all 6s... It just so happens that disordered arrangements happen to have higher probabilities in most systems. But if you look at a system of loaded dice or the solar system... in those cases Ordered configurations have higher probabilities. That's really all it is. The entire phenomenon of entropy comes down to probability and the root of probability is the law of large numbers.
empiricus 4y ago
My understanding of entropy: it is a measure of how big a system (matter + energy from a space region) is, and how much its components have interacted with each other: Entropy ~ log(number of possible system states). As the universe unfolds, systems originally isolated are starting to interact and to form bigger systems, hence the number of possible states increases, and entropy increases too.
- Koshkin 4y ago
  Entropy implies that these states are indistinguishable from each other.
Gravityloss 4y ago
Gas molecules in a box - entropy seems quite straighforward there. An even distribution is the most likely state and has the highest entropy. In space, at large scales, gravity starts dominating - so stars and planets are actually a higher likelihood state than an even distribtion. Isn't this just about statistical independence? In a small amount of gas (almost by definition of what is a gas), the particles don't have much effect on each other. One can assume statistical independence. While in space with gravity overwhelming other effects, the particles have very much effect on each other. Hence the statistics about their state are affected by these dependencies. So the previous intuition about entropy can't hold.
pkrumins 4y ago
I think the word entropy is science’s largest mistake.
- keithalewis 4y ago
  Shannon explained the name 'entropy' in (McIrvine and Tribus 1971): My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'
  - pkrumins 4y ago
    Very cool! I didn’t know that.
m0llusk 4y ago
Alternatively it took ten years for Aurelien Pelissier's misunderstandings of entropy to decay.
mellavora 4y ago
The typical measure of entropy (Shannon or Gibbs, and let's spare details for later and after you've read up on the theory of large deviations) is - sum (p log(p)) which is not that different than the formula for the mean sum (p 1/n) the critical difference is the normalization constant is based on the probability of the state rather than assuming a uniform probability over all states. So, in effect, the entropy is a measure of the mean. It is a measure adopted to the case where "mean" is ill-defined because the number of modes and/or the variation around those modes is not handled well by simpler metrics.
- prof-dr-ir 4y ago
  If there was anyone who taught you this then they should be fired. More constructively, principal among the many things wrong with your comment is the formula for the mean; sum_i p_i = 1, so sum_i p_i / n = 1 / n. The mean would instead be sum_i p_i x_i.
- im3w1l 4y ago
  It can be related to compression. If some phrase has a probability p_i of occuring, then the optimal length for the code is -log(p_i). The entropy sum(-p_i log_pi) = mean(-log(p_i)) is how long code you will use on average.
- gjm11 4y ago
  Perhaps I'm misunderstanding or missing something, but I'm afraid this seems completely wrongheaded to me. (My apologies for being so blunt, but right now your comment appears to be the most-upvoted, and I therefore think it needs some pushback.) [EDITED to add: I was looking at an old version of the page; by the time I wrote this the parent was no longer the top comment. I'll leave the bluntness in, especially as at least one other person was even blunter.] You refer to "the mean" and I think you mean the mean of the probabilities. Now, when you've got a probability distribution, by far the usual thing for "the mean" to mean is the sum of Pr(x) x -- the mean of the values. Taking the mean of the probabilities is a really strange thing to do. One reason why it's a really strange thing to do is that this thing you call n is really kinda meaningless. There's no difference between these two probability distributions: (a) 1, 2, 3, or 4, with probabilities 0.1, 0.2, 0.3, 0.4 respectively; (b) 1, 2, 3, 4, or 5, with probabilities 0.1, 0.2, 0.3, 0.4, 0 respectively. But (a) has n=4 and (b) has n=5. Maybe you want n to be the number of nonzero probabilities? But now consider (a) along with the following probability distribution parameterized by a (small, positive) number h: 1, 2, 3, 4, or 4+h, with probabilities 0.1, 0.2, 0.3, 0.4-h, h. Every version of this distribution with h>0 has n=5, but when h is very small it's practically indistinguishable from (a) with n=4. Further, since the sum of probabilities is always 1, what you write as sum (p 1/n) is just the same as the number 1/n. You can call it "the mean" if you want to, but I don't see what this adds over calling it what it is: the reciprocal of the number of possibilities. There is something to what you say: the entropy is kinda related to the number of possibilities; if the probabilities are all equal, the entropy is log(#possibilities); if the probabilities are equal-ish then it's modestly smaller than that. But note e.g. that this relationship is exactly the inverse of what you say, in that "the mean" decreases with the number of possibilities, and the entropy increases with the number of possibilities. The entropy is not "a measure of the mean". It kinda-sorta is related to "the number of possibilities", which is the reciprocal of "the mean". It is not at all the case, as your last paragraph suggests, that for most purposes we should be using "the mean" but we need to use the entropy when "the number of modes ... is not handled well by simpler metrics", whatever that means; for most purposes we should be using the entropy, and in the special case where all the probabilities are equal we can get away with just counting possibilities. (In some important situations it turns out that what you have is some number of possibilities with roughly equal probabilities, and a whole lot more whose probabilities rapidly decrease to almost zero, and then you can get away with counting the number of reasonably-probable possibilities and taking its log. E.g., various situations in communications theory can fruitfully be thought of this way. But the entropy is still the more fundamental quantity, and "the mean" is still a needless obfuscation of "the (effectively) number of possibilities".)
mensetmanusman 4y ago
I had the pain, and pleasure, of taking and then (assistant) teaching thermodynamics at MIT. One of the tidbits that always stuck with me was that astronomers have estimated that observable universe’s total entropy: When you compare that value to the maximum possible entropy, i.e. the heat death of the universe, and then to the ridiculously low entropy state of the beginning of the universe, we are currently halfway along in that ‘timeline’. It always brought to mind a grandfather clock; the clock stops when the weight hits the floor, and we are halfway there…
andrewgleave 4y ago
The Science of Can and Can't[1] is interesting in how it looks to address a number of fundamentals via counterfactuals including the 2nd Law of Thermodynamics. Edit: See [2] for background about Constructor Theory. [1] https://www.chiaramarletto.com/books/the-science-of-can-and-cant/ https://www.chiaramarletto.com/books/the-science-of-can-and-... [2] https://www.youtube.com/watch?v=8DH2xwIYuT0 https://www.youtube.com/watch?v=8DH2xwIYuT0
snikeris 4y ago
> Boltzmann imagined that our universe could have reached thermodynamical equilibrium and its maximal entropy state a long time ago, but that a spontaneous entropy decrease to the level of our early universe occured after an extremely long period of time, just for statistical reasons. I’m interested in reading more about this. Any pointers?
poulpy123 4y ago
not to brag but it took only 2 months to forget anything about it
sylware 4y ago
scientific method: it started with thermodynamic entropy, but scientits found out that this truth is much deeper engrained in our universe, then we got a mathematically generalized version, which is now used used to explain the "arrow of time" which our time reversable physics equations would not be able to explain alone.
dandanua 4y ago
The problem is that entropy is a subjective notion. It's a measure of our lack of knowledge about the state of a system.
AnthonyAguirre 4y ago
Actually a quite nice article. After also spending years as a professional physicist not understanding entropy, I finally decided that I was not necessarily the problem, and spent the last 5 years or so trying to understand it better by rewording the foundations with my research group. (I'm one of the papers the author cites is part of a series from our group developing "observational entropy" in order to do so.) A lot of what makes this topic confusing is just that there are the two basic definitions — Gibbs (\sum p_i log l_i) and "Boltzmann" (log \Omega) — entropy, and they're really rather different. There's usually some confusing handwaving about how to relate them, but the fact is that in a closed system one of them (generally) rises and the other doesn't, and one of them depends on a coarse-graining into macrostates and the other doesn't. The better way to relate them, I've come to believe, is to consider them both as limits of a more general entropy (the one we developed — first in fact written down in some form by von Neumann but for some reason not pursued much over the years.) There's a brief version here: https://link.springer.com/article/10.1007/s10701-021-00498-x https://link.springer.com/article/10.1007/s10701-021-00498-x. This entropy has Gibbs and Bolztmann entropy as limits, is good in and out of equilibrium, is defined in quantum theory and with a very nice classical-quantum correspondence, and has been shown to reproduce thermodynamic entropy in both our papers and the elegant one by Strasberg and Winter: https://journals.aps.org/prxquantum/abstract/10.1103/PRXQuantum.2.030202 https://journals.aps.org/prxquantum/abstract/10.1103/PRXQuan... After all this work I finally feel that entropy makes sense to me, which it never quite did before — so I hope this is helpful to others. p.s. If you're not convinced a new definition of entropy is called for, ask a set of working physicists what it would mean to say "the entropy of the universe is increasing." Since von Neumann entropy is conserved in a closed system (which the universe is if anything is), and there really is no definition of a quantum Boltzmann entropy (until observational entropy), the answers you'll get will be either a mush or a properly furrowed brows.
- stazz1 4y ago
  The universe is not a closed system
  - AnthonyAguirre 4y ago
    How do you define "the universe"?
raxxorraxor 4y ago
As a computer scientist it isn't helpful that the entropy in thermodynamics and that in computer science (informational content - I don't know a good English term) collide a bit.
Buttons840 4y ago
I recommend Information Theory for Intelligent People: http://tuvalu.santafe.edu/~simon/it.pdf http://tuvalu.santafe.edu/~simon/it.pdf
- Buttons840 4y ago
  Let me add: This PDF is only 13 pages and presents a few different ways of viewing entropy. It was easy for me to follow as an undergrad.
cb321 4y ago
Of possible related interest: https://arxiv.org/abs/chao-dyn/9603009 https://arxiv.org/abs/chao-dyn/9603009 I think Bricmont is a clear thinker/presenter on these matters and this article actually showed up in a "for humanities people" anthology. [1] [1] https://www.amazon.com/Flight-Science-Reason-Academy-Sciences/dp/0801856760 https://www.amazon.com/Flight-Science-Reason-Academy-Science...
zwieback 4y ago
Of course you don't need to really understand entropy for it to be useful. It's definitely an interesting concept but when I was crunching equations for Thermodynamics, one of the weeder classes for ME, it becomes clear you need it for things to balance out. Once you've cranked threw a dozen or so problems you get a feel for what it is even if the physics and the spiritual side of it remains murky. Now, 35 years later, when I marvel at my new engine or what have you, I still vaguely remember my entropy-problems days and appreciate that someone worked this stuff out.
est 4y ago
Shannon called the function "entropy" and used it as a measure of "uncertainty," interchanging the two words in his writings without discrimination

398 points by dil8 4y ago

Nice writeup! BTW statistical thermodynamics has a name for that set of possible microstates, perhaps the most pretentious sounding name in all of physics, the "canonical ensemble".

jleyank 4y ago
Nah, Ultraviolet Catastrophe is worse, then there's wavefunction collapse. And there's gotta be something in particle physics that puts these terms to shame. Given that the particle names are drawn from literature and whimsy. I looked up the dictionary definition of pretentious, and so it doesn't really apply, but the hERG channel is a critical ion channel and various drugs block this and cause Really Bad Things. hERG stands for "the human Ether-a-go-go-Related Gene" - a pretty bloody stupid name but whimsey is not restricted to particle physics. Canonical ensemble, along with microcanonical and grand canonical ensembles are all over statistical mechanics. And I suspect there's not one bit of whimsey in their naming. There was not any humour in my stat mech course aside from me trying to make sense of it.
- javajosh 4y ago
  "Ultraviolet Catastrophe"'s problem is that it sounds way cooler than it is. I mean, that's the title of a cyberpunk novel; more hyperbolic (and disappointing) than pretentious. Eigenthings would be second on my list of pretentiousness (oddly, not gedankenthings; I'm inconsistent I guess). Standard Model names strike me whimsical to the point of being undignified.
  - ravi-delia 4y ago
    I'm still mad about top and bottom when truth and beauty were right there!
    - lvncelot 4y ago
      If I'm not mistaken, the "official" names are just "t" and "b", so top and bottom, just like truth and beauty, are more mnemonics. So referring to them as truth and beauty would be just as correct.
      - jleyank 4y ago
        Thought one was their names and the other their attributes?
        lvncelot 4y ago
        Seems like you're right and I misremembered.
  - dr_dshiv 4y ago
    Ultraviolet catastrophe is cool! It’s why we needed quantum physics, otherwise there would be catastrophic infinite energy emerging from random high frequency oscillations!
    - javajosh 4y ago
      Well that's not quite what it is, first of all, and second of all I don't like the practice of calling something a catastrophe when its just predicted by a theory, but didn't happen. I mean, bad theories always predict something horrible should be happening. What's next, do flat earthers get to have an "ocean water catastrophe" because their theory implies all the water drains away? Or a "super luminal" catastrophe because their theory requires infinite unending linear acceleration of the earth (and moon and sun)? I mean, it really is a catastrophe I guess...for the theorist.
- antognini 4y ago
  My favorite is "gravothermal catastrophe" with "violent relaxation" being a close runner-up.
- XorNot 4y ago
  There's actually a sort of general principle in medical science that people should avoid whimsical names for things, since in all likelihood someone with a life-altering or fatal condition or their family shouldn't be told that it's due to a mutation in the Sonic Hedgehog protein [1] [1] https://en.wikipedia.org/wiki/Sonic_hedgehog https://en.wikipedia.org/wiki/Sonic_hedgehog
effie 4y ago
> the most pretentious sounding name in all of physics Also, "the free will theorem" and "the god particle".
- hexxagone 4y ago
  the "god particle" was actually the "goddamn particle" https://www.businessinsider.com/why-the-higgs-is-called-the-god-particle-2015-5 https://www.businessinsider.com/why-the-higgs-is-called-the-...
allisvancity 4y ago
"well, I never!" —the Grand canonical ensemble

vivekd 4y ago

This reminds me of a great article that I saw on Hacker news that really helped explain the concept of Entrophy to me. Linked here: https://news.ycombinator.com/item?id=24140808 https://news.ycombinator.com/item?id=24140808

ctxc 4y ago
Gotta love these interactive websites.

deleted 4y ago

[deleted]

tamaharbor 4y ago

English was not my college professors native language. It took me a while to realize entropy was not the same as enthalpy. Very confusing.

jleyank 4y ago
Yeah, and while computational modeling has a good handle on enthalpy in drug design, it's all handwaving and statistics for entropy. And yet it's entropy that breaks predictions and drives medchem. Solve how to properly model the free energy of interaction between a ligand and a protein (or two proteins) with proper solvent treatment and (a) you'll be famous, not Kardashian famous, but famous and (b) a whole lot of people will buy or download your software.
lambdatronics 4y ago
Wait until you find out about enstrophy.

RappingBoomer 4y ago

but is it not hubris to think that we really know much about the origin and outcome of the universe? is it wise to make decisions based on this modicum of knowledge that we currently have regarding thermodynamics and the universe? I suspect that the scientists of a trillion years from now will know a lot more than we do know...so, I don't really much that much confidence in current pronouncements regarding the beginning and possible end of the universe.. and yes I do have a degree in science and courses in physics & thermodynamics

lazide 4y ago
Does it hurt to try?
- RappingBoomer 4y ago
  it hurts to make decisions based on our current knowledge that is a child's knowledge
  - lazide 4y ago
    How exactly? And how exactly would we ever get past a ‘child’s knowledge’ without learning and making mistakes?
mhh__ 4y ago
The models we have work and are relatively parsimonious. It would be silly to assume they are final but equally silly to shy away from using them for obvious reasons. The modernization of physics also means we have outlines for what theories should look like, so even if our current theories are wrong we can still use the principles of (say) symmetry and information to constrain future work.
tomcatfish 4y ago
As the old quote runs... "[W]hen people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together." -- Asimov It isn't wise to say you're ignorant, it's wise to know how ignorant you are. If I see a coin come up HHH and have to bet on the next 2 flips, you can be damn sure I'll bet HH. You can bet HT, TH, or whatever else at equal probability, but I suspect I'll come out the winner more frequently than you.

l33t2328 4y ago

Like the great Von Neuman once quipped, “ Why don't you call it entropy”, von Neumann suggested. “In the first place, a mathematical development very much like yours already exists in Boltzmann's statistical mechanics, and in the second place, no one understands entropy very well, so in any discussion you will be in a position of advantage.”

SnowHill9902 4y ago

Statistical mechanics is one way of representing entropy but you don’t need it. The second law of thermodynamics can be expressed in other much more general terms. Also it requires that the system be isolated not “thermally isolated”. There’s other types of interactions such as gravitational and electromagnetic.

ravi-delia 4y ago
I mean, come on. You know and I know that the statistical mechanics definition gets you 99% of the way there in terms of intuition. Obviously if I spin a rotor in my thermally insulated box with a magnet on the outside I can add energy to order things with, I don't think anyone is confused on that point.
- SnowHill9902 4y ago
  Well, claiming to having understood entropy is no joke. He should be flawless then. Just like with enlightenment he who claims to understand entropy really does not and he who does will not say so.
chermi 4y ago
Can you explain what you mean by "not needing stat mech" and thermo entropy being "more general"?

shannifin 4y ago

> "Entropy is not Disorder One of the most popular belief about entropy is that it represents disorder." This is what confused me the most about entropy in high school, the "order / disorder" lingo. Isn't "order" a metaphysical concept, something a conscious entity thinks about a system? How would nature know the difference? It took me some years to understand that that lingo is indeed misleading. (Still definitely not an expert of course.)

hackernewds 4y ago
What is a way to articulate it then?
- andrewflnr 4y ago
  OP is about as good as any explanation I've seen.
  - XorNot 4y ago
    "Compressibility" (in the software sense) would perhaps carry the most meaning. Entropy is ultimately all about the ability to extract information (i.e. work) out of a system.
    - shannifin 4y ago
      Indeed, it was actually James Gleick's book The Information that helped me understand the concept better.
    - andrewflnr 4y ago
      "Compressibility" is really not that great an illustration for physical entropy. Information-theoretic entropy is not quite the same thing as physical entropy, but close enough to confuse you if you're not paying attention.
      - XorNot 4y ago

raven105x 4y ago

Entropy: "to describe energy loss in irreversible processes". We have no clue about what is or is not reversible. Complex systems exhibit self-organizing behavior for no reason (that we understand), and we continue to identify more conditions under which this occurs. How does a Nobel Prize get handed out for identifying/quantifying "self-organization" http://pespmc1.vub.ac.be/COMPNATS.html http://pespmc1.vub.ac.be/COMPNATS.html without bringing everything we think we know about entropy under scrutiny? Self-organization does not consume energy any more than entropic decay emits it. Irreversibility is a poor assumption.

birdyrooster 4y ago
Welp, I suppose the author has another 10 years to work on their article about how it took them 20 years to actually understand entropy.
deleted 4y ago
[deleted]
Retric 4y ago
Entropy isn’t measuring a loss of energy, but the loss of the ability for a closed system to do useful work. Order is often used to describe what’s going on but it’s not the kind of order we normally think of. Sufficient cold water in a warm room is just as capable of preforming work as warm water in a cold room.
ravi-delia 4y ago
Complex systems are net increases in entropy. The water is flowing downhill, but it takes a really weird organism-shaped path to get there. Self organization is supposedly interesting because we don't know why such a path manifests. Thus far, nothing has given the second law a second's (ha ha) pause. It's not impossible, but considering most of our foundational physics is time-symmetric it makes sense to call entropy irreversible. Even if it could be reversed (and don't hold your breath on that one), it's still the cause of the arrow of time.
XorNot 4y ago
> Self-organization does not consume energy any more than entropic decay emits it. This statement is incredibly wrong - this is exactly what both these processes do. We calculate chemistry reaction kinetics by including entropy terms, and optimize reactions by manipulating the entropy on one side of the equation (a classic is getting a liquid phase to precipitate out as you produce it). I mean the reason coal can be turned into electricity is because there's a big increase in entropy going from "solid carbon in a specific location" to "CO2 diffused everywhere".

7speter 4y ago

Of all places, it was a conversation about the Socratic Forms in a Political Theory course I took in college that really brought the weight of the concept home to me. It went something like "Unlike the realm Socratic forms exist in, everything in our universe is subject to entropy; it is in everything's nature to degrade or decay over time." Maybe there's more to that? I'm all ears.

dr_dshiv 4y ago
The forms are a low energy state that materially arise from an entropic process. Degradation and decay can lead to more organization and beauty.

nfc 4y ago

I enjoyed the article but have a very minor nitpick. I didn't understand why the author added this sentence. "However, the timescales involved in these calculation are so unreasonably large and abstract that one could wonder if these makes any sense at all." Apart from the fact that we could wonder about anything and everything I think the author does not state what evidence do we have to suspect that large enough timescales would change the laws of physics. It could be the case of course, and it would be great to talk about them if they exist but without further justification I feel that this sentence is an unjustified opinion in what is otherwise a very nice article that helps better understand enthropy.

cheese_van 4y ago

I read with interest most well written articles explaining entropy. I often leave the article mildly satisfied that I understood it. Until the next day when I again have to figure out the difference between "high" and "low" entropy in a particular model, and invariably I mix up the two.

evouga 4y ago

One aspect of entropy that I always find counterintuitive is that unlike mass, charge, etc. it is not a physical quantity. In fact, from the point of view of an experimenter with perfect information about a physical system, the entropy of the system is exactly conserved over time (as made precise by Liouville's Theorem). The Second Law survives in this setting only in the most trivial sense that a constant function does not decrease. It's only when you start making crude measurements---lumping positions into pixels, clouds of particles each with their own kinetic energy into a single scalar called "temperature," etc---that you start to see a nontrivial entropy and Second Law. Different ways of lumping microstates into macrostates will give you different (and inconsistent) notions of entropy.

jawarner 4y ago
If I can pick your brain, there’s a related concept — entropy production. How does that relate with these ideas?
- jbay808 4y ago
  Not the GP, but entropy production is any process that increases your ignorance about a system (and usually, if you're doing it intentionally, everybody else's ignorance as well). To produce entropy you have to grow the number of possible microstates that are consistent with available knowledge of the macrostate. Usually you accomplish that by converting stored energy into heat somehow. A charged capacitor has low entropy compared for the energy it holds; discharge it through a resistor, and you produce a bunch of entropy because that energy can now be distributed in a lot more ways among a lot more degrees of freedom, and nobody can possibly keep track of them.
Hermel 4y ago
Not really. Even from a point of view of an experimenter with perfect information, the entropy of the system declines over time as fewer and fewer bits are needed to describe the system. For example, start with a Glas of warm water and an ice cube in it. Over time, the ice will melt and the range of different temperatures of the molecules decline. Consequently, you need fewer and fewer bits to describe the complete state of the system. It takes fewer Bits to encode all the velocities of a million molecules that all move at a similar speed than to encode all the velocities of a million molecules that move at very different speeds. The more similar the state of the molecules becomes, the shorter a text becomes that has to describe the complete state of a system. Therefore, entropy is decreasing even from the point of view of an observer with perfect information.

leereeves 4y ago

Small nit in case the author sees this: the image labelled "Entropy of each configuration of system with two dices where the observed macrostate is their sum" is either incorrect or mislabeled. For example, 2 and 12 each have 1 microstate, and ln 1 = 0, so the entropy of 2 and 12 is 0, but the image says 0.028 (which is the probability of 2 or 12, not the entropy).

sjg007 4y ago

I view entropy as a probability distribution of some set of configurations of something. Entropy is low if there’s only one configuration and high if uniformly distributed. There’s also some observer/interaction effect which is like introduction a conditional probability which would cause crystallization in an otherwise homogeneous system. Essentially a catalyst. I also find it fascinating that when it is super cold outside and you throw a pan of boiling water out the window it turns to snow instantly vs a cup of room temperature water which does not. It probably fits in terms of activation energy as well.

hatware 4y ago

Google freewall? Guess I won't read this article...

jwilk 4y ago
What do you mean?
- hatware 4y ago
  It is required that I log-in to the article using my google or facebook account to read it. Garbage. Your article isn't worth reading if I need to jump through hoops to get there.
  - jwilk 4y ago
    Here's an archived copy: https://archive.today/vRD7i https://archive.today/vRD7i
JKCalhoun 4y ago
Seeing "Read the rest of this story with a free account." Nope.

topspin 4y ago

That's pretty good as far as I'm concerned. Took me a couple years to really grasp electrical impedance. Breakthrough for me was a concise book written in 1976 by Rufus P. Turner. Subtle things take a while to get.

chemmail 4y ago

I had an art teacher who was very philosophical. One day he described to the class what entropy was. I took a lot of physics and even astrophysics. Little did i know he had a better conceptual understanding and explanation than i've ever heard before. Too bad i don't remember exactly what he said.

Koshkin 4y ago
In art, a high-entropy painting is one that would be hard to tell apart from similar paintings, one example being paintings created by simply splattering paints all over the canvas.
l33t2328 4y ago
Not to poke holes in your nostalgia, but how do you know he had a great explanation if you don’t remember it after further study?
- mhh__ 4y ago
  "Don't try to understand it, feel it" - from tenet, but does sort of apply to entropy as a way of looking at problems. That being said I heard a sports science student try to recall their working definition of entry and it was some mess of locks and keys floating around randomly hitting eachother?
  - denton-scratch 4y ago
    > definition of entry and it was some mess of locks and keys floating around randomly hitting eachother? Locks and keys does indeed sound like "entry". I thought we were discussing "entropy".
- deleted 4y ago
  [deleted]

hilbert42 4y ago

Of all of physics, entropy is the most depressing part.

stevebmark 4y ago

The author mentions Boltzman brains and that a human body could theoretically spontaneously form out of particles given a long enough time span. Of course, nothing like this can ever happen. It’s the fallacy of thinking infinite time means infinite possibilities.

nomel 4y ago
> that a human body could theoretically spontaneously form out of particles given a long enough time span. To be fair, isn't this precisely what happened?
- dilawar 4y ago
  If you leave out "spontaneously"!
  - _carbyau_ 4y ago
    Specifically if you leave it in! It's a matter of viewpoint or scope. "Spontaneous" can be defined a few different ways: https://www.merriam-webster.com/dictionary/spontaneous https://www.merriam-webster.com/dictionary/spontaneous From the link "2: arising from a momentary impulse" you get the implied meaning from the original comment if you assume standard human experience of a "moment" IE a few seconds or less. However you could argue that human evolution is but a moment in the scope of the universe. And then with definition "5: developing or occurring without apparent external influence, force, cause, or treatment" The only way it wouldn't be spontaneous would be if an external actor (Deity of your choice?) directed human evolution somehow. To say this is debatable is an understatement.. And so we have fun in word play and hopefully appreciation of different viewpoints. :-) edit: rearranged for better flow
- dj_mc_merlin 4y ago
  "Oh, that was easy," says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing."
ulamia 4y ago

photochemsyn 4y ago

Here's another head-spinning application of the concept of entropy, in quantum information theory: https://www.cambridge.org/core/books/abs/quantum-information-theory/quantum-information-and-entropy/5BCF610B966AFB2468D0F0F46249DCD0 https://www.cambridge.org/core/books/abs/quantum-information... > "The first fundamental measure that we introduce is the von Neumman entropy. It is the quantum analog of the Shannon entropy, but it captures both classical and quantum uncertainty in a quantum state. The von Neumann entropy gives meaning to a notion of the information qubit. This notion is different from that of the physical qubit, which is the description of a quantum state in an electron or a photon. The information qubit is the fundamental quantum informational unit of measure, determining how much quantum information is in a quantum system." Incidentally chem.libretexts.org, a collection of open-source chemistry textbooks, has a good overview of the physical-chemical applications. The site is kind of a mess but you'd want chapter 18.3: https://chem.libretexts.org/Bookshelves/General_Chemistry/Map:_A_Molecular_Approach_(Tro) https://chem.libretexts.org/Bookshelves/General_Chemistry/Ma...

ogogmad 4y ago

I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). It's worth considering the similarities and differences between entropy and standard deviation. I thought the 2nd law of thermodynamics was saying that with incomplete knowledge, the probability distribution of possible states becomes more and more spread out as time goes on. It's almost a limit to how you can make predictions or simulations of physics when the initial state of the system is not fully known. Equivalently, it's a banal statement about chaos in the sense of chaos theory. The only thing I don't get is how physicists get around the discrete and finite restriction. Maybe the state of the system is not what has entropy. Rather, one can define an arbitrary function f from the system to a finite set S, and then talk about the entropy of f(System at time t), because this is indeed a discrete and finite probability distribution which you can take the entropy of. Hmmm. Maybe I understand entropy.

hanche 4y ago
In Shannon’s 1948 paper, part V deals with continuous sources. The key is to realise that you cannot measure a continuous signal exactly, and so you can define a rate of information relative to the fidelity of your measurement. (I only skimmed that part years ago, and never studied it carefully. But it makes perfect sense.)
- ogogmad 4y ago
  If you mean differential entropy (which Shannon supposedly suggested as a generalisation to continuous random variables), this is not a good generalisation of entropy to continuous random variables. It lacks all the interesting properties of entropy. The "proper" generalisation of entropy to continuous random variables is something called relative entropy, or in some books it's called KL divergence. But this is now a property of how two probability distributions relate to each other, rather than a property of a single probability distribution alone. I'm not an expert in probability theory or physics, but this is what I've learnt from a brief study of these areas.

threatripper 4y ago

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equally likely. If you randomly flip bits you go from one state to another state but each one is equally likely to occur. There is no special meaning to a specific configuration if you don't give it one. If you look at the average of all bits you start grouping all states together with the equal number of 1s. If you talk about the average there is only one configuration that is all 1s but most configurations have roughly 50% 1s. If you now start flipping bits you will meander through all possible bit-states but the average will most likely be close to 50% 1s most of the time. In physics we usually look at averages such as the average velocity expressed as temperature. Therefore it makes sense to group together all states using the average and then the states with very low or very high averages are few. But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I don't know what Entropy is supposed to mean on the level of individual states/configurations. I don't understand what kind of macroscopic "averaging" function we may use to group up those states. There could be more than one possibility - from that would follow that there is more than one definition of macro-Entropy. Ideally there should be one general definition of how we have to look at those microstates and from that follows our general definition of Entropy. Sadly I didn't study Physics and this topic still continues to confuse me. The usual explanations fail to enlighten me.

oakwhiz 4y ago
>There could be more than one possibility This seems to connect with the idea behind Chaitin's incompleteness theorem. Making specific statements about the reducible complexity of something is not always possible.
pfortuny 4y ago
It is also because “the system” is intrinsic to the notion: as you say, any configuration of bits is equally likely; this only takes into account the “system of the bits “. The moment your system is “the bits and their mean value” everything changes, as there are systems with a single possible configuration. That is what happens when he starts the first example: “a system of particles INSIDE A VOLUME. The volume is what makes the entropy larger or smaller. The particles in a different volume (or just by themselves) have a different entropy.

fysicsnurd 4y ago

entropy does not increase. The universe has organized itself into people, brains, cities, iPhones

dieselerator 4y ago

That's the problem with randomness. The required Dilbert reference: https://dilbert.com/strip/2001-10-25 https://dilbert.com/strip/2001-10-25

complex_pi 4y ago

In short: the authors make a good summary of these ideas: - Entropy in thermodynamic equilibrium is well understood. The early theory (before statistical mechanics was developed) fits well with our modern understanding. - The analogies made about entropy are not always good and indeed, if you try to match the physics with "entropy is disorder" it does not always work. - In non-equilibrium situations it is, as the author points out, more complex. Regarding the last item, even Stephen Hawking postulated some strange ideas about the universe having to rewind past some point in time, so that the big crush would be the mirror of the big bang.

sytelus 4y ago

Anyone who think they understand entropy is living in a state of sin.

kgwgk 4y ago
For those missing the reference, the original quote is also great: "Anyone who attempts to generate random numbers by deterministic means is, of course, living in a state of sin." John von Neumann

deltaonefour 4y ago

>Contrary to popular opinion, uniformly distributed matter is unstable when interactions are dominated by gravity (Jeans instability) and is actually the least likely state, thus with very low entropy. Most probable states, with high-entropy, are those where matter is all lumped together in massive objects. That means over time the system becomes more ordered and starts organizing itself into spheres. I once brought this question up on physics stack exchange and basically the answers were either some form of rolling their eyes at me or dismissing me outright. The people who did answer the question stated that as particles organize themselves into spheres some other part of the universe gets hotter as a result and that the seemingly self organization I see going on with the solar system was just an isolated system. This answer still seemed far fetched to me. It still looks as if some overall self organization is still going on if the universe gets hotter on one side and matter gets organized into solar systems on another side. It took me 3 years to somewhat understand what entropy is. If you have loaded dice that always roll 6s then the dice rolling ALL 6s is the highest entropic state. rolling Random numbers would then be a low entropy state. Entropy is simply a phenomenon of probability. As time moves forward, particles enter high probability configurations. Like rolling dice. As you roll dice more and more... rolling random numbers has a higher probability then rolling all 6s... It just so happens that disordered arrangements happen to have higher probabilities in most systems. But if you look at a system of loaded dice or the solar system... in those cases Ordered configurations have higher probabilities. That's really all it is. The entire phenomenon of entropy comes down to probability and the root of probability is the law of large numbers.

empiricus 4y ago

My understanding of entropy: it is a measure of how big a system (matter + energy from a space region) is, and how much its components have interacted with each other: Entropy ~ log(number of possible system states). As the universe unfolds, systems originally isolated are starting to interact and to form bigger systems, hence the number of possible states increases, and entropy increases too.

Koshkin 4y ago
Entropy implies that these states are indistinguishable from each other.

Gravityloss 4y ago

Gas molecules in a box - entropy seems quite straighforward there. An even distribution is the most likely state and has the highest entropy. In space, at large scales, gravity starts dominating - so stars and planets are actually a higher likelihood state than an even distribtion. Isn't this just about statistical independence? In a small amount of gas (almost by definition of what is a gas), the particles don't have much effect on each other. One can assume statistical independence. While in space with gravity overwhelming other effects, the particles have very much effect on each other. Hence the statistics about their state are affected by these dependencies. So the previous intuition about entropy can't hold.

pkrumins 4y ago

I think the word entropy is science’s largest mistake.

keithalewis 4y ago
Shannon explained the name 'entropy' in (McIrvine and Tribus 1971): My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'
- pkrumins 4y ago
  Very cool! I didn’t know that.

m0llusk 4y ago

Alternatively it took ten years for Aurelien Pelissier's misunderstandings of entropy to decay.

mellavora 4y ago

The typical measure of entropy (Shannon or Gibbs, and let's spare details for later and after you've read up on the theory of large deviations) is - sum (p log(p)) which is not that different than the formula for the mean sum (p 1/n) the critical difference is the normalization constant is based on the probability of the state rather than assuming a uniform probability over all states. So, in effect, the entropy is a measure of the mean. It is a measure adopted to the case where "mean" is ill-defined because the number of modes and/or the variation around those modes is not handled well by simpler metrics.

prof-dr-ir 4y ago
If there was anyone who taught you this then they should be fired. More constructively, principal among the many things wrong with your comment is the formula for the mean; sum_i p_i = 1, so sum_i p_i / n = 1 / n. The mean would instead be sum_i p_i x_i.
im3w1l 4y ago
It can be related to compression. If some phrase has a probability p_i of occuring, then the optimal length for the code is -log(p_i). The entropy sum(-p_i log_pi) = mean(-log(p_i)) is how long code you will use on average.
gjm11 4y ago
Perhaps I'm misunderstanding or missing something, but I'm afraid this seems completely wrongheaded to me. (My apologies for being so blunt, but right now your comment appears to be the most-upvoted, and I therefore think it needs some pushback.) [EDITED to add: I was looking at an old version of the page; by the time I wrote this the parent was no longer the top comment. I'll leave the bluntness in, especially as at least one other person was even blunter.] You refer to "the mean" and I think you mean the mean of the probabilities. Now, when you've got a probability distribution, by far the usual thing for "the mean" to mean is the sum of Pr(x) x -- the mean of the values. Taking the mean of the probabilities is a really strange thing to do. One reason why it's a really strange thing to do is that this thing you call n is really kinda meaningless. There's no difference between these two probability distributions: (a) 1, 2, 3, or 4, with probabilities 0.1, 0.2, 0.3, 0.4 respectively; (b) 1, 2, 3, 4, or 5, with probabilities 0.1, 0.2, 0.3, 0.4, 0 respectively. But (a) has n=4 and (b) has n=5. Maybe you want n to be the number of nonzero probabilities? But now consider (a) along with the following probability distribution parameterized by a (small, positive) number h: 1, 2, 3, 4, or 4+h, with probabilities 0.1, 0.2, 0.3, 0.4-h, h. Every version of this distribution with h>0 has n=5, but when h is very small it's practically indistinguishable from (a) with n=4. Further, since the sum of probabilities is always 1, what you write as sum (p 1/n) is just the same as the number 1/n. You can call it "the mean" if you want to, but I don't see what this adds over calling it what it is: the reciprocal of the number of possibilities. There is something to what you say: the entropy is kinda related to the number of possibilities; if the probabilities are all equal, the entropy is log(#possibilities); if the probabilities are equal-ish then it's modestly smaller than that. But note e.g. that this relationship is exactly the inverse of what you say, in that "the mean" decreases with the number of possibilities, and the entropy increases with the number of possibilities. The entropy is not "a measure of the mean". It kinda-sorta is related to "the number of possibilities", which is the reciprocal of "the mean". It is not at all the case, as your last paragraph suggests, that for most purposes we should be using "the mean" but we need to use the entropy when "the number of modes ... is not handled well by simpler metrics", whatever that means; for most purposes we should be using the entropy, and in the special case where all the probabilities are equal we can get away with just counting possibilities. (In some important situations it turns out that what you have is some number of possibilities with roughly equal probabilities, and a whole lot more whose probabilities rapidly decrease to almost zero, and then you can get away with counting the number of reasonably-probable possibilities and taking its log. E.g., various situations in communications theory can fruitfully be thought of this way. But the entropy is still the more fundamental quantity, and "the mean" is still a needless obfuscation of "the (effectively) number of possibilities".)

mensetmanusman 4y ago

I had the pain, and pleasure, of taking and then (assistant) teaching thermodynamics at MIT. One of the tidbits that always stuck with me was that astronomers have estimated that observable universe’s total entropy: When you compare that value to the maximum possible entropy, i.e. the heat death of the universe, and then to the ridiculously low entropy state of the beginning of the universe, we are currently halfway along in that ‘timeline’. It always brought to mind a grandfather clock; the clock stops when the weight hits the floor, and we are halfway there…

andrewgleave 4y ago

The Science of Can and Can't[1] is interesting in how it looks to address a number of fundamentals via counterfactuals including the 2nd Law of Thermodynamics. Edit: See [2] for background about Constructor Theory. [1] https://www.chiaramarletto.com/books/the-science-of-can-and-cant/ https://www.chiaramarletto.com/books/the-science-of-can-and-... [2] https://www.youtube.com/watch?v=8DH2xwIYuT0 https://www.youtube.com/watch?v=8DH2xwIYuT0

snikeris 4y ago

> Boltzmann imagined that our universe could have reached thermodynamical equilibrium and its maximal entropy state a long time ago, but that a spontaneous entropy decrease to the level of our early universe occured after an extremely long period of time, just for statistical reasons. I’m interested in reading more about this. Any pointers?

poulpy123 4y ago

not to brag but it took only 2 months to forget anything about it

sylware 4y ago

scientific method: it started with thermodynamic entropy, but scientits found out that this truth is much deeper engrained in our universe, then we got a mathematically generalized version, which is now used used to explain the "arrow of time" which our time reversable physics equations would not be able to explain alone.

dandanua 4y ago

The problem is that entropy is a subjective notion. It's a measure of our lack of knowledge about the state of a system.

AnthonyAguirre 4y ago

Actually a quite nice article. After also spending years as a professional physicist not understanding entropy, I finally decided that I was not necessarily the problem, and spent the last 5 years or so trying to understand it better by rewording the foundations with my research group. (I'm one of the papers the author cites is part of a series from our group developing "observational entropy" in order to do so.) A lot of what makes this topic confusing is just that there are the two basic definitions — Gibbs (\sum p_i log l_i) and "Boltzmann" (log \Omega) — entropy, and they're really rather different. There's usually some confusing handwaving about how to relate them, but the fact is that in a closed system one of them (generally) rises and the other doesn't, and one of them depends on a coarse-graining into macrostates and the other doesn't. The better way to relate them, I've come to believe, is to consider them both as limits of a more general entropy (the one we developed — first in fact written down in some form by von Neumann but for some reason not pursued much over the years.) There's a brief version here: https://link.springer.com/article/10.1007/s10701-021-00498-x https://link.springer.com/article/10.1007/s10701-021-00498-x. This entropy has Gibbs and Bolztmann entropy as limits, is good in and out of equilibrium, is defined in quantum theory and with a very nice classical-quantum correspondence, and has been shown to reproduce thermodynamic entropy in both our papers and the elegant one by Strasberg and Winter: https://journals.aps.org/prxquantum/abstract/10.1103/PRXQuantum.2.030202 https://journals.aps.org/prxquantum/abstract/10.1103/PRXQuan... After all this work I finally feel that entropy makes sense to me, which it never quite did before — so I hope this is helpful to others. p.s. If you're not convinced a new definition of entropy is called for, ask a set of working physicists what it would mean to say "the entropy of the universe is increasing." Since von Neumann entropy is conserved in a closed system (which the universe is if anything is), and there really is no definition of a quantum Boltzmann entropy (until observational entropy), the answers you'll get will be either a mush or a properly furrowed brows.

stazz1 4y ago
The universe is not a closed system
- AnthonyAguirre 4y ago
  How do you define "the universe"?

raxxorraxor 4y ago

As a computer scientist it isn't helpful that the entropy in thermodynamics and that in computer science (informational content - I don't know a good English term) collide a bit.

Buttons840 4y ago

I recommend Information Theory for Intelligent People: http://tuvalu.santafe.edu/~simon/it.pdf http://tuvalu.santafe.edu/~simon/it.pdf

Buttons840 4y ago
Let me add: This PDF is only 13 pages and presents a few different ways of viewing entropy. It was easy for me to follow as an undergrad.

cb321 4y ago

Of possible related interest: https://arxiv.org/abs/chao-dyn/9603009 https://arxiv.org/abs/chao-dyn/9603009 I think Bricmont is a clear thinker/presenter on these matters and this article actually showed up in a "for humanities people" anthology. [1] [1] https://www.amazon.com/Flight-Science-Reason-Academy-Sciences/dp/0801856760 https://www.amazon.com/Flight-Science-Reason-Academy-Science...

zwieback 4y ago

Of course you don't need to really understand entropy for it to be useful. It's definitely an interesting concept but when I was crunching equations for Thermodynamics, one of the weeder classes for ME, it becomes clear you need it for things to balance out. Once you've cranked threw a dozen or so problems you get a feel for what it is even if the physics and the spiritual side of it remains murky. Now, 35 years later, when I marvel at my new engine or what have you, I still vaguely remember my entropy-problems days and appreciate that someone worked this stuff out.

est 4y ago

Shannon called the function "entropy" and used it as a measure of "uncertainty," interchanging the two words in his writings without discrimination