12 ms·

Entropy explained, with sheep (2016)

216 points by mlejva 6y ago

brummm 6y ago
As a physicist, this is such a great explanation that's actually correct for a change. Entropy must be one of the if not the most misunderstood physical concept out there (along with Planck metrics like Planck energy or Planck length). Entropy is commonly used and written about by so many people that clearly lack the understanding of it that this blog post is a refreshing change.
- cinntaile 6y ago
  The author is a physicist as well. https://mobile.twitter.com/aatishb https://mobile.twitter.com/aatishb
  - brummm 6y ago
    Yeah, it showed in the way he explained things.
paultopia 6y ago
Alternative name: an explanation of entropy that isn't at all baaaaaaaad.
xwdv 6y ago
If the entropy of the universe were to suddenly go in reverse would we be able to detect it or would our memory formation and perceptions being reversed make it indistinguishable from what we experience now?
- jbay808 6y ago
  This is a very good question. Yes, time reversal would also reverse the process of our perception and memory, so we would experience time "moving forward" even if time were "moving" backward. We would experience entropy increasing even if entropy were "decreasing with time". (And it's perfectly valid to call the past "+t" and the future "-t"; the laws of physics don't care; if you do that, you'll see that entropy decreases as "t" increases, but despite changing definitions you'll still only have memories of a universe with lower entropy than the present). This is one reason why it might be more correct to say that time doesn't move at all. We just perceive a direction of time wherever the universe has an entropy gradient along "the time axis".
  - PaulDavisThe1st 6y ago
    Entropy decreasing is not equivalent to time reversal. [Edit: to expand a bit: time reversal requires that some previous macrostate is achieved again. Entropy decreasing merely requires that the system enters any macrostate represented by less microstates than the current one. Put in more simple terms, the ice cube could reform but in a different shape. Entropy would have decreased, but it would not "look like" time reversal - it would just look like something very strange had happened. ]
    - jbay808 6y ago
      This is true! I oversimplified a bit, because it's much easier to concretely discuss that scenario and its consequences, and it's more important from a philosophical perspective. Just like it's much easier to catch ice cubes forming on film by rewinding a tape than to run the camera waiting for spontaneous entropy decrease. Nonetheless, if the entropy of the whole universe were to consistently decrease for a sufficient time, even along a different route than time reversal, you should gradually start to notice that it's easier to form predictions or "memories" of the future than to remember the past, due to Landauer's principle. Before long, you should go back to considering the future to be the state of higher entropy.
      - PaulDavisThe1st 6y ago
        I think you're priviledging microstates. There are plenty of entropy decreases that are possible without the ice cube reforming. They're (extremely) unlikely of course, but somewhat less unlikely than the new, singular microstate you're asserting is somehow "less of a coincidence". It's vastly more likely (for example) that 1% of the particle velocities reverse than all of them doing so, and that could (depending ...) still lead to a decrease in the entropy of the system (but not a reformed ice cube). I don't see how a long term entropy decrease that does not follow the time reversal route could be easier to form predictions in. There are many possible higher entropy states for the water than the water sitting in the glass, and if the universe enjoyed random entropy decreases, I see no reason why it would tend to pick certain macrostates over others.
keenmaster 6y ago
Great explanation. What are the top theories for why the universe began in a low entropy state? The mere fact of that seems to contradict our current understanding of entropy. That implies that there is something very fundamental about the universe which we don't understand.
- selim_tiotoys 6y ago
  There is a fantastic book by Leonard Susskind [0] that tackles this questions. His answer - rooted in string theory - is based on a "landscape of possibilities" and we happen to be in the place where all elementary particles (more specifically the Higgs) had the right energy configuration. [0] The cosmic Landscape: https://www.amazon.com/Cosmic-Landscape-String-Illusion-Intelligent/dp/0316013331 https://www.amazon.com/Cosmic-Landscape-String-Illusion-Inte...
  - keenmaster 6y ago
    Where did those particles come from? Is there any scientific theory for the beginning of the beginning? A friend who studied physics told me that there's no reason that matter/energy couldn't have just existed forever. I don't find that explanation satisfying. If "entropy always increases" is a universal rule, then there must have been a time where it went from zero to non-zero. If "matter cannot be created or destroyed" is supposed to be a universal rule, then the mere existence of matter seems to contradict the rule, rendering it a non-universal tautology with vast but nonetheless bounded application.
    - AnimalMuppet 6y ago
      No. In a forever-existing universe, as you go back in time, entropy could asymtotically approach zero (or some other limit) without ever actually reaching it. It doesn't have to be zero at some point in the past. In fact, if it is zero at some point in the past, then what happened before that? Did it not increase before that point, or was it less than zero before that point?
      - 6y ago
ramadis 6y ago
I recommend you the story "The Last Question", by Isaac Asimov[0], one of my favourite stories of all time. It's about how the question about reverting the direction of the entropy keeps rising on people, throughout the life of the universe. [0] https://en.wikipedia.org/wiki/The_Last_Question https://en.wikipedia.org/wiki/The_Last_Question
djaque 6y ago
Great article! I'm almost done with a PhD in physics and statistical physocs is still the hardest thing for me to wrap my mind around. Even more so than quantum and relativity. There's something about how unintuitive math becomes in ultra high dimension and the old timey feeling way that it was taught to me (steam engines and stuff) that makes it difficult to learn.
- atrettel 6y ago
  Thermodynamics as a subject is taught in two entirely different traditions, the engineering tradition and the physics tradition. I've taken courses in both traditions and I personally find that physics tradition explains a lot more. When I took engineering thermodynamics, the focus was on solving "practical" problems, and I never got a good understanding of what entropy was. The point of the class was not to teach physics, though. The point was to get engineers to solve thermodynamics problems (like heat engines and cycles). The primary thing I remember from that course was looking up things in the tables in the back of the book and converting between Btus and other units. I only understood entropy after taking a statistic physics course. The course's explanation of entropy is basically the same explanation as the article. The point of that class was entirely different than the engineering course, so the professor spent several lectures going into detail about what entropy is and how it relates to everything else. I never had to do any table lookups in that class too. Side note: you can easily tell what tradition you were taught in based on the sign convention in the first law of thermodynamics you learned: https://en.wikipedia.org/wiki/First_law_of_thermodynamics#Sign_conventions https://en.wikipedia.org/wiki/First_law_of_thermodynamics#Si...
iandanforth 6y ago
Great article, I had hoped it would also cover locally anti-entropic processes. We do see liquids turn into regular solids after all: https://www.youtube.com/watch?v=caGX6PoVneU https://www.youtube.com/watch?v=caGX6PoVneU
- surajs 6y ago
  Profound and enlightening, thanks!
- trhway 6y ago
  >locally anti-entropic processes. We do see liquids turn into regular solids after all: Crystallization is exothermic process. It releases heat into environment. It converts potential chemical energy into heat as result of bond making. Thus the total entropy of the system "crystal forming liquid + environment" is increased. Life is another famous process of local anti-entropy which is driven by the increase of the total entropy.
maxov 6y ago
Loved this article. There are so many applications of entropy and statistical physics in computer science, and I find it fascinating that the same general properties are useful in such different contexts. For example, there's a well-known phenomenon in probability called concentration of measure. One of the most important examples in computer science is if you flip n coins independently, then the number of heads concentrates very tightly. In particular, the probability that you are more than an epsilon-fraction away from 0.5n heads is at most around e^{-epsilon^2 n}. This is exactly the setting described in the article with the sheep in two pens, and this inequality is used in the design of many important randomized algorithms! A classic example is in load balancing, where random assignment sometimes produces 'nice' configurations that look very much like the high-entropy states described in this article (but unfortunately, many times random assignments don't behave very well, see e.g. the birthday paradox). The sum of independent random variables is well known to have concentration properties. An interesting question to me is what sorts of other statistics will exhibit these concentration phenomena. An important finding in this area is the bounded differences inequality (https://web.eecs.umich.edu/~cscott/past_courses/eecs598w14/notes/09_bounded_difference.pdf https://web.eecs.umich.edu/~cscott/past_courses/eecs598w14/n...), which generally states that any function that doesn't "depend too much" on each individual random argument (and the sum of bounded variables satisfies this assumption) exhibits the same concentration phenomenon. There are some applications in statistical learning, where we can bound the estimation error using certain learning complexity measures that rely on the bounded differences inequality. In the context of this article, that means there's a whole class of statistics that will concentrate similarly, and perhaps exhibit irreversibility at a macroscopic level.
- pabo 6y ago
  There's a related anecdote about John von Neumann: he used to joke that he has superpowers and can easily tell truly random and pseudo random sequences apart. He asked people to sit down in another room and generate a 0/1 sequence via coin flips, and record it. Then, generate another sequence by heart, trying to mimick randomness as much as possible. When people finally showed the two sequences to him, Neumann could instantly declare which one was which. People were amazed. The trick he used was based on the "burstiness" rule you describe: a long enough random sequence will likely contain a long homogeneous block. While humans tend to avoid long streaks of the same digit, as it does not feel random enough. So, all he did was he quickly checked with a glimpse, which of the two sequences contained the longest homogeneous block, and recognized that as the one generated via the coin flips.
andrewflnr 6y ago
So one thing I've never understood is how you can "count" microstates, or bits required to describe them, when the relevant physical parameters all seem to be real numbers. For instance, a gas of N atoms is described by 6N real numbers (3d position and velocity) regardless of how hot it is. The article talks about quanta of energy, but that seems like a simplification at best: a given interaction might be quantized, but quanta in general (e.g. photons) exist on a real-number spectrum, so it's possible to have an uncountable infinity of energy packet sizes right? (That's the point of a black body, right?) What am I missing? Is this a weird measure theory thing, where hot objects have even bigger uncountable infinities of states and we get rid them all with something like a change of variables? If you told me spacetime was secretly a cellular automaton I could deal with it, but real numbers ruin everything.
- foota 6y ago
  IANAP but there's two different things that come to mind. Even if the number of states were infinite, as long as there's a reasonable probability distribution you can know you can integrate over the probabilities that have some property vs another (say: solid vs not). Secondly, energy is quantized (on a very very very small level) in reality. Ymmv though, I tried googling this and read some stuff about waves being quantized and particles not and about how it depends on what system you're looking at and I had to drop out of quantum physics when I took it :-)
  - kergonath 6y ago
    For quantum objects, the distinction between a wave and a particle is not very meaningful. The energy levels of an electron around a nucleus are discrete, regardless of whether the electron behaves more like a wave or more like a particle in the specific experiment you’re doing. Otherwise, you’re right: we count states by integrating (at times discrete, continuous, and often very complex) probability distributions.
- keithlfrost 6y ago
  This is an excellent, and puzzling, question! Let me try to provide some insight into how physicists think about such paradoxes, by addressing the specific example you mention, of the presumed uncountable infinity of different possible photon energies in a finite range of frequencies. In the case of blackbody radiation, when physicists analyze the set of possible photon energies more carefully, they find that there is really only an infinite number of different possible photon energies (for a finite range of frequencies) if the volume of space containing the photons is infinite. In any finite volume of space, if we allow ourselves to place boundary conditions on the electromagnetic fields at the edges of that volume (for example, suppose we think of our volume as a cube with mirrored walls), we find that there are only a finite number of oscillating modes of the electromagnetic field in any finite range of frequency. In the case of a cube, there is a very lowest frequency of radiation whose wavelength will allow it to form a standing wave in the box, and all the other allowed modes are multiples of that lowest frequency. So the density of possible photon states per unit of frequency is actually proportional to the volume of space we allow to hold the photons. (By the way, this is also precisely related to the quantum complementarity of uncertainty between momentum and position. To confine a photon to a volume of space, the uncertainty in its momentum must be the same order as that carried by the lowest frequency standing waves which would be compatible with the container.)
pawelmi 6y ago
Reminded me of Anna Karenina principle. https://en.m.wikipedia.org/wiki/Anna_Karenina_principle https://en.m.wikipedia.org/wiki/Anna_Karenina_principle
PaulDavisThe1st 6y ago
This was really great. It's how I was taught entropy at college (biophysics and molecular biology) though without the sheep. "Statistical Mechanics" was the name our professor used. The only tiny change I'd like to make is to add a line or two near the end, something along the following lines: There's a lot fewer ways to arrange water molecules so that they form an ice cube than there are to arrange them as a liquid. Most arrangements of water molecules look like a liquid, and so that's the likely endpoint even if they start arranged as an ice cube. The same is true of more or less any macroscopic object: the thing that we recognize and name ("chair", "table", "pen", "apple") requires the atoms to remain in one of a fairly small set of particular arrangements. Compared to the vast number of other arrangements of the same atoms ("dust"), the ones where the atoms form the "thing" are quite unlikely. Hence over time it's more likely that we'll find the atoms in one of the other ("random", or "dust-like") arrangements than the one we have a name for. The reason things "fall apart" isn't that there's some sort of preference for it - it's that there are vastly more ways for atoms to be in a "fallen apart" state than arranged as a "thing".
- deleted 6y ago
  [deleted]
- kgwgk 6y ago
  This kind of intuition can be dangerous: "Students who believe that spontaneous processes always yield greater disorder could be somewhat surprised when shown a demonstration of supercooled liquid water at many degrees below 00 C. The students have been taught that liquid water is disorderly compared to solid ice. When a seed of ice or a speck of dust is added, crystallization of some of the liquid is immediate. Orderly solid ice has spontaneously formed from the disorderly liquid. "Of course, thermal energy is evolved in the process of this thermodynamically metastable state changing to one that is stable. Energy is dispersed from the crystals, as they form, to the solution and thus the final temperature of the crystals of ice and liquid water are higher than originally. This, the instructor ordinarily would point out as a system-surroundings energy transfer. However, the dramatic visible result of this spontaneous process is in conflict with what the student has learned about the trend toward disorder as a test of spontaneity. "Such a picture might not take a thousand words of interpretation from an instructor to be correctly understood by a student, but they would not be needed at all if the misleading relation of disorder with entropy had not been mentioned." http://entropysite.oxy.edu/cracked_crutch.html http://entropysite.oxy.edu/cracked_crutch.html
jaspal747 6y ago
excuse my naivete, but do black holes help reduce entropy by capturing/engulfing things around them? Is that the cycle how universe keeps creating and recreating itself?
- adeledeweylopez 6y ago
  Nope, black holes have entropy proportional to the surface area of their event horizon. So the more stuff they engulf, the more their entropy increases, and thus they satisfy the 2nd law of thermodynamics just like everything else. It's a very interesting and active area of physics actually: https://en.wikipedia.org/wiki/Black_hole_thermodynamics https://en.wikipedia.org/wiki/Black_hole_thermodynamics
  - WealthVsSurvive 6y ago
    Has this been observed, or is it more: here's some math that makes entropy even possible because the alternatives sound unlikely? I'm betting on the weird, things like the universe being generative on a macro scale and entropic in local timespace, wherein dark matter is merely newly created matter not in another universe, but in this universe, maybe some unknown interactions between the unstoppable force of expansion and the immovable object of a black hole's gravity. I'm probably blathering, I'm not a physicist.
layer8 6y ago
That still doesn’t answer the question how, if the laws of physics are time-symmetric, the universe as a whole can have a time-asymmetric evolution of entropy. I.e., if something forces entropy to increase in the long run, then that should hold in both directions of time. So what is it that causes entropy to only increase in the direction of the future, but not in the direction of the past, given that the laws pf physics do not distinguish between both directions?
- kergonath 6y ago
  We don’t have an answer to this question. I don’t want to discuss metaphysics here, but there is a very interesting discussion on that subject here: https://youtube.com/watch?v=-6rWqJhDv7M https://youtube.com/watch?v=-6rWqJhDv7M
  - layer8 6y ago
    Let me rephrase maybe: Given the state of affairs I described above, I don’t understand what is the convincing argument that entropy does indeed increase in the long run. Any argument given should also work in the reverse direction, given the symmetry of time, shouldn’t it? (And thereby create a kind of reductio ad absurdum.) If not, why not?
    - 0134340 6y ago
      >symmetry of time Time isn't symmetric. "Since the second law of thermodynamics states that entropy increases as time flows toward the future, in general, the macroscopic universe does not show symmetry under time reversal." - https://en.wikipedia.org/wiki/T-symmetry https://en.wikipedia.org/wiki/T-symmetry
    - kergonath 6y ago
      It’s fundamentally an empirical law, even if one of the best-verified ones. I don’t think there really is a convincing argument, other than “we haven’t seen a single instance of the opposite being true”. It’s a missing piece in quantum mechanics (linked to decoherence) and quantum loop gravity (in which time is...interesting). The issue you mention (how can the 2nd law and the fact that the laws of Physics don’t care about the direction of time be true at the same time) is a big problem. I really recommend watching the video if you haven’t, it is fascinating and well worth the time.
bobsoap 6y ago
I love how the notion of entropy permeates into so many other things. It's fundamental, universal, and at the heart of nearly every aspect of our existence. Take philosophy. If the ultimate state of everything culminates in chaos (according to the theory of entropy), the human existence constitues the exact opposite: controlling the chaos that surrounds us, and shaping it into something useful and, in entropy-speak, progressively unprobable. Making ice cubes out of water. It follows that our existence can at least be described as a function of entropy. This describes life. Survival is the battle against entropy. Procreation is the chosen weapon against chaos, a force of order in a universe that can't help itself but fall into chaos -- and, undoubtedly, will ultimately prevail in that fight. In the long run, life will lose. Great article. Fascinating stuff.
- antonios 6y ago
  Alternatively, all that attempts to control chaos and decrease entropy actually results in faster entropy increase overall on a systemic level. I remember reading about a (Russian?) physicist that believed that life simply happens as a result of the universe's attempt to increase entropy faster on sufficiently complicated systems. If someone remembers his name I'd be obliged.
  - 0134340 6y ago
    Dorion Sagan (Yes, Carl Sagan's son) also covers this in his book Into The Cool. That life is only a force multiplier in increasing it universally, basically.
- imcoconut 6y ago
  It's true. Have you read the book "What is Life?" By Schrodinger? He discusses the very concept you're talking about as he speculates on the physical substrate of genetic information (which was discovered to be DNA not long after the book was published).
- combatentropy 6y ago
  > Survival is the battle against entropy See my username ;) But my inspiration was from trying to make a difference within corporate culture, not raw survival. > controlling the chaos that surrounds us, and shaping it into something useful i.e., work > In the long run, life will lose. Unless there is a transcendent being beyond our universe, an explanation believed by many people for why, a long time ago, entropy was microscopic. The article admits it can't come up with a materialistic explanation and ends by calling it a "mystery".
rstuart4133 6y ago
That's remarkably well done, but the underlying principle can be summarised as "regression towards the mean" which is not so difficult to understand. Now explain this: when the sheep are cooped up in one box you can extract energy from the system. This becomes apparent if you imagine the boundary between the boxes is a fan. As the sheep move from what is a high pressure on one side to 0 pressure on the other they will move the fan. Attach a generator to the fan, and you get energy out. This configuration, with high pressure on the side low on the other, will happen every so often through random chance. Admittedly not very often, in fact so rarely it's useless to us. But in principle we have a perpetual motion machine. We don't, of course. But why not? And more to the point is the 2nd law nothing more than a statement of averages? By which I mean "order always tends towards disorder" is in fact false. We will every so often see order arise from disorder. So the 2nd law is not really a hard and fast law, any more then "you always loose when gambling at the casino" is a hard and fast law.
- deleted 6y ago
  [deleted]
boyobo 6y ago
Great article. Can energy be explained in a similarly simple fashion? I'm pretty comfortable with probability theory so this entroppy explaination makes sense, but I still don't understand what energy is. Also, who determines what a 'macroscopic variable' is? Why do there only seem to be 3 for gassess (V, T, P?)
- petschge 6y ago
  There is more macroscopic variables, for example the three components of the bulk flow velocity or the density of the gas.
  - boyobo 6y ago
    But are those the only 6?
leafboi 6y ago
https://www.youtube.com/watch?v=wI-qAxKJoSU https://www.youtube.com/watch?v=wI-qAxKJoSU In the video above you will witness a metal wire in a disorderly shape spontaneously form into an organized spring shape when entropy is increased (the wire is heated). There are no special effects or video rewinding trickery here, the phenomenon is very real. It is as if you were looking at water spontaneously form into ice cubes but entropy is NOT reversing! It is moving forward. The video is a really good example of entropy. It's good in the sense that if you understand why entropy is increasing when the metal is heated than you truly understand what entropy is... as the video gets rid of the notion of order and disorder all together. That's right. There are many cases of increasing entropy where the system spontaneously progresses from a greater disorder to more organization. Many people say that the when some system becomes more organized that means entropy is leaving the local system and increasing overall in the global system. This is not what's happening here. The wire is being heated. Atoms are becoming more organized and less disorderly by virtue of MORE entropy entering the system. If you understand this concept then you truly understand entropy. If not you still don't get it.
aoowii 6y ago
Order as humans understand it is different from physical order. We see order as an array of ascending numbers or a house of cards. The universe sees order more like a state of 'useful' energy, where it's possible to extract it, and seemingly became ordered by putting energy in. This is why I think the information theory definition of entropy is a bit misleading. I like to see it as a radioactive atom, which starts as a useful structure with potential energy that has an inherent timer until this energy is lost due to the universe wanting to return to equilibrium. So it's statistically extremely unlikely to get the atom back to its original energetic state by nature, but not because because it's literally impossible, it's just impossible to do this without putting energy back in and that's not something that happens naturally. A reversal of entropy is entirely possible, it's just called doing work. Of course, we don't have the knowledge necessary to reverse each individual atom in the melting process of an ice cube, but one day we might. It's theoretically possible with enough work. Of course, there will always be a loss of energy, but i'm pretty sure that's an entirely different thermodynamic law.
roimor 6y ago
I don't really understand this "Entropy Is All About Arrangements" takeaway. A fair coin has higher entropy than a biased one. What are the "arrangements" in this case?
- kgwgk 6y ago
  The coins don't have entropy, the sequences they produce do (in information theory sense, this is not about physics). For a long sequence (say 1000), the former will produce close to 500 heads and 500 tails. The latter, assuming one heads are three times as probable as tails, will produce around 750 heads and 250 tails. There are many more different sequences of the first kind.
  - roimor 6y ago
    I'm referring to the entropy of the Bernoulli distribution. If the coin is fair, the entropy is 1 bit... if the coin isn't fair, then the entropy of the distribution is less than 1 bit. I'm having trouble reconciling the information theory way of thinking about entropy as a function of a distribution, with how physicists tend to think of entropy of arrangements.
    - deleted 6y ago
      [deleted]
    - kgwgk 6y ago
      There is a relationship between the distribution of x_i and the sequences generated from that distribution x_1, x_2, ..., x_n. If the coin isn't fair there are less arrangements possible. If the coin has two heads there is only one possible sequence.
ba0bab 6y ago
great explanation and visuals. but I do not quite get the way the arrangements of sheep are treated. They seem to be counted in the standard "Unordered Sampling with Replacement" fashion. >Just as the sheep wander about the plots of land in the farm, these packets of energy randomly shuffle among the atoms in the solid. this would mean that any "packet" of energy is equally likely to be in any of the buckets and the "packets" are independent of each other. so we have an equal distribution on the product space with 6^6 elements. but then later > Now, let’s assume the sheep are equally likely to be in any of these 462 arrangements. (Since they move randomly, there's no reason to prefer one arrangement over another.) under the prior assumptions these arrangements would not be equally likely. e.g. "all sheeps in plot 1" would be far less likely than "each sheep in a different plot" am I missing something here? in any case the same conclusions can be drawn in both cases, only that the concentration around 3 is already more pronounced in the "6 sheep, 6 plots" case using the product space model.
- emdubbelyou 6y ago
  I think the confusion is in the way that sheep as a word can be both plural and singular. Specifically, one sheep is as likely to be in any single spot compared to any other single spot. It’s when you get to more than one sheep that you see the distributions

216 points by mlejva 6y ago

As a physicist, this is such a great explanation that's actually correct for a change. Entropy must be one of the if not the most misunderstood physical concept out there (along with Planck metrics like Planck energy or Planck length). Entropy is commonly used and written about by so many people that clearly lack the understanding of it that this blog post is a refreshing change.

cinntaile 6y ago
The author is a physicist as well. https://mobile.twitter.com/aatishb https://mobile.twitter.com/aatishb
- brummm 6y ago
  Yeah, it showed in the way he explained things.

paultopia 6y ago

Alternative name: an explanation of entropy that isn't at all baaaaaaaad.

xwdv 6y ago

If the entropy of the universe were to suddenly go in reverse would we be able to detect it or would our memory formation and perceptions being reversed make it indistinguishable from what we experience now?

jbay808 6y ago
This is a very good question. Yes, time reversal would also reverse the process of our perception and memory, so we would experience time "moving forward" even if time were "moving" backward. We would experience entropy increasing even if entropy were "decreasing with time". (And it's perfectly valid to call the past "+t" and the future "-t"; the laws of physics don't care; if you do that, you'll see that entropy decreases as "t" increases, but despite changing definitions you'll still only have memories of a universe with lower entropy than the present). This is one reason why it might be more correct to say that time doesn't move at all. We just perceive a direction of time wherever the universe has an entropy gradient along "the time axis".
- PaulDavisThe1st 6y ago
  Entropy decreasing is not equivalent to time reversal. [Edit: to expand a bit: time reversal requires that some previous macrostate is achieved again. Entropy decreasing merely requires that the system enters any macrostate represented by less microstates than the current one. Put in more simple terms, the ice cube could reform but in a different shape. Entropy would have decreased, but it would not "look like" time reversal - it would just look like something very strange had happened. ]
  - jbay808 6y ago
    This is true! I oversimplified a bit, because it's much easier to concretely discuss that scenario and its consequences, and it's more important from a philosophical perspective. Just like it's much easier to catch ice cubes forming on film by rewinding a tape than to run the camera waiting for spontaneous entropy decrease. Nonetheless, if the entropy of the whole universe were to consistently decrease for a sufficient time, even along a different route than time reversal, you should gradually start to notice that it's easier to form predictions or "memories" of the future than to remember the past, due to Landauer's principle. Before long, you should go back to considering the future to be the state of higher entropy.
    - PaulDavisThe1st 6y ago
      I think you're priviledging microstates. There are plenty of entropy decreases that are possible without the ice cube reforming. They're (extremely) unlikely of course, but somewhat less unlikely than the new, singular microstate you're asserting is somehow "less of a coincidence". It's vastly more likely (for example) that 1% of the particle velocities reverse than all of them doing so, and that could (depending ...) still lead to a decrease in the entropy of the system (but not a reformed ice cube). I don't see how a long term entropy decrease that does not follow the time reversal route could be easier to form predictions in. There are many possible higher entropy states for the water than the water sitting in the glass, and if the universe enjoyed random entropy decreases, I see no reason why it would tend to pick certain macrostates over others.

keenmaster 6y ago

Great explanation. What are the top theories for why the universe began in a low entropy state? The mere fact of that seems to contradict our current understanding of entropy. That implies that there is something very fundamental about the universe which we don't understand.

selim_tiotoys 6y ago
There is a fantastic book by Leonard Susskind [0] that tackles this questions. His answer - rooted in string theory - is based on a "landscape of possibilities" and we happen to be in the place where all elementary particles (more specifically the Higgs) had the right energy configuration. [0] The cosmic Landscape: https://www.amazon.com/Cosmic-Landscape-String-Illusion-Intelligent/dp/0316013331 https://www.amazon.com/Cosmic-Landscape-String-Illusion-Inte...
- keenmaster 6y ago
  Where did those particles come from? Is there any scientific theory for the beginning of the beginning? A friend who studied physics told me that there's no reason that matter/energy couldn't have just existed forever. I don't find that explanation satisfying. If "entropy always increases" is a universal rule, then there must have been a time where it went from zero to non-zero. If "matter cannot be created or destroyed" is supposed to be a universal rule, then the mere existence of matter seems to contradict the rule, rendering it a non-universal tautology with vast but nonetheless bounded application.
  - AnimalMuppet 6y ago
    No. In a forever-existing universe, as you go back in time, entropy could asymtotically approach zero (or some other limit) without ever actually reaching it. It doesn't have to be zero at some point in the past. In fact, if it is zero at some point in the past, then what happened before that? Did it not increase before that point, or was it less than zero before that point?
    - 6y ago

ramadis 6y ago

I recommend you the story "The Last Question", by Isaac Asimov[0], one of my favourite stories of all time. It's about how the question about reverting the direction of the entropy keeps rising on people, throughout the life of the universe. [0] https://en.wikipedia.org/wiki/The_Last_Question https://en.wikipedia.org/wiki/The_Last_Question

djaque 6y ago

Great article! I'm almost done with a PhD in physics and statistical physocs is still the hardest thing for me to wrap my mind around. Even more so than quantum and relativity. There's something about how unintuitive math becomes in ultra high dimension and the old timey feeling way that it was taught to me (steam engines and stuff) that makes it difficult to learn.

atrettel 6y ago
Thermodynamics as a subject is taught in two entirely different traditions, the engineering tradition and the physics tradition. I've taken courses in both traditions and I personally find that physics tradition explains a lot more. When I took engineering thermodynamics, the focus was on solving "practical" problems, and I never got a good understanding of what entropy was. The point of the class was not to teach physics, though. The point was to get engineers to solve thermodynamics problems (like heat engines and cycles). The primary thing I remember from that course was looking up things in the tables in the back of the book and converting between Btus and other units. I only understood entropy after taking a statistic physics course. The course's explanation of entropy is basically the same explanation as the article. The point of that class was entirely different than the engineering course, so the professor spent several lectures going into detail about what entropy is and how it relates to everything else. I never had to do any table lookups in that class too. Side note: you can easily tell what tradition you were taught in based on the sign convention in the first law of thermodynamics you learned: https://en.wikipedia.org/wiki/First_law_of_thermodynamics#Sign_conventions https://en.wikipedia.org/wiki/First_law_of_thermodynamics#Si...

iandanforth 6y ago

Great article, I had hoped it would also cover locally anti-entropic processes. We do see liquids turn into regular solids after all: https://www.youtube.com/watch?v=caGX6PoVneU https://www.youtube.com/watch?v=caGX6PoVneU

surajs 6y ago
Profound and enlightening, thanks!
trhway 6y ago
>locally anti-entropic processes. We do see liquids turn into regular solids after all: Crystallization is exothermic process. It releases heat into environment. It converts potential chemical energy into heat as result of bond making. Thus the total entropy of the system "crystal forming liquid + environment" is increased. Life is another famous process of local anti-entropy which is driven by the increase of the total entropy.

maxov 6y ago

Loved this article. There are so many applications of entropy and statistical physics in computer science, and I find it fascinating that the same general properties are useful in such different contexts. For example, there's a well-known phenomenon in probability called concentration of measure. One of the most important examples in computer science is if you flip n coins independently, then the number of heads concentrates very tightly. In particular, the probability that you are more than an epsilon-fraction away from 0.5n heads is at most around e^{-epsilon^2 n}. This is exactly the setting described in the article with the sheep in two pens, and this inequality is used in the design of many important randomized algorithms! A classic example is in load balancing, where random assignment sometimes produces 'nice' configurations that look very much like the high-entropy states described in this article (but unfortunately, many times random assignments don't behave very well, see e.g. the birthday paradox). The sum of independent random variables is well known to have concentration properties. An interesting question to me is what sorts of other statistics will exhibit these concentration phenomena. An important finding in this area is the bounded differences inequality (https://web.eecs.umich.edu/~cscott/past_courses/eecs598w14/notes/09_bounded_difference.pdf https://web.eecs.umich.edu/~cscott/past_courses/eecs598w14/n...), which generally states that any function that doesn't "depend too much" on each individual random argument (and the sum of bounded variables satisfies this assumption) exhibits the same concentration phenomenon. There are some applications in statistical learning, where we can bound the estimation error using certain learning complexity measures that rely on the bounded differences inequality. In the context of this article, that means there's a whole class of statistics that will concentrate similarly, and perhaps exhibit irreversibility at a macroscopic level.

pabo 6y ago
There's a related anecdote about John von Neumann: he used to joke that he has superpowers and can easily tell truly random and pseudo random sequences apart. He asked people to sit down in another room and generate a 0/1 sequence via coin flips, and record it. Then, generate another sequence by heart, trying to mimick randomness as much as possible. When people finally showed the two sequences to him, Neumann could instantly declare which one was which. People were amazed. The trick he used was based on the "burstiness" rule you describe: a long enough random sequence will likely contain a long homogeneous block. While humans tend to avoid long streaks of the same digit, as it does not feel random enough. So, all he did was he quickly checked with a glimpse, which of the two sequences contained the longest homogeneous block, and recognized that as the one generated via the coin flips.

andrewflnr 6y ago

So one thing I've never understood is how you can "count" microstates, or bits required to describe them, when the relevant physical parameters all seem to be real numbers. For instance, a gas of N atoms is described by 6N real numbers (3d position and velocity) regardless of how hot it is. The article talks about quanta of energy, but that seems like a simplification at best: a given interaction might be quantized, but quanta in general (e.g. photons) exist on a real-number spectrum, so it's possible to have an uncountable infinity of energy packet sizes right? (That's the point of a black body, right?) What am I missing? Is this a weird measure theory thing, where hot objects have even bigger uncountable infinities of states and we get rid them all with something like a change of variables? If you told me spacetime was secretly a cellular automaton I could deal with it, but real numbers ruin everything.

foota 6y ago
IANAP but there's two different things that come to mind. Even if the number of states were infinite, as long as there's a reasonable probability distribution you can know you can integrate over the probabilities that have some property vs another (say: solid vs not). Secondly, energy is quantized (on a very very very small level) in reality. Ymmv though, I tried googling this and read some stuff about waves being quantized and particles not and about how it depends on what system you're looking at and I had to drop out of quantum physics when I took it :-)
- kergonath 6y ago
  For quantum objects, the distinction between a wave and a particle is not very meaningful. The energy levels of an electron around a nucleus are discrete, regardless of whether the electron behaves more like a wave or more like a particle in the specific experiment you’re doing. Otherwise, you’re right: we count states by integrating (at times discrete, continuous, and often very complex) probability distributions.
keithlfrost 6y ago
This is an excellent, and puzzling, question! Let me try to provide some insight into how physicists think about such paradoxes, by addressing the specific example you mention, of the presumed uncountable infinity of different possible photon energies in a finite range of frequencies. In the case of blackbody radiation, when physicists analyze the set of possible photon energies more carefully, they find that there is really only an infinite number of different possible photon energies (for a finite range of frequencies) if the volume of space containing the photons is infinite. In any finite volume of space, if we allow ourselves to place boundary conditions on the electromagnetic fields at the edges of that volume (for example, suppose we think of our volume as a cube with mirrored walls), we find that there are only a finite number of oscillating modes of the electromagnetic field in any finite range of frequency. In the case of a cube, there is a very lowest frequency of radiation whose wavelength will allow it to form a standing wave in the box, and all the other allowed modes are multiples of that lowest frequency. So the density of possible photon states per unit of frequency is actually proportional to the volume of space we allow to hold the photons. (By the way, this is also precisely related to the quantum complementarity of uncertainty between momentum and position. To confine a photon to a volume of space, the uncertainty in its momentum must be the same order as that carried by the lowest frequency standing waves which would be compatible with the container.)

pawelmi 6y ago

Reminded me of Anna Karenina principle. https://en.m.wikipedia.org/wiki/Anna_Karenina_principle https://en.m.wikipedia.org/wiki/Anna_Karenina_principle

PaulDavisThe1st 6y ago

This was really great. It's how I was taught entropy at college (biophysics and molecular biology) though without the sheep. "Statistical Mechanics" was the name our professor used. The only tiny change I'd like to make is to add a line or two near the end, something along the following lines: There's a lot fewer ways to arrange water molecules so that they form an ice cube than there are to arrange them as a liquid. Most arrangements of water molecules look like a liquid, and so that's the likely endpoint even if they start arranged as an ice cube. The same is true of more or less any macroscopic object: the thing that we recognize and name ("chair", "table", "pen", "apple") requires the atoms to remain in one of a fairly small set of particular arrangements. Compared to the vast number of other arrangements of the same atoms ("dust"), the ones where the atoms form the "thing" are quite unlikely. Hence over time it's more likely that we'll find the atoms in one of the other ("random", or "dust-like") arrangements than the one we have a name for. The reason things "fall apart" isn't that there's some sort of preference for it - it's that there are vastly more ways for atoms to be in a "fallen apart" state than arranged as a "thing".

deleted 6y ago
[deleted]
kgwgk 6y ago
This kind of intuition can be dangerous: "Students who believe that spontaneous processes always yield greater disorder could be somewhat surprised when shown a demonstration of supercooled liquid water at many degrees below 00 C. The students have been taught that liquid water is disorderly compared to solid ice. When a seed of ice or a speck of dust is added, crystallization of some of the liquid is immediate. Orderly solid ice has spontaneously formed from the disorderly liquid. "Of course, thermal energy is evolved in the process of this thermodynamically metastable state changing to one that is stable. Energy is dispersed from the crystals, as they form, to the solution and thus the final temperature of the crystals of ice and liquid water are higher than originally. This, the instructor ordinarily would point out as a system-surroundings energy transfer. However, the dramatic visible result of this spontaneous process is in conflict with what the student has learned about the trend toward disorder as a test of spontaneity. "Such a picture might not take a thousand words of interpretation from an instructor to be correctly understood by a student, but they would not be needed at all if the misleading relation of disorder with entropy had not been mentioned." http://entropysite.oxy.edu/cracked_crutch.html http://entropysite.oxy.edu/cracked_crutch.html

jaspal747 6y ago

excuse my naivete, but do black holes help reduce entropy by capturing/engulfing things around them? Is that the cycle how universe keeps creating and recreating itself?

adeledeweylopez 6y ago
Nope, black holes have entropy proportional to the surface area of their event horizon. So the more stuff they engulf, the more their entropy increases, and thus they satisfy the 2nd law of thermodynamics just like everything else. It's a very interesting and active area of physics actually: https://en.wikipedia.org/wiki/Black_hole_thermodynamics https://en.wikipedia.org/wiki/Black_hole_thermodynamics
- WealthVsSurvive 6y ago
  Has this been observed, or is it more: here's some math that makes entropy even possible because the alternatives sound unlikely? I'm betting on the weird, things like the universe being generative on a macro scale and entropic in local timespace, wherein dark matter is merely newly created matter not in another universe, but in this universe, maybe some unknown interactions between the unstoppable force of expansion and the immovable object of a black hole's gravity. I'm probably blathering, I'm not a physicist.

layer8 6y ago

That still doesn’t answer the question how, if the laws of physics are time-symmetric, the universe as a whole can have a time-asymmetric evolution of entropy. I.e., if something forces entropy to increase in the long run, then that should hold in both directions of time. So what is it that causes entropy to only increase in the direction of the future, but not in the direction of the past, given that the laws pf physics do not distinguish between both directions?

kergonath 6y ago
We don’t have an answer to this question. I don’t want to discuss metaphysics here, but there is a very interesting discussion on that subject here: https://youtube.com/watch?v=-6rWqJhDv7M https://youtube.com/watch?v=-6rWqJhDv7M
- layer8 6y ago
  Let me rephrase maybe: Given the state of affairs I described above, I don’t understand what is the convincing argument that entropy does indeed increase in the long run. Any argument given should also work in the reverse direction, given the symmetry of time, shouldn’t it? (And thereby create a kind of reductio ad absurdum.) If not, why not?
  - 0134340 6y ago
    >symmetry of time Time isn't symmetric. "Since the second law of thermodynamics states that entropy increases as time flows toward the future, in general, the macroscopic universe does not show symmetry under time reversal." - https://en.wikipedia.org/wiki/T-symmetry https://en.wikipedia.org/wiki/T-symmetry
  - kergonath 6y ago
    It’s fundamentally an empirical law, even if one of the best-verified ones. I don’t think there really is a convincing argument, other than “we haven’t seen a single instance of the opposite being true”. It’s a missing piece in quantum mechanics (linked to decoherence) and quantum loop gravity (in which time is...interesting). The issue you mention (how can the 2nd law and the fact that the laws of Physics don’t care about the direction of time be true at the same time) is a big problem. I really recommend watching the video if you haven’t, it is fascinating and well worth the time.

bobsoap 6y ago

I love how the notion of entropy permeates into so many other things. It's fundamental, universal, and at the heart of nearly every aspect of our existence. Take philosophy. If the ultimate state of everything culminates in chaos (according to the theory of entropy), the human existence constitues the exact opposite: controlling the chaos that surrounds us, and shaping it into something useful and, in entropy-speak, progressively unprobable. Making ice cubes out of water. It follows that our existence can at least be described as a function of entropy. This describes life. Survival is the battle against entropy. Procreation is the chosen weapon against chaos, a force of order in a universe that can't help itself but fall into chaos -- and, undoubtedly, will ultimately prevail in that fight. In the long run, life will lose. Great article. Fascinating stuff.

antonios 6y ago
Alternatively, all that attempts to control chaos and decrease entropy actually results in faster entropy increase overall on a systemic level. I remember reading about a (Russian?) physicist that believed that life simply happens as a result of the universe's attempt to increase entropy faster on sufficiently complicated systems. If someone remembers his name I'd be obliged.
- 0134340 6y ago
  Dorion Sagan (Yes, Carl Sagan's son) also covers this in his book Into The Cool. That life is only a force multiplier in increasing it universally, basically.
imcoconut 6y ago
It's true. Have you read the book "What is Life?" By Schrodinger? He discusses the very concept you're talking about as he speculates on the physical substrate of genetic information (which was discovered to be DNA not long after the book was published).
combatentropy 6y ago
> Survival is the battle against entropy See my username ;) But my inspiration was from trying to make a difference within corporate culture, not raw survival. > controlling the chaos that surrounds us, and shaping it into something useful i.e., work > In the long run, life will lose. Unless there is a transcendent being beyond our universe, an explanation believed by many people for why, a long time ago, entropy was microscopic. The article admits it can't come up with a materialistic explanation and ends by calling it a "mystery".

rstuart4133 6y ago

That's remarkably well done, but the underlying principle can be summarised as "regression towards the mean" which is not so difficult to understand. Now explain this: when the sheep are cooped up in one box you can extract energy from the system. This becomes apparent if you imagine the boundary between the boxes is a fan. As the sheep move from what is a high pressure on one side to 0 pressure on the other they will move the fan. Attach a generator to the fan, and you get energy out. This configuration, with high pressure on the side low on the other, will happen every so often through random chance. Admittedly not very often, in fact so rarely it's useless to us. But in principle we have a perpetual motion machine. We don't, of course. But why not? And more to the point is the 2nd law nothing more than a statement of averages? By which I mean "order always tends towards disorder" is in fact false. We will every so often see order arise from disorder. So the 2nd law is not really a hard and fast law, any more then "you always loose when gambling at the casino" is a hard and fast law.

deleted 6y ago
[deleted]

boyobo 6y ago

Great article. Can energy be explained in a similarly simple fashion? I'm pretty comfortable with probability theory so this entroppy explaination makes sense, but I still don't understand what energy is. Also, who determines what a 'macroscopic variable' is? Why do there only seem to be 3 for gassess (V, T, P?)

petschge 6y ago
There is more macroscopic variables, for example the three components of the bulk flow velocity or the density of the gas.
- boyobo 6y ago
  But are those the only 6?

leafboi 6y ago

https://www.youtube.com/watch?v=wI-qAxKJoSU https://www.youtube.com/watch?v=wI-qAxKJoSU In the video above you will witness a metal wire in a disorderly shape spontaneously form into an organized spring shape when entropy is increased (the wire is heated). There are no special effects or video rewinding trickery here, the phenomenon is very real. It is as if you were looking at water spontaneously form into ice cubes but entropy is NOT reversing! It is moving forward. The video is a really good example of entropy. It's good in the sense that if you understand why entropy is increasing when the metal is heated than you truly understand what entropy is... as the video gets rid of the notion of order and disorder all together. That's right. There are many cases of increasing entropy where the system spontaneously progresses from a greater disorder to more organization. Many people say that the when some system becomes more organized that means entropy is leaving the local system and increasing overall in the global system. This is not what's happening here. The wire is being heated. Atoms are becoming more organized and less disorderly by virtue of MORE entropy entering the system. If you understand this concept then you truly understand entropy. If not you still don't get it.

aoowii 6y ago

Order as humans understand it is different from physical order. We see order as an array of ascending numbers or a house of cards. The universe sees order more like a state of 'useful' energy, where it's possible to extract it, and seemingly became ordered by putting energy in. This is why I think the information theory definition of entropy is a bit misleading. I like to see it as a radioactive atom, which starts as a useful structure with potential energy that has an inherent timer until this energy is lost due to the universe wanting to return to equilibrium. So it's statistically extremely unlikely to get the atom back to its original energetic state by nature, but not because because it's literally impossible, it's just impossible to do this without putting energy back in and that's not something that happens naturally. A reversal of entropy is entirely possible, it's just called doing work. Of course, we don't have the knowledge necessary to reverse each individual atom in the melting process of an ice cube, but one day we might. It's theoretically possible with enough work. Of course, there will always be a loss of energy, but i'm pretty sure that's an entirely different thermodynamic law.

roimor 6y ago

I don't really understand this "Entropy Is All About Arrangements" takeaway. A fair coin has higher entropy than a biased one. What are the "arrangements" in this case?

kgwgk 6y ago
The coins don't have entropy, the sequences they produce do (in information theory sense, this is not about physics). For a long sequence (say 1000), the former will produce close to 500 heads and 500 tails. The latter, assuming one heads are three times as probable as tails, will produce around 750 heads and 250 tails. There are many more different sequences of the first kind.
- roimor 6y ago
  I'm referring to the entropy of the Bernoulli distribution. If the coin is fair, the entropy is 1 bit... if the coin isn't fair, then the entropy of the distribution is less than 1 bit. I'm having trouble reconciling the information theory way of thinking about entropy as a function of a distribution, with how physicists tend to think of entropy of arrangements.
  - deleted 6y ago
    [deleted]
  - kgwgk 6y ago
    There is a relationship between the distribution of x_i and the sequences generated from that distribution x_1, x_2, ..., x_n. If the coin isn't fair there are less arrangements possible. If the coin has two heads there is only one possible sequence.

ba0bab 6y ago

great explanation and visuals. but I do not quite get the way the arrangements of sheep are treated. They seem to be counted in the standard "Unordered Sampling with Replacement" fashion. >Just as the sheep wander about the plots of land in the farm, these packets of energy randomly shuffle among the atoms in the solid. this would mean that any "packet" of energy is equally likely to be in any of the buckets and the "packets" are independent of each other. so we have an equal distribution on the product space with 6^6 elements. but then later > Now, let’s assume the sheep are equally likely to be in any of these 462 arrangements. (Since they move randomly, there's no reason to prefer one arrangement over another.) under the prior assumptions these arrangements would not be equally likely. e.g. "all sheeps in plot 1" would be far less likely than "each sheep in a different plot" am I missing something here? in any case the same conclusions can be drawn in both cases, only that the concentration around 3 is already more pronounced in the "6 sheep, 6 plots" case using the product space model.

emdubbelyou 6y ago
I think the confusion is in the way that sheep as a word can be both plural and singular. Specifically, one sheep is as likely to be in any single spot compared to any other single spot. It’s when you get to more than one sheep that you see the distributions