14 ms·

What Is Entropy?

345 points by ainoobler 2y ago

Jun8 2y ago
A well known anecdote reported by Shannon: "My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'" See the answers to this MathOverflow SE question (https://mathoverflow.net/questions/403036/john-von-neumanns-remark-on-entropy https://mathoverflow.net/questions/403036/john-von-neumanns-...) for references on the discussion whether Shannon's entropy is the same as the one from thermodynamics.
- BigParm 2y ago
  Von Neumann was the king of kings
  - tonetegeatinst 2y ago
    Its odd...as someone interested but not fully into the sciences I see his name pop up everywhere.
    - otteromkram 2y ago
      [flagged]
    - farias0 2y ago
      I've seen many people arguing he's the most intelligent person that ever lived
      - wrycoder 2y ago
        Some say Hungarians are actually aliens.
        jack_pp 2y ago
        https://slatestarcodex.com/2017/05/26/the-atomic-bomb-considered-as-hungarian-high-school-science-fair-project/ https://slatestarcodex.com/2017/05/26/the-atomic-bomb-consid...
    - bee_rider 2y ago
      He was really brilliant, made contributions all over the place in the math/physics/tech field, and had a sort of wild and quirky personality that people love telling stories about. A funny quote about him from a Edward “a guy with multiple equations named after him” Teller: > Edward Teller observed "von Neumann would carry on a conversation with my 3-year-old son, and the two of them would talk as equals, and I sometimes wondered if he used the same principle when he talked to the rest of us."
      - strogonoff 2y ago
        Are there many von-Neumann-like multidisciplinaries nowadays? It feels like unless one is razor sharp fully into one field one is not to be treated seriously by those who made careers in it (and who have the last word on it).
        i_am_proteus 2y ago
        There have been a very small number of thinkers as publicly accomplished as von Neumann ever. One other who comes to mind is Carl F. Gauss.
        strogonoff 2y ago
        Is it fair to say that the number of publicly accomplished multidisciplinaries alive at a particular moment is not rising as it may be expected, proportionally to the total number of suitably educated people?
        djd3 2y ago
        Euler. JVM was one of the smartest ever, but Euler was there centuries before and shows up in so many places. If I had a Time Machine I'd love to get those two together for a stiff drink and a banter.
        passion__desire 2y ago
        Genius Edward Teller Describes 1950s Genius John Von Neumann https://youtu.be/Oh31I1F2vds?t=189 https://youtu.be/Oh31I1F2vds?t=189 Describes Von Neumann's final days struggle when he couldn't think. Thinking, an activity which he loved the most.
    - rramadass 2y ago
      An Introduction here : https://www.youtube.com/watch?v=IPMjVcLiNKc https://www.youtube.com/watch?v=IPMjVcLiNKc
    - complaintdept 2y ago
      Even mortals such as ourselves can apply some of Von Neumann's ideas in our everyday lives: https://en.m.wikipedia.org/wiki/Fair_coin#Fair_results_from_a_biased_coin https://en.m.wikipedia.org/wiki/Fair_coin#Fair_results_from_...
  - vinnyvichy 2y ago
    So much so, he has his own entropy! https://en.wikipedia.org/wiki/Von_Neumann_entropy https://en.wikipedia.org/wiki/Von_Neumann_entropy
  - penguin_booze 2y ago
    He's a certified Martian: https://en.wikipedia.org/wiki/The_Martians_(scientists) https://en.wikipedia.org/wiki/The_Martians_(scientists).
    - khana 2y ago
      [dead]
    - zeristor 2y ago
      I was hoping the Wikipedia might explain why this might have been.
      - cubefox 2y ago
        https://emilkirkegaard.dk/en/2022/11/a-theory-of-ashkenazi-genius-intelligence-and-mental-illness/ https://emilkirkegaard.dk/en/2022/11/a-theory-of-ashkenazi-g...
        bglazer 2y ago
        Emil Kirkegaard is a self-described white nationalist eugenicist who thinks the age of consent is too high. I wouldn't trust anything he has to say.
        YeGoblynQueenne 2y ago
        No need for ad hominems. This suffices to place doubt on the article's premises (and therefore any conclusion): >> This hasn’t been strictly shown mathematically, but I think it is true.
        cubefox 2y ago
        > Emil Kirkegaard is a self-described white nationalist That's simply a lie. > who thinks the age of consent is too high Too high in which country? Such laws vary strongly, even by US state, and he is from Denmark. Anyway, this has nothing to do with the topic at hand.
  - topologie 2y ago
    I disagree... Von Neumann went beyond being a King of Kings, the man was a God (or a "Monster Mind" according to Feynman) :)
dekhn 2y ago
I really liked the approach my stat mech teacher used. In nearly all situations, entropy just ends up being the log of the number of ways a system can be arranged (https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula) although I found it easiest to think in terms of pairs of dice rolls.
- petsfed 2y ago
  And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties. Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.
  - enugu 2y ago
    Assuming each of the N microstates for a given macrostate are equally possible with probability p=1/N, the Shannon Entropy is -Σp.log(p) = -N.p.log(p)=-1.log(1/N)=log(N), which is the physics interpretation. In the continuous version, you would get log(V) where V is the volume in phase space occupied by the microstates for a given macrostate. Liouville's theorem that the volume is conserved in phase space implies that any macroscopic process can only move all the microstates from a macrostate A into a macrostate B only if the volume of B is bigger than the volume of A. This implies that the entropy of B should be bigger than the entropy of A which is the Second Law.
    - cubefox 2y ago
      The second law of thermodynamics is time-asymmetric, but the fundamental physical laws are time-symmetric, so from them you can only predict that the entropy of B should be bigger than the entropy of A irrespective of whether B is in the future or the past of A. You need the additional assumption (Past Hypothesis) that the universe started in a low entropy state in order to get the second law of thermodynamics. > If our goal is to predict the future, it suffices to choose a distribution that is uniform in the Liouville measure given to us by classical mechanics (or its quantum analogue). If we want to reconstruct the past, in contrast, we need to conditionalize over trajectories that also started in a low-entropy past state — that the “Past Hypothesis” that is required to get stat mech off the ground in a world governed by time-symmetric fundamental laws. https://www.preposterousuniverse.com/blog/2013/07/09/cosmology-and-the-past-hypothesis/ https://www.preposterousuniverse.com/blog/2013/07/09/cosmolo...
Tomte 2y ago
PBS Spacetime‘s entropy playlist: https://youtube.com/playlist?list=PLsPUh22kYmNCzNFNDwxIug8q1Zz0Mj60H&si=y2XIvxDJOFyCpfIu https://youtube.com/playlist?list=PLsPUh22kYmNCzNFNDwxIug8q1...
- foobarian 2y ago
  A bit off-color but classic: https://www.youtube.com/watch?v=wgltMtf1JhY https://www.youtube.com/watch?v=wgltMtf1JhY
drojas 2y ago
My definition: Entropy is a measure of the accumulation of non-reversible energy transfers. Side note: All reversible energy transfers involve an increase in potential energy. All non-reversible energy transfers involve a decrease in potential energy.
- snarkconjecture 2y ago
  That definition doesn't work well because you can have changes in entropy even if no energy is transferred, e.g. by exchanging some other conserved quantity. The side note is wrong in letter and spirit; turning potential energy into heat is one way for something to be irreversible, but neither of those statements is true. For example, consider an iron ball being thrown sideways. It hits a pile of sand and stops. The iron ball is not affected structurally, but its kinetic energy is transferred (almost entirely) to heat energy. If the ball is thrown slightly upwards, potential energy increases but the process is still irreversible. Also, the changes of potential energy in corresponding parts of two Carnot cycles are directionally the same, even if one is ideal (reversible) and one is not (irreversible).
- deleted 2y ago
  [deleted]
- space_oddity 2y ago
  However, while your definition effectively captures a significant aspect of entropy, it might be somewhat limited in scope
ooterness 2y ago
For information theory, I've always thought of entropy as follows: "If you had a really smart compression algorithm, how many bits would it take to accurately represent this file?" i.e., Highly repetitive inputs compress well because they don't have much entropy per bit. Modern compression algorithms are good enough on most data to be used as a reasonable approximation for the true entropy.
- space_oddity 2y ago
  The essence of entropy as a measure of information content
glial 2y ago
I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each have received different information that about X. H(X) should be H_{observer}(X), or even better, H_{observer, time}(X). As clear as Shannon's work is in other respects, he glosses over this.
- JumpCrisscross 2y ago
  > it's a subjective quantity -- a property of the observer, not the observed Shannon's entropy is a property of the source-channel-receiver system.
  - glial 2y ago
    Can you explain this in more detail? Entropy is calculated as a function of a probability distribution over possible messages or symbols. The sender might have a distribution P over possible symbols, and the receiver might have another distribution Q over possible symbols. Then the "true" distribution over possible symbols might be another distribution yet, call it R. The mismatch between these is what leads to various inefficiencies in coding, decoding, etc [1]. But both P and Q are beliefs about R -- that is, they are properties of observers. [1] https://en.wikipedia.org/wiki/Kullback–Leibler_divergence#Coding https://en.wikipedia.org/wiki/Kullback–Leibler_divergence#Co...
- rachofsunshine 2y ago
  This doesn't really make entropy itself observer dependent. (Shannon) entropy is a property of a distribution. It's just that when you're measuring different observers' beliefs, you're looking at different distributions (which can have different entropies the same way they can have different means, variances, etc).
  - mitthrowaway2 2y ago
niemandhier 2y ago
My goto source for understanding entropy: http://philsci-archive.pitt.edu/8592/1/EntropyPaperFinal.pdf http://philsci-archive.pitt.edu/8592/1/EntropyPaperFinal.pdf
prof-dr-ir 2y ago
If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probability distributions. (Elementary exercises: show that S >= 0 and it is maximized on the uniform distribution.) That is it. I think there is little need for all the mystery.
- kgwgk 2y ago
  That covers one and a half of the twelve points he discusses.
  - prof-dr-ir 2y ago
    Correct! And it took me just one paragraph, not the 18 pages of meandering (and I think confusing) text that it takes the author of the pdf to introduce the same idea.
    - kgwgk 2y ago
      You didn’t introduce any idea. You said it’s “just a number” and wrote down a formula without any explanation or justification. I concede that it was much shorter though. Well done!
      - bdjsiqoocwk 2y ago
        Haha you reminded me of that idea in software engineering that "it's easy to make an algorithm faster if you accept that at times it might output the wrong result; in fact you can make infinitely fast"
- rachofsunshine 2y ago
  The problem is that this doesn't get at many of the intuitive properties of entropy. A different explanation (based on macro- and micro-states) makes it intuitively obvious why entropy is non-decreasing with time or, with a little more depth, what entropy has to do with temperature.
eointierney 2y ago
Ah JCB, how I love your writing, you are always so very generous. Your This Week's Finds were a hugely enjoyable part of my undergraduate education and beyond. Thank you again.
dmn322 2y ago
This seems like a great resource for referencing the various definitions. I've tried my hand at developing an intuitive understanding: https://spacechimplives.substack.com/p/observers-and-entropy https://spacechimplives.substack.com/p/observers-and-entropy. TLDR - it's an artifact of the model we're using. In the thermodynamic definition, the energy accounted for in the terms of our model is information. The energy that's not is entropic energy. Hence why it's not "useable" energy, and the process isn't reversible.
deleted 2y ago
[deleted]
zoenolan 2y ago
Hawking on the subject https://youtu.be/wgltMtf1JhY https://youtu.be/wgltMtf1JhY
bdjsiqoocwk 2y ago
Hmmm that list of things that contribute to entropy I've noticed omits particles which under "normal circumstances" on earth exist in bound states, for example it doesn't mentions W bosons or gluons. But in some parts of the universe they're not bound but in different state of matter, e.g. quark gluon plasma. I wonder how or if this was taken I to account.
yellowcake0 2y ago
Information entropy is literally the strict lower bound on how efficiently information can be communicated (expected number of transmitted bits) if the probability distribution which generates this information is known, that's it. Even in contexts such as calculating the information entropy of a bit string, or the English language, you're just taking this data and constructing some empirical probability distribution from it using the relative frequencies of zeros and ones or letters or n-grams or whatever, and then calculating the entropy of that distribution. I can't say I'm overly fond of Baez's definition, but far be it from me to question someone of his stature.
arjunlol 2y ago
ΔS = ΔQ/T
utkarsh858 2y ago
I sometimes ponder where new entropy/randomness is coming from, like if we take the earliest state of universe as an infinitely dense point particle which expanded. So there must be some randomness or say variety which led it to expand in a non uniform way which led to the dominance of matter over anti-matter, or creation of galaxies, clusters etc. If we take an isolated system in which certain static particles are present, will there be the case that a small subset of the particles will get motion and this introduce entropy? Can entropy be induced automatically, atleast on a quantum level? If anyone can help me explain that it will be very helpful and thus can help explain origin of universe in a better way.
- pseidemann 2y ago
  I saw this video, which explained it for me (it's german, maybe the automatic subtitles will work for you): https://www.youtube.com/watch?v=hrJViSH6Klo https://www.youtube.com/watch?v=hrJViSH6Klo He argues that the randomness you are looking for comes from quantum fluctuations, and if this randomness did not exist, the universe would probably never have "happened".
  - utkarsh858 2y ago
    Thanks for the reference will take some time before I see the whole video. Can you tell me what those quantum fluctuations are in short? Are they part of some physical law?
- empath75 2y ago
  Symmetry breaking is the general phenomenon that underlies most of that. The classic example is this: Imagine you have a perfectly symmetrical sombrero[1], and there's a ball balanced on top of the middle of the hat. There's no preferred direction it should fall in, but it's _unstable_. Any perturbation will make it roll down hill and come to rest in a stable configuration on the brim of the hat. The symmetry of the original configuration is now broken, but it's stable. 1: https://m.media-amazon.com/images/I/61M0LFKjI9L.__AC_SX300_SY300_QL70_FMwebp_.jpg https://m.media-amazon.com/images/I/61M0LFKjI9L.__AC_SX300_S...
tasteslikenoise 2y ago
I've always favored this down-to-earth characterization of the entropy of a discrete probability distribution. (I'm a big fan of John Baez's writing, but I was surprised glancing through the PDF to find that he doesn't seem to mention this viewpoint.) Think of the distribution as a histogram over some bins. Then, the entropy is a measurement of, if I throw many many balls at random into those bins, the probability that the distribution of balls over bins ends up looking like that histogram. What you usually expect to see is a uniform distribution of balls over bins, so the entropy measures the probability of other rare events (in the language of probability theory, "large deviations" from that typical behavior). More specifically, if P = (P1, ..., Pk) is some distribution, then the probability that throwing N balls (for N very large) gives a histogram looking like P is about 2^(-N * [log(k) - H(P)]), where H(P) is the entropy. When P is the uniform distribution, then H(P) = log(k), the exponent is zero, and the estimate is 1, which says that by far the most likely histogram is the uniform one. That is the largest possible entropy, so any other histogram has probability 2^(-c*N) of appearing for some c > 0, i.e., is very unlikely and exponentially moreso the more balls we throw, but the entropy measures just how much. "Less uniform" distributions are less likely, so the entropy also measures a certain notion of uniformity. In large deviations theory this specific claim is called "Sanov's theorem" and the role the entropy plays is that of a "rate function." The counting interpretation of entropy that some people are talking about is related, at least at a high level, because the probability in Sanov's theorem is the number of outcomes that "look like P" divided by the total number, so the numerator there is indeed counting the number of configurations (in this case of balls and bins) having a particular property (in this case looking like P). There are lots of equivalent definitions and they have different virtues, generalizations, etc, but I find this one especially helpful for dispelling the air of mystery around entropy.
- vinnyvichy 2y ago
  Hey did you want to say relative entropy ~ rate function ~ KL divergence. Might be more familiar to ML enthusiasts here, get them to be curious about Sanov or large deviations.
  - tasteslikenoise 2y ago
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
vinnyvichy 2y ago
The book might disappoint some.. >I have largely avoided the second law of thermodynamics ... Thus, the aspects of entropy most beloved by physics popularizers will not be found here. But personally, this bit is the most exciting to me. >I have tried to say as little as possible about quantum mechanics, to keep the physics prerequisites low. However, Planck’s constant shows up in the formulas for the entropy of the three classical systems mentioned above. The reason for this is fascinating: Planck’s constant provides a unit of volume in position-momentum space, which is necessary to define the entropy of these systems. Thus, we need a tiny bit of quantum mechanics to get a good approximate formula for the entropy of hydrogen, even if we are trying our best to treat this gas classically.
foobarbecue 2y ago
How do you get to the actual book / tweets? The link just takes me back to the forward...
- vishnugupta 2y ago
  http://math.ucr.edu/home/baez/what_is_entropy.pdf http://math.ucr.edu/home/baez/what_is_entropy.pdf
GoblinSlayer 2y ago
There's fundamental nature of entropy, but as usual it's not very enlightening for poor monkey brain, so to explain you need to enumerate all its high level behavior, but its high level behavior is accidental and can't be summarized in a concise form.
- space_oddity 2y ago
  This complexity underscores the richness of the concept
  - GoblinSlayer 2y ago
    I'd say it underscores its accidental nature.
ctafur 2y ago
The way I understand it is with an analogy to probability. To me, events are to microscopic states like random variable is to entropy.
- ctafur 2y ago
  My first contact with entropy was in chemistry and thermodynamics and I didn't get it. Actually I didn't get anything from engineering thermodynamics books such as Çengel and so. You must go to statistical mechanics or information theory to understand entropy. Or trying these PRICELESS NOTES from Prof. Suo: https://docs.google.com/document/d/1UMwpoDRZLlawWlL2Dz6YEomyhSNveQHp6pt7AUiP37Q/edit https://docs.google.com/document/d/1UMwpoDRZLlawWlL2Dz6YEomy...
jsomedon 2y ago
Am I only one that can't download the pdf, or is the file server down? I can see the blog page but when I try downloading the ebook it just doesn't work.. If the file server is down.. anyone could upload the ebook for download?
tromp 2y ago
Closely related recent discussion: https://news.ycombinator.com/item?id=40972589 https://news.ycombinator.com/item?id=40972589
tromp 2y ago
Closely related recent discussion on The Second Law of Thermodynamics (2011) (franklambert.net): https://news.ycombinator.com/item?id=40972589 https://news.ycombinator.com/item?id=40972589
ThrowawayTestr 2y ago
MC Hawking already explained this https://youtu.be/wgltMtf1JhY https://youtu.be/wgltMtf1JhY
ccosm 2y ago
"I have largely avoided the second law of thermodynamics, which says that entropy always increases. While fascinating, this is so problematic that a good explanation would require another book!" For those interested I am currently reading "Entropy Demystified" by Arieh Ben-Naim which tackles this side of things from much the same direction.
suoduandao3 2y ago
I like the formulation of 'the amount of information we don't know about a system that we could in theory learn'. I'm surprised there's no mention of the Copenhagen interpretation's interaction with this definition, under a lot of QM theories 'unavailable information' is different from available information.
tsoukase 2y ago
After years of thought I dare to say the 2nd TL is a tautology. Entropy is increasing means every system tends to higher probability means the most probable is the most probable.
- tel 2y ago
  I think that’s right, though it’s non-obvious that more probable systems are disordered. At least as non-obvious as Pascal’s triangle is. Which is to say, worth saying from a first principles POV, but not all that startling.
hjfjh 2y ago
[flagged]
illuminant 2y ago
Entropy is the distribution of potential over negative potential. This could be said "the distribution of what ever may be over the surface area of where it may be." This is erroneously taught in conventional information theory as "the number of configurations in a system" or the available information that has yet to be retrieved. Entropy includes the unforseen, and out of scope. Entropy is merely the predisposition to flow from high to low pressure (potential). That is it. Information is a form of potential. Philosophically what are entropy's guarantees? - That there will always be a super-scope, which may interfere in ways unanticipated; - everything decays the only mystery is when and how.
- mwbajor 2y ago
  All definitions of entropy stem from one central, universal definition: Entropy is the amount of energy unable to be used for useful work. Or better put grammatically: entropy describes the effect that not all energy consumed can be used for work.
  - ajkjk 2y ago
    There's a good case to be made that the information-theoretic definition of entropy is the most fundamental one, and the version that shows up in physics is just that concept as applied to physics.
    - rimunroe 2y ago
      My favorite course I took as part of my physics degree was statistical mechanics. It leaned way closer to information theory than I would have expected going in, but in retrospect should have been obvious. Unrelated: my favorite bit from any physics book is probably still the introduction of the first chapter of "States of Matter" by David Goodstein: "Ludwig Boltzmann, who spent much of his life studying statistical mechanics, died in 1906, by his own hand. Paul Ehrenfest, carrying on the work, died similarly in 1933. Now it is our turn to study statistical mechanics."
    - galaxyLogic 2y ago

345 points by ainoobler 2y ago

Jun8 2y ago

A well known anecdote reported by Shannon: "My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'" See the answers to this MathOverflow SE question (https://mathoverflow.net/questions/403036/john-von-neumanns-remark-on-entropy https://mathoverflow.net/questions/403036/john-von-neumanns-...) for references on the discussion whether Shannon's entropy is the same as the one from thermodynamics.

BigParm 2y ago
Von Neumann was the king of kings
- tonetegeatinst 2y ago
  Its odd...as someone interested but not fully into the sciences I see his name pop up everywhere.
  - otteromkram 2y ago
    [flagged]
  - farias0 2y ago
    I've seen many people arguing he's the most intelligent person that ever lived
    - wrycoder 2y ago
      Some say Hungarians are actually aliens.
      - jack_pp 2y ago
        https://slatestarcodex.com/2017/05/26/the-atomic-bomb-considered-as-hungarian-high-school-science-fair-project/ https://slatestarcodex.com/2017/05/26/the-atomic-bomb-consid...
  - bee_rider 2y ago
    He was really brilliant, made contributions all over the place in the math/physics/tech field, and had a sort of wild and quirky personality that people love telling stories about. A funny quote about him from a Edward “a guy with multiple equations named after him” Teller: > Edward Teller observed "von Neumann would carry on a conversation with my 3-year-old son, and the two of them would talk as equals, and I sometimes wondered if he used the same principle when he talked to the rest of us."
    - strogonoff 2y ago
      Are there many von-Neumann-like multidisciplinaries nowadays? It feels like unless one is razor sharp fully into one field one is not to be treated seriously by those who made careers in it (and who have the last word on it).
      - i_am_proteus 2y ago
        There have been a very small number of thinkers as publicly accomplished as von Neumann ever. One other who comes to mind is Carl F. Gauss.
        strogonoff 2y ago
        Is it fair to say that the number of publicly accomplished multidisciplinaries alive at a particular moment is not rising as it may be expected, proportionally to the total number of suitably educated people?
        djd3 2y ago
        Euler. JVM was one of the smartest ever, but Euler was there centuries before and shows up in so many places. If I had a Time Machine I'd love to get those two together for a stiff drink and a banter.
        passion__desire 2y ago
        Genius Edward Teller Describes 1950s Genius John Von Neumann https://youtu.be/Oh31I1F2vds?t=189 https://youtu.be/Oh31I1F2vds?t=189 Describes Von Neumann's final days struggle when he couldn't think. Thinking, an activity which he loved the most.
  - rramadass 2y ago
    An Introduction here : https://www.youtube.com/watch?v=IPMjVcLiNKc https://www.youtube.com/watch?v=IPMjVcLiNKc
  - complaintdept 2y ago
    Even mortals such as ourselves can apply some of Von Neumann's ideas in our everyday lives: https://en.m.wikipedia.org/wiki/Fair_coin#Fair_results_from_a_biased_coin https://en.m.wikipedia.org/wiki/Fair_coin#Fair_results_from_...
- vinnyvichy 2y ago
  So much so, he has his own entropy! https://en.wikipedia.org/wiki/Von_Neumann_entropy https://en.wikipedia.org/wiki/Von_Neumann_entropy
- penguin_booze 2y ago
  He's a certified Martian: https://en.wikipedia.org/wiki/The_Martians_(scientists) https://en.wikipedia.org/wiki/The_Martians_(scientists).
  - khana 2y ago
    [dead]
  - zeristor 2y ago
    I was hoping the Wikipedia might explain why this might have been.
    - cubefox 2y ago
      https://emilkirkegaard.dk/en/2022/11/a-theory-of-ashkenazi-genius-intelligence-and-mental-illness/ https://emilkirkegaard.dk/en/2022/11/a-theory-of-ashkenazi-g...
      - bglazer 2y ago
        Emil Kirkegaard is a self-described white nationalist eugenicist who thinks the age of consent is too high. I wouldn't trust anything he has to say.
        YeGoblynQueenne 2y ago
        No need for ad hominems. This suffices to place doubt on the article's premises (and therefore any conclusion): >> This hasn’t been strictly shown mathematically, but I think it is true.
        cubefox 2y ago
        > Emil Kirkegaard is a self-described white nationalist That's simply a lie. > who thinks the age of consent is too high Too high in which country? Such laws vary strongly, even by US state, and he is from Denmark. Anyway, this has nothing to do with the topic at hand.
- topologie 2y ago
  I disagree... Von Neumann went beyond being a King of Kings, the man was a God (or a "Monster Mind" according to Feynman) :)

dekhn 2y ago

I really liked the approach my stat mech teacher used. In nearly all situations, entropy just ends up being the log of the number of ways a system can be arranged (https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula) although I found it easiest to think in terms of pairs of dice rolls.

petsfed 2y ago
And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties. Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.
- enugu 2y ago
  Assuming each of the N microstates for a given macrostate are equally possible with probability p=1/N, the Shannon Entropy is -Σp.log(p) = -N.p.log(p)=-1.log(1/N)=log(N), which is the physics interpretation. In the continuous version, you would get log(V) where V is the volume in phase space occupied by the microstates for a given macrostate. Liouville's theorem that the volume is conserved in phase space implies that any macroscopic process can only move all the microstates from a macrostate A into a macrostate B only if the volume of B is bigger than the volume of A. This implies that the entropy of B should be bigger than the entropy of A which is the Second Law.
  - cubefox 2y ago
    The second law of thermodynamics is time-asymmetric, but the fundamental physical laws are time-symmetric, so from them you can only predict that the entropy of B should be bigger than the entropy of A irrespective of whether B is in the future or the past of A. You need the additional assumption (Past Hypothesis) that the universe started in a low entropy state in order to get the second law of thermodynamics. > If our goal is to predict the future, it suffices to choose a distribution that is uniform in the Liouville measure given to us by classical mechanics (or its quantum analogue). If we want to reconstruct the past, in contrast, we need to conditionalize over trajectories that also started in a low-entropy past state — that the “Past Hypothesis” that is required to get stat mech off the ground in a world governed by time-symmetric fundamental laws. https://www.preposterousuniverse.com/blog/2013/07/09/cosmology-and-the-past-hypothesis/ https://www.preposterousuniverse.com/blog/2013/07/09/cosmolo...

Tomte 2y ago

PBS Spacetime‘s entropy playlist: https://youtube.com/playlist?list=PLsPUh22kYmNCzNFNDwxIug8q1Zz0Mj60H&si=y2XIvxDJOFyCpfIu https://youtube.com/playlist?list=PLsPUh22kYmNCzNFNDwxIug8q1...

foobarian 2y ago
A bit off-color but classic: https://www.youtube.com/watch?v=wgltMtf1JhY https://www.youtube.com/watch?v=wgltMtf1JhY

drojas 2y ago

My definition: Entropy is a measure of the accumulation of non-reversible energy transfers. Side note: All reversible energy transfers involve an increase in potential energy. All non-reversible energy transfers involve a decrease in potential energy.

snarkconjecture 2y ago
That definition doesn't work well because you can have changes in entropy even if no energy is transferred, e.g. by exchanging some other conserved quantity. The side note is wrong in letter and spirit; turning potential energy into heat is one way for something to be irreversible, but neither of those statements is true. For example, consider an iron ball being thrown sideways. It hits a pile of sand and stops. The iron ball is not affected structurally, but its kinetic energy is transferred (almost entirely) to heat energy. If the ball is thrown slightly upwards, potential energy increases but the process is still irreversible. Also, the changes of potential energy in corresponding parts of two Carnot cycles are directionally the same, even if one is ideal (reversible) and one is not (irreversible).
deleted 2y ago
[deleted]
space_oddity 2y ago
However, while your definition effectively captures a significant aspect of entropy, it might be somewhat limited in scope

ooterness 2y ago

For information theory, I've always thought of entropy as follows: "If you had a really smart compression algorithm, how many bits would it take to accurately represent this file?" i.e., Highly repetitive inputs compress well because they don't have much entropy per bit. Modern compression algorithms are good enough on most data to be used as a reasonable approximation for the true entropy.

space_oddity 2y ago
The essence of entropy as a measure of information content

glial 2y ago

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each have received different information that about X. H(X) should be H_{observer}(X), or even better, H_{observer, time}(X). As clear as Shannon's work is in other respects, he glosses over this.

JumpCrisscross 2y ago
> it's a subjective quantity -- a property of the observer, not the observed Shannon's entropy is a property of the source-channel-receiver system.
- glial 2y ago
  Can you explain this in more detail? Entropy is calculated as a function of a probability distribution over possible messages or symbols. The sender might have a distribution P over possible symbols, and the receiver might have another distribution Q over possible symbols. Then the "true" distribution over possible symbols might be another distribution yet, call it R. The mismatch between these is what leads to various inefficiencies in coding, decoding, etc [1]. But both P and Q are beliefs about R -- that is, they are properties of observers. [1] https://en.wikipedia.org/wiki/Kullback–Leibler_divergence#Coding https://en.wikipedia.org/wiki/Kullback–Leibler_divergence#Co...
rachofsunshine 2y ago
This doesn't really make entropy itself observer dependent. (Shannon) entropy is a property of a distribution. It's just that when you're measuring different observers' beliefs, you're looking at different distributions (which can have different entropies the same way they can have different means, variances, etc).
- mitthrowaway2 2y ago

niemandhier 2y ago

My goto source for understanding entropy: http://philsci-archive.pitt.edu/8592/1/EntropyPaperFinal.pdf http://philsci-archive.pitt.edu/8592/1/EntropyPaperFinal.pdf

prof-dr-ir 2y ago

If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probability distributions. (Elementary exercises: show that S >= 0 and it is maximized on the uniform distribution.) That is it. I think there is little need for all the mystery.

kgwgk 2y ago
That covers one and a half of the twelve points he discusses.
- prof-dr-ir 2y ago
  Correct! And it took me just one paragraph, not the 18 pages of meandering (and I think confusing) text that it takes the author of the pdf to introduce the same idea.
  - kgwgk 2y ago
    You didn’t introduce any idea. You said it’s “just a number” and wrote down a formula without any explanation or justification. I concede that it was much shorter though. Well done!
    - bdjsiqoocwk 2y ago
      Haha you reminded me of that idea in software engineering that "it's easy to make an algorithm faster if you accept that at times it might output the wrong result; in fact you can make infinitely fast"
rachofsunshine 2y ago
The problem is that this doesn't get at many of the intuitive properties of entropy. A different explanation (based on macro- and micro-states) makes it intuitively obvious why entropy is non-decreasing with time or, with a little more depth, what entropy has to do with temperature.

eointierney 2y ago

Ah JCB, how I love your writing, you are always so very generous. Your This Week's Finds were a hugely enjoyable part of my undergraduate education and beyond. Thank you again.

dmn322 2y ago

This seems like a great resource for referencing the various definitions. I've tried my hand at developing an intuitive understanding: https://spacechimplives.substack.com/p/observers-and-entropy https://spacechimplives.substack.com/p/observers-and-entropy. TLDR - it's an artifact of the model we're using. In the thermodynamic definition, the energy accounted for in the terms of our model is information. The energy that's not is entropic energy. Hence why it's not "useable" energy, and the process isn't reversible.

deleted 2y ago

[deleted]

zoenolan 2y ago

Hawking on the subject https://youtu.be/wgltMtf1JhY https://youtu.be/wgltMtf1JhY

bdjsiqoocwk 2y ago

Hmmm that list of things that contribute to entropy I've noticed omits particles which under "normal circumstances" on earth exist in bound states, for example it doesn't mentions W bosons or gluons. But in some parts of the universe they're not bound but in different state of matter, e.g. quark gluon plasma. I wonder how or if this was taken I to account.

yellowcake0 2y ago

Information entropy is literally the strict lower bound on how efficiently information can be communicated (expected number of transmitted bits) if the probability distribution which generates this information is known, that's it. Even in contexts such as calculating the information entropy of a bit string, or the English language, you're just taking this data and constructing some empirical probability distribution from it using the relative frequencies of zeros and ones or letters or n-grams or whatever, and then calculating the entropy of that distribution. I can't say I'm overly fond of Baez's definition, but far be it from me to question someone of his stature.

arjunlol 2y ago

ΔS = ΔQ/T

utkarsh858 2y ago

I sometimes ponder where new entropy/randomness is coming from, like if we take the earliest state of universe as an infinitely dense point particle which expanded. So there must be some randomness or say variety which led it to expand in a non uniform way which led to the dominance of matter over anti-matter, or creation of galaxies, clusters etc. If we take an isolated system in which certain static particles are present, will there be the case that a small subset of the particles will get motion and this introduce entropy? Can entropy be induced automatically, atleast on a quantum level? If anyone can help me explain that it will be very helpful and thus can help explain origin of universe in a better way.

pseidemann 2y ago
I saw this video, which explained it for me (it's german, maybe the automatic subtitles will work for you): https://www.youtube.com/watch?v=hrJViSH6Klo https://www.youtube.com/watch?v=hrJViSH6Klo He argues that the randomness you are looking for comes from quantum fluctuations, and if this randomness did not exist, the universe would probably never have "happened".
- utkarsh858 2y ago
  Thanks for the reference will take some time before I see the whole video. Can you tell me what those quantum fluctuations are in short? Are they part of some physical law?
empath75 2y ago
Symmetry breaking is the general phenomenon that underlies most of that. The classic example is this: Imagine you have a perfectly symmetrical sombrero[1], and there's a ball balanced on top of the middle of the hat. There's no preferred direction it should fall in, but it's _unstable_. Any perturbation will make it roll down hill and come to rest in a stable configuration on the brim of the hat. The symmetry of the original configuration is now broken, but it's stable. 1: https://m.media-amazon.com/images/I/61M0LFKjI9L.__AC_SX300_SY300_QL70_FMwebp_.jpg https://m.media-amazon.com/images/I/61M0LFKjI9L.__AC_SX300_S...

tasteslikenoise 2y ago

I've always favored this down-to-earth characterization of the entropy of a discrete probability distribution. (I'm a big fan of John Baez's writing, but I was surprised glancing through the PDF to find that he doesn't seem to mention this viewpoint.) Think of the distribution as a histogram over some bins. Then, the entropy is a measurement of, if I throw many many balls at random into those bins, the probability that the distribution of balls over bins ends up looking like that histogram. What you usually expect to see is a uniform distribution of balls over bins, so the entropy measures the probability of other rare events (in the language of probability theory, "large deviations" from that typical behavior). More specifically, if P = (P1, ..., Pk) is some distribution, then the probability that throwing N balls (for N very large) gives a histogram looking like P is about 2^(-N * [log(k) - H(P)]), where H(P) is the entropy. When P is the uniform distribution, then H(P) = log(k), the exponent is zero, and the estimate is 1, which says that by far the most likely histogram is the uniform one. That is the largest possible entropy, so any other histogram has probability 2^(-c*N) of appearing for some c > 0, i.e., is very unlikely and exponentially moreso the more balls we throw, but the entropy measures just how much. "Less uniform" distributions are less likely, so the entropy also measures a certain notion of uniformity. In large deviations theory this specific claim is called "Sanov's theorem" and the role the entropy plays is that of a "rate function." The counting interpretation of entropy that some people are talking about is related, at least at a high level, because the probability in Sanov's theorem is the number of outcomes that "look like P" divided by the total number, so the numerator there is indeed counting the number of configurations (in this case of balls and bins) having a particular property (in this case looking like P). There are lots of equivalent definitions and they have different virtues, generalizations, etc, but I find this one especially helpful for dispelling the air of mystery around entropy.

vinnyvichy 2y ago
Hey did you want to say relative entropy ~ rate function ~ KL divergence. Might be more familiar to ML enthusiasts here, get them to be curious about Sanov or large deviations.
- tasteslikenoise 2y ago

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

vinnyvichy 2y ago

The book might disappoint some.. >I have largely avoided the second law of thermodynamics ... Thus, the aspects of entropy most beloved by physics popularizers will not be found here. But personally, this bit is the most exciting to me. >I have tried to say as little as possible about quantum mechanics, to keep the physics prerequisites low. However, Planck’s constant shows up in the formulas for the entropy of the three classical systems mentioned above. The reason for this is fascinating: Planck’s constant provides a unit of volume in position-momentum space, which is necessary to define the entropy of these systems. Thus, we need a tiny bit of quantum mechanics to get a good approximate formula for the entropy of hydrogen, even if we are trying our best to treat this gas classically.

foobarbecue 2y ago

How do you get to the actual book / tweets? The link just takes me back to the forward...

vishnugupta 2y ago
http://math.ucr.edu/home/baez/what_is_entropy.pdf http://math.ucr.edu/home/baez/what_is_entropy.pdf

GoblinSlayer 2y ago

There's fundamental nature of entropy, but as usual it's not very enlightening for poor monkey brain, so to explain you need to enumerate all its high level behavior, but its high level behavior is accidental and can't be summarized in a concise form.

space_oddity 2y ago
This complexity underscores the richness of the concept
- GoblinSlayer 2y ago
  I'd say it underscores its accidental nature.

ctafur 2y ago

The way I understand it is with an analogy to probability. To me, events are to microscopic states like random variable is to entropy.

ctafur 2y ago
My first contact with entropy was in chemistry and thermodynamics and I didn't get it. Actually I didn't get anything from engineering thermodynamics books such as Çengel and so. You must go to statistical mechanics or information theory to understand entropy. Or trying these PRICELESS NOTES from Prof. Suo: https://docs.google.com/document/d/1UMwpoDRZLlawWlL2Dz6YEomyhSNveQHp6pt7AUiP37Q/edit https://docs.google.com/document/d/1UMwpoDRZLlawWlL2Dz6YEomy...

jsomedon 2y ago

Am I only one that can't download the pdf, or is the file server down? I can see the blog page but when I try downloading the ebook it just doesn't work.. If the file server is down.. anyone could upload the ebook for download?

tromp 2y ago

Closely related recent discussion: https://news.ycombinator.com/item?id=40972589 https://news.ycombinator.com/item?id=40972589

tromp 2y ago

Closely related recent discussion on The Second Law of Thermodynamics (2011) (franklambert.net): https://news.ycombinator.com/item?id=40972589 https://news.ycombinator.com/item?id=40972589

ThrowawayTestr 2y ago

MC Hawking already explained this https://youtu.be/wgltMtf1JhY https://youtu.be/wgltMtf1JhY

ccosm 2y ago

"I have largely avoided the second law of thermodynamics, which says that entropy always increases. While fascinating, this is so problematic that a good explanation would require another book!" For those interested I am currently reading "Entropy Demystified" by Arieh Ben-Naim which tackles this side of things from much the same direction.

suoduandao3 2y ago

I like the formulation of 'the amount of information we don't know about a system that we could in theory learn'. I'm surprised there's no mention of the Copenhagen interpretation's interaction with this definition, under a lot of QM theories 'unavailable information' is different from available information.

tsoukase 2y ago

After years of thought I dare to say the 2nd TL is a tautology. Entropy is increasing means every system tends to higher probability means the most probable is the most probable.

tel 2y ago
I think that’s right, though it’s non-obvious that more probable systems are disordered. At least as non-obvious as Pascal’s triangle is. Which is to say, worth saying from a first principles POV, but not all that startling.

hjfjh 2y ago

[flagged]

illuminant 2y ago

Entropy is the distribution of potential over negative potential. This could be said "the distribution of what ever may be over the surface area of where it may be." This is erroneously taught in conventional information theory as "the number of configurations in a system" or the available information that has yet to be retrieved. Entropy includes the unforseen, and out of scope. Entropy is merely the predisposition to flow from high to low pressure (potential). That is it. Information is a form of potential. Philosophically what are entropy's guarantees? - That there will always be a super-scope, which may interfere in ways unanticipated; - everything decays the only mystery is when and how.

mwbajor 2y ago
All definitions of entropy stem from one central, universal definition: Entropy is the amount of energy unable to be used for useful work. Or better put grammatically: entropy describes the effect that not all energy consumed can be used for work.
- ajkjk 2y ago
  There's a good case to be made that the information-theoretic definition of entropy is the most fundamental one, and the version that shows up in physics is just that concept as applied to physics.
  - rimunroe 2y ago
    My favorite course I took as part of my physics degree was statistical mechanics. It leaned way closer to information theory than I would have expected going in, but in retrospect should have been obvious. Unrelated: my favorite bit from any physics book is probably still the introduction of the first chapter of "States of Matter" by David Goodstein: "Ludwig Boltzmann, who spent much of his life studying statistical mechanics, died in 1906, by his own hand. Paul Ehrenfest, carrying on the work, died similarly in 1933. Now it is our turn to study statistical mechanics."
  - galaxyLogic 2y ago