5 ms·

So it's an article on entropy (as disorder) without one mention of Shannon[1]? This seems like a pretty big oversight. In the context of information theory, en

by dvt 7y ago

So it's an article on entropy (as disorder) without one mention of Shannon[1]? This seems like a pretty big oversight. In the context of information theory, entropy is absolutely a decay of information. I understand that this is physics (not math -- although the formulae are virtually identical), but there's a very clear, accurate, and formal use of "entropy" to mean exactly what the author purports it doesn't mean: chaos. [1] https://heliosphan.org/shannon-entropy.html https://heliosphan.org/shannon-entropy.html

dr_dshiv 7y ago
I don't think it's an oversight. Shannon entropy, Von Neumann entropy and other information theory entropies are useful, but not physically fundamental in the sense of the second law of thermodynamics. That said, I don't under whether Gibbs entropy or Boltzmann entropy is fundamental-- ?
- dvt 7y ago
  > Shannon entropy, Von Neumann entropy and other information theory entropies are useful, but not physically fundamental... I have to disagree, especially given that one of the great mysteries of our generation (the Black Hole Information Paradox) hinges on what happens to information -- sure, it's information about particles, whatever that might mean. But Jacob Bekenstein (among others) argues that Thermodynamic entropy is analogous to information (Shannon) entropy[1]. [1] http://webhome.phy.duke.edu/~mkruse/Phy255/255S06_Joyce_Coppock_bh.pdf http://webhome.phy.duke.edu/~mkruse/Phy255/255S06_Joyce_Copp...
  - dr_dshiv 7y ago
    But, analogous?
  - hcs 7y ago
    I take your point, it might have been interesting to talk about how a lot of relativity and quantum mechanics can be thought of in terms of propagation and destruction of information. And for many students, talking about codes is a lot more interesting than pressure gradients in ideal gasses. A missed opportunity, perhaps.
  - kgwgk 7y ago
    http://backreaction.blogspot.com/2019/08/how-do-black-holes-destroy-information.html http://backreaction.blogspot.com/2019/08/how-do-black-holes-... “As you have probably noticed, I didn’t say anything about information. That’s because really the reference to information in “black hole information loss” is entirely unnecessary and just causes confusion. The problem of black hole “information loss” really has nothing to do with just exactly what you mean by information. It’s just a term that loosely speaking says you can’t tell from the final state what was the exact initial state.“
    - abdullahkhalids 7y ago
      This is not the consensus opinion in the physics community. Many physicists consider information as fundamental to this "paradox".
      - kgwgk 7y ago
        Right. The other view isn’t the consensus either. I was just presenting an alternative exposition.
- jbay808 7y ago
  Shannon / information entropy absolutely is fundamental to thermodynamics. It clears up a lot of confusion when you realize that the entropy isn't in the system itself, it's in your head (your lack of information about a system's exact state, knowing only summary statistics like its mass, volume, and temperature).
  - antpls 7y ago
    I studied Shannon entropy like any CS student back in University, so I'm not an expert in all interpretations of Shannon entropy, but I don't think Shannon entropy tells you anything about what or how much information you are lacking about the system. It may be used to this end, but it's not what Shannon entropy directly gives you. There is no way to reduce the entropy of a totally random system by knowing more about it, unless your hypothesis is that true randomness doesn't exist in the universe
    - jbay808 7y ago
      To connect them, the entropy of a system (such as an ideal gas) can be computed as the logarithm of the number of equally probable microstates in your probability distribution for that system. (You need to define a sensible measure, because the state variables are continuous, but that's not important). You can express this in terms of the message length or number of bits of information you would need to gain about that container of ideal gas in order to narrow it down to a particular microstate. It's no different than the entropy of, say, a password. > There is no way to reduce the entropy of a totally random system by knowing more about it There absolutely is! For example, Maxwell's demon. Maxwell's demon will fail to reduce system entropy if it doesn't know the microstate to begin with, but otherwise it can use its knowledge to reduce the entropy of a gas by separating its parts. > unless your hypothesis is that true randomness doesn't exist in the universe I do also subscribe to this hypothesis, that like entropy, randomness also only exists in our heads as a reflection of our lack of information about a system, and isn't a property of any system itself. But I don't think I need to rely on this assumption.
      - dr_dshiv 7y ago
upofadown 7y ago
It seems to me that the "Typical vs. average" section of the article specifically covers informational entropy. The question seems to hinge on the difference between disorder and randomness.
- dvt 7y ago
  > specifically covers informational entropy It really doesn't. The little Enterprise thought-experiment and anecdotal impressions of what "looks" and "doesn't look" disordered is nonsense. From my source above: > Shannon entropy is defined for a given discrete probability distribution; it measures how much information is required, on average, to identify random samples from that distribution. It's carefully defined, and used in basically every piece of technology we own. Further, it's not even particularly tough to wrap your head around (a middle-schooler could understand Huffman encoding).
  - hcs 7y ago
    > The little Enterprise thought-experiment and anecdotal impressions of what "looks" and "doesn't look" disordered is nonsense. I think you're misreading it, the author is trying to make the point that you can't meaningfully answer that question just by looking at the dots. It's a trick question, but it's used as a part of teaching how you compute entropy (which involves knowing the distribution). Yes, this is essentially information theory, but he's a physics teacher.
    - dvt 7y ago
      I can't tell you the "randomness" of a bunch of points any more than I can tell you the location of sunspots by looking at our star with the naked eye. That doesn't mean that there isn't a way of calculating the entropy, though (just how there is a way of making out sunspots with the right equipment). Is that really the idea here? You're right: maybe I'm just not getting the point.
      - hcs 7y ago
        The "equipment" needed is a model of how the points came about. You may already get the point but permit me an example: From an information theory perspective, consider 'apple' vs 'aaaaa'. I submit that if you have nothing besides the string itself, you'd conclude that 'aaaaa' is lower entropy than 'apple', but if your model is word frequency on the web, it'd be the other way around. There isn't anything inherent in the strings that makes one or the other true. There are interesting cases in-between where you could derive a model with more math, e.g. the first million digits of pi has very low entropy, but in general exactly how much is a matter of the uncomputable Kolmogorov complexity. With a physical system, if you know the configuration of the system and the laws of physics it obeys, you can compute this in a meaningful way, and it can tell you what direction the system "wants to go" to satisfy the second law of thermodynamics. Wikipedia has a neat article on the topic which I'm just starting to read now: https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_information_theory https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_...
hcs 7y ago
The author's point is that "chaos" and "disorder" aren't well defined, usually. The entropy content of a situation can't be computed without knowing the physics that would have gotten it there, just as I can't compute the entropy of a message (my surprise at receiving it, or the number of bits needed to transmit it) consisting of a single symbol without some knowledge of the distribution it was drawn from (the expectations of myself, or the receiver). (If it is part of a longer message I can use that message itself as a model, but that is only a first approximation.)
acqq 7y ago
> there's a very clear, accurate, and formal use of "entropy" to mean exactly what the author purports it doesn't mean: chaos. Shannon didn't define it as "chaos" rather as a: "measure of information, choice and uncertainty." http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop... It also matches what we know: the less "uncertain" some amount of the information the less it is compressible. However he does mention "chaotic" in one sentence, in evaluating the "constraints" imposed by the language when making the crossword puzzles: "If the redundancy is too high the language imposes too many constraints for large crossword puzzles to be possible. A more detailed analysis shows that if we assume the constraints imposed by the language are of a rather chaotic and random nature, large crossword puzzles are just possible when the redundancy is 50%."

by dvt 7y ago

dr_dshiv 7y ago

I don't think it's an oversight. Shannon entropy, Von Neumann entropy and other information theory entropies are useful, but not physically fundamental in the sense of the second law of thermodynamics. That said, I don't under whether Gibbs entropy or Boltzmann entropy is fundamental-- ?

dvt 7y ago
> Shannon entropy, Von Neumann entropy and other information theory entropies are useful, but not physically fundamental... I have to disagree, especially given that one of the great mysteries of our generation (the Black Hole Information Paradox) hinges on what happens to information -- sure, it's information about particles, whatever that might mean. But Jacob Bekenstein (among others) argues that Thermodynamic entropy is analogous to information (Shannon) entropy[1]. [1] http://webhome.phy.duke.edu/~mkruse/Phy255/255S06_Joyce_Coppock_bh.pdf http://webhome.phy.duke.edu/~mkruse/Phy255/255S06_Joyce_Copp...
- dr_dshiv 7y ago
  But, analogous?
- hcs 7y ago
  I take your point, it might have been interesting to talk about how a lot of relativity and quantum mechanics can be thought of in terms of propagation and destruction of information. And for many students, talking about codes is a lot more interesting than pressure gradients in ideal gasses. A missed opportunity, perhaps.
- kgwgk 7y ago
  http://backreaction.blogspot.com/2019/08/how-do-black-holes-destroy-information.html http://backreaction.blogspot.com/2019/08/how-do-black-holes-... “As you have probably noticed, I didn’t say anything about information. That’s because really the reference to information in “black hole information loss” is entirely unnecessary and just causes confusion. The problem of black hole “information loss” really has nothing to do with just exactly what you mean by information. It’s just a term that loosely speaking says you can’t tell from the final state what was the exact initial state.“
  - abdullahkhalids 7y ago
    This is not the consensus opinion in the physics community. Many physicists consider information as fundamental to this "paradox".
    - kgwgk 7y ago
      Right. The other view isn’t the consensus either. I was just presenting an alternative exposition.
jbay808 7y ago
Shannon / information entropy absolutely is fundamental to thermodynamics. It clears up a lot of confusion when you realize that the entropy isn't in the system itself, it's in your head (your lack of information about a system's exact state, knowing only summary statistics like its mass, volume, and temperature).
- antpls 7y ago
  I studied Shannon entropy like any CS student back in University, so I'm not an expert in all interpretations of Shannon entropy, but I don't think Shannon entropy tells you anything about what or how much information you are lacking about the system. It may be used to this end, but it's not what Shannon entropy directly gives you. There is no way to reduce the entropy of a totally random system by knowing more about it, unless your hypothesis is that true randomness doesn't exist in the universe
  - jbay808 7y ago
    To connect them, the entropy of a system (such as an ideal gas) can be computed as the logarithm of the number of equally probable microstates in your probability distribution for that system. (You need to define a sensible measure, because the state variables are continuous, but that's not important). You can express this in terms of the message length or number of bits of information you would need to gain about that container of ideal gas in order to narrow it down to a particular microstate. It's no different than the entropy of, say, a password. > There is no way to reduce the entropy of a totally random system by knowing more about it There absolutely is! For example, Maxwell's demon. Maxwell's demon will fail to reduce system entropy if it doesn't know the microstate to begin with, but otherwise it can use its knowledge to reduce the entropy of a gas by separating its parts. > unless your hypothesis is that true randomness doesn't exist in the universe I do also subscribe to this hypothesis, that like entropy, randomness also only exists in our heads as a reflection of our lack of information about a system, and isn't a property of any system itself. But I don't think I need to rely on this assumption.
    - dr_dshiv 7y ago

upofadown 7y ago

It seems to me that the "Typical vs. average" section of the article specifically covers informational entropy. The question seems to hinge on the difference between disorder and randomness.

dvt 7y ago
> specifically covers informational entropy It really doesn't. The little Enterprise thought-experiment and anecdotal impressions of what "looks" and "doesn't look" disordered is nonsense. From my source above: > Shannon entropy is defined for a given discrete probability distribution; it measures how much information is required, on average, to identify random samples from that distribution. It's carefully defined, and used in basically every piece of technology we own. Further, it's not even particularly tough to wrap your head around (a middle-schooler could understand Huffman encoding).
- hcs 7y ago
  > The little Enterprise thought-experiment and anecdotal impressions of what "looks" and "doesn't look" disordered is nonsense. I think you're misreading it, the author is trying to make the point that you can't meaningfully answer that question just by looking at the dots. It's a trick question, but it's used as a part of teaching how you compute entropy (which involves knowing the distribution). Yes, this is essentially information theory, but he's a physics teacher.
  - dvt 7y ago
    I can't tell you the "randomness" of a bunch of points any more than I can tell you the location of sunspots by looking at our star with the naked eye. That doesn't mean that there isn't a way of calculating the entropy, though (just how there is a way of making out sunspots with the right equipment). Is that really the idea here? You're right: maybe I'm just not getting the point.
    - hcs 7y ago
      The "equipment" needed is a model of how the points came about. You may already get the point but permit me an example: From an information theory perspective, consider 'apple' vs 'aaaaa'. I submit that if you have nothing besides the string itself, you'd conclude that 'aaaaa' is lower entropy than 'apple', but if your model is word frequency on the web, it'd be the other way around. There isn't anything inherent in the strings that makes one or the other true. There are interesting cases in-between where you could derive a model with more math, e.g. the first million digits of pi has very low entropy, but in general exactly how much is a matter of the uncomputable Kolmogorov complexity. With a physical system, if you know the configuration of the system and the laws of physics it obeys, you can compute this in a meaningful way, and it can tell you what direction the system "wants to go" to satisfy the second law of thermodynamics. Wikipedia has a neat article on the topic which I'm just starting to read now: https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_information_theory https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_...

hcs 7y ago

The author's point is that "chaos" and "disorder" aren't well defined, usually. The entropy content of a situation can't be computed without knowing the physics that would have gotten it there, just as I can't compute the entropy of a message (my surprise at receiving it, or the number of bits needed to transmit it) consisting of a single symbol without some knowledge of the distribution it was drawn from (the expectations of myself, or the receiver). (If it is part of a longer message I can use that message itself as a model, but that is only a first approximation.)

acqq 7y ago

> there's a very clear, accurate, and formal use of "entropy" to mean exactly what the author purports it doesn't mean: chaos. Shannon didn't define it as "chaos" rather as a: "measure of information, choice and uncertainty." http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop... It also matches what we know: the less "uncertain" some amount of the information the less it is compressible. However he does mention "chaotic" in one sentence, in evaluating the "constraints" imposed by the language when making the crossword puzzles: "If the redundancy is too high the language imposes too many constraints for large crossword puzzles to be possible. A more detailed analysis shows that if we assume the constraints imposed by the language are of a rather chaotic and random nature, large crossword puzzles are just possible when the redundancy is 50%."