14 ms·

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equally likely. If you randomly flip bits you go from one state to another state but each one is equally likely to occur. There is no special meaning to a specific configuration if you don't give it one. If you look at the average of all bits you start grouping all states together with the equal number of 1s. If you talk about the average there is only one configuration that is all 1s but most configurations have roughly 50% 1s. If you now start flipping bits you will meander through all possible bit-states but the average will most likely be close to 50% 1s most of the time. In physics we usually look at averages such as the average velocity expressed as temperature. Therefore it makes sense to group together all states using the average and then the states with very low or very high averages are few. But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I don't know what Entropy is supposed to mean on the level of individual states/configurations. I don't understand what kind of macroscopic "averaging" function we may use to group up those states. There could be more than one possibility - from that would follow that there is more than one definition of macro-Entropy. Ideally there should be one general definition of how we have to look at those microstates and from that follows our general definition of Entropy. Sadly I didn't study Physics and this topic still continues to confuse me. The usual explanations fail to enlighten me.

oakwhiz 4y ago
>There could be more than one possibility This seems to connect with the idea behind Chaitin's incompleteness theorem. Making specific statements about the reducible complexity of something is not always possible.
pfortuny 4y ago
It is also because “the system” is intrinsic to the notion: as you say, any configuration of bits is equally likely; this only takes into account the “system of the bits “. The moment your system is “the bits and their mean value” everything changes, as there are systems with a single possible configuration. That is what happens when he starts the first example: “a system of particles INSIDE A VOLUME. The volume is what makes the entropy larger or smaller. The particles in a different volume (or just by themselves) have a different entropy.
jbay808 4y ago
> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a thermometer in it and measured X. You don't know anything more than the average temperature. It could be in any state consistent with that temperature. If you know more about the system, it has less entropy. If you know it down to the exact microstate, it has zero entropy.
- andi999 4y ago
  Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.
  - oldsecondhand 4y ago
    I think he is confusing the usage of entropy in physics and computer science. In computer science entropy is conditional probability and depends on what we know about a system.
    - kgwgk 4y ago
      As it does in physics! "which parameters [thermodynamic variables] you know" ~ "which parameters [thermodynamic variables] you hold fixed" (or know in average, like the energy for a system in a heat bath where the temperature is fixed) https://bayes.wustl.edu/etj/articles/theory.1.pdf https://bayes.wustl.edu/etj/articles/theory.1.pdf http://nicf.net/articles/thermodynamics-statistical-mechanics/ http://nicf.net/articles/thermodynamics-statistical-mechanic...
      - oldsecondhand 4y ago
        You can observe the movement of molecules beyond macro properties like temperature.
d--b 4y ago
Not a physicist either, and I don't claim to understand entropy that well either but maybe it would help to consider that entropy may not be a universal variable of systems in the universe. I think you should rather consider it as a mathematical construct that applies to some systems where the microscopic quantities are well defined, and where the 'averaging' that we can observe is also well defined. So if you look at thermodynamics, entropy is well defined, but you may be totally right, that what we call "microscopic states" in a gas can be broken down further in elementary particles, that may or may not behave in quantic ways, and what not, and counting the micro-states considering the elementary particles is a whole different game. But it doesn't really matter. What matters is that at the scale we're at and with the microscopic/macroscopic relation that's defined, entropy works. The calculations that give some numbers to entropy show that it looks like entropy cannot decrease. They call it a universal principle of thermodynamics, because there is nothing (to my understanding), that explains it microscopically. And it works for a variety of situation in physics, such that it seems that it's a universal property of nature. But it's mostly mathematical. It seems to say that "given a system we know everything about, there is no way to go to a system that has some unknown things to us". Anyways. I mostly wrote this to see if I could articulate it to myself, hopefully it helps you as well.
- monkeycantype 4y ago
  Thanks for writing that. From the perspective you’ve articulated I sometimes wonder whether the idea of the heat death of the universe is a matter of perspective, it only applies to the matter and properties of the universe that we consider significant, are we living within the heat deaths of past forms of the universe in which physical interactions we have overlooked dominated?
- lambdatronics 4y ago
  >It seems to say that "given a system we know everything about, there is no way to go to a system that has some unknown things to us". That's backwards: information is the negative of entropy. The 2nd law says that entropy never decreases, so information never increases (it can only be preserved or lost).
mathgenius 4y ago
Have you looked at Huffman coding? I'd recommend the (free) book by David MacKay, it is secretly the "hackers guide to thermodynamics". http://www.inference.org.uk/mackay/itila/book.html http://www.inference.org.uk/mackay/itila/book.html
- chermi 4y ago
  This book is phenomenal and I highly second this recommendation.
contravariant 4y ago
Yeah the author is conflating low entropy with a low number of microstates, which is consistent with the thermodynamic assumption that maximal entropy means a uniform distribution of microstates, but is confusing. The purest mathematical justification for why low entropy means a low number of microstates probably comes from the fact that (classical) physical systems are a dynamical systems that preserve the measure induced by the standard metric of the phase space. The measure theoretic definition of entropy then implies the entropy of a partition of the phase space (i.e. a set of macrostates) is indeed the average of the logarithm of the number of microstates. So indeed if W is the number of microstates in the current macrostate then entropy = log(W) (on average). And using the typical set you can show that the probability that the average of log(W) over n samples is within 'epsilon' of the 'exact' entropy goes to 1 as n goes to infinity. This is the mathematical justification for the second law of thermodynamics. The trick is that all of this is true no matter how you partition phase-space. Though that does mean that what is and isn't a high entropy state depends on your perspective.
- oh_my_goodness 4y ago
  >The trick is that all of this is true no matter how you partition phase-space. Though that does mean that what is and isn't a high entropy state depends on your perspective. That seems to be correct. >Yeah the author is conflating low entropy with a low number of microstates Since entropy is found by counting micro-states (for example your third paragraph), that should be ok. What am I missing?
  - contravariant 4y ago
    The equivalence is an important theorem (important enough to be engraved on Boltzmann's gravestone), and if you're switching back and forth in an explanation of what entropy is then you're skipping over some important details that answer what it means for something to be a 'low entropy state'.
tiasbtw 4y ago
I suppose the problem is because entropy is a proxy for the number of states of an abstract configuration space which has the same observed quantity as the concrete object that you take to measure its entropy. So, for example, if you know that your object with mass M and temperature T then to measure its entropy you take all the posible states for an abstract object with mass M and temperature T and the logarithm of that number of states is the definition of the entropy of one object that has mass M and temperature T. So the more you know about the the concrete object the less number of posible states for the abstract model and so the entropy is not a property of the concrete object rather is a property of an abstract model with same fixed global properties.
danbruc 4y ago
First, entropy is a macroscopic property, it makes no sense to talk about the entropy of a single particle. Second, entropy is not a fundamental property, it depends on what the observer cares about. Take the common example of a gas in the corner of a box, in that case we care about the density distribution in the box, a macroscopic property. To make this more concrete, one way to quantify the density distribution could be to tessellate the box with cubes of some size, find the ones with the lowest and highest density, and use the difference between those two densities as a measure of density variation. Once we have decided that all we care about is the density variation in the box, we can go through all possible microscopic states and group them together by their density variation. Finally a low entropy macroscopic state is simply a macroscopic state - a certain density variation - for which there are only a few microscopic states that have the corresponding density variation. On the other hand a high entropy macroscopic state is a macroscopic state for which there are many microscopic states that have the corresponding density variation. You can also call the microscopic states low or high entropy but only with reference to the macroscopic property you use to group them, in themselves microscopic states are not low or high entropy. If you observe a low entropy macroscopic state, then you know a lot about the microscopic state, after all there are only very few. If you observe a high entropy macroscopic state, then you know a lot less about the microscopic state, there are much more possibilities even though they are microscopically indistinguishable. And if there are no limiting constraints on how the microscopic states can evolve, if the evolution is essentially a random walk through all possible microscopic states, then the entropy of the system will increase with high probability as it is much more probable to randomly walk into one of the many microscopic states associated with a high entropy macroscopic state than to walk into one of the few microscopic states associated with a low entropy macroscopic state.
- Banana699 4y ago
  >there are much more possibilities even though they are microscopically indistinguishable. You meant "macroscopically" ?
  - danbruc 4y ago
    True, but to late to edit.
ricardobeat 4y ago
To stay within your bits analogy, I imagine an increase in entropy would be the equivalent of each bit becoming base-3, base-4, and so on, hence increasing the number of possible states (and reducing your ability to predict them).
coldtea 4y ago
>Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. Well, there are only 2 states with all 1 or all 0. But there are 2^N states of mixed 1 and 0. Even if you treat the sets of bits as opaque items, and pick one from a bucket, I'd expect getting one of the 2^N - 2 configurations to be a far more likely outcome than one of the 2 remaining. In fact, we could bet on it...
- jbay808 4y ago
  But there's only one state that is 10010001111110101000.
  - coldtea 4y ago
    Sure, but that's irrelevant. There are billions that are similar, and only one that's all 0.
    - jbay808 4y ago
      "similar" is in the eye of the observer! This fact should be especially clear in the context of bit strings. If 10010001111110101000 is my login password, don't be surprised if other permutations fail to grant you access, even if you have the correct number of 1's and 0's.
      - coldtea 4y ago
        >"similar" is in the eye of the observer! Nope, also similar in algorithmic complexity theory (e.g. counting compressibility).
        jbay808 4y ago
        That's a very good, and deep, point, and I agree that there is something important there. However, algorithmic complexity is still only defined relative to an arbitrary reference computer. If my reference computer happens to, in hardware, XOR its inputs and outputs with the bitstring 10010001111110101000, then (in terms of bits that end up represented on the drive), that will be the one that has the lowest algorithmic complexity, although the algorithm might think that it's outputting all 0's.
oh_my_goodness 4y ago
Here's a concrete example of entropy with just two macro-states, 'broken' and 'unbroken'. If it becomes unclear or unconvincing, can you point out where that happens? It's intended to be ELI5, clear enough to discuss coherently. Question: Why is it when I drop a vase it smashes into a million pieces; however when I then drop the million pieces it does not form a vase? Answer: Stop! Don't drop any more expensive vases. Start with these simpler systems that do repair themselves sometimes when you drop them. Take a coin and align it so the 'heads' side faces up. 'Heads' means 'unbroken'. (The reason for that will become clearer as we do more experiments.) Now drop the coin on the floor. How often is the 'heads' side still facing up? Now if you drop it again, how often does it 'repair' itself so that the 'heads' side is up? (Really do this.) Try the experiment with 2 coins. Align them all heads-up, drop them, then see if your pattern is 'broken'. ('Broken' means not-all-heads-up.) Drop the 'broken' coins again. How often do they 'repair' themselves? ('Repaired' means all heads-up.) ( Don't think about it! Don't solve for it! Do it! ) Try again with 5 coins. How often does a 5-coin system 'break' when you drop it? How often does a broken 5-coin system 'repair itself' when you drop it again? How about 10 coins? How often does a broken pattern of 10 mixed heads/tails repair itself to all heads when you drop it again? Sometimes it does, but you'll have to be very lucky or patient to see it happen. I think from here you can probably see (part of) the answer to your question about the vase. The word people use for this kind of thing is 'entropy'. With enough coins, the 'broken' state is much more probable than the 'repaired' state. The log of a probability is called 'entropy.' https://www.quora.com/Why-is-it-when-I-drop-a-vase-it-smashes-into-a-million-pieces-however-when-I-then-drop-the-million-pieces-it-does-not-form-a-vase/answer/Tom-McNamara-4 https://www.quora.com/Why-is-it-when-I-drop-a-vase-it-smashe...
sixo 4y ago
> I don't know what Entropy is supposed to mean on the level of individual states/configurations. I don't understand what kind of macroscopic "averaging" function we may use to group up those states I find it helpful to think of entropy as a property of not of the system, or any individual state (micro- or macro-), but as a property of the "compression" process that summarizes microstates with a coarser-grained macro-description. Given a choice of compression, classical physics says a system will tend to spent most of their time in the most likely compressed state. Different choices of compressions can lead to different macro descriptions, with different "entropies" and different dynamics among their macroscopic variables. In this light it's not meaningful to think of the entropy of individual states. You could think about the "identity" compression, but you would end up with a description that was exactly as complicated as the full micro-state time-evolution dynamics; you wouldn't end up with any smaller set of variables that could describe the equilibrium of the whole system (really this would not admit an "equilibrium" at all)
deltaonefour 4y ago
Another thing that contributes to the confusion which you have noted but not fully realized is that there are two different concepts that use the same equation and the same word: "entropy". Information entropy and statistical entropy are two different things.
- deleted 4y ago
  [deleted]
raxxorraxor 4y ago
In computer science a high entropy means a high informational value. So any information that is not the expected value has a high content of information and therefore a high entropy. In thermodynamics the case is a bit to the opposite as a high entropy means a low state of energy and therefore less internal processes within a system or none at all.
foobarqux 4y ago
For information theoretic entropy (I don’t know anything about thermodynamics): The first thing you describe, that if you draw many symbols from your source distribution at random then you see a distribution of symbols that is equal to the probability distribution of the source, is called the asymptotic equipartition principle. I think your confusion comes from two things. First, conflating bits and symbols and second assuming symbols are equiprobable. Take English text where our symbols could be letters of the alphabet. These are not equiprobable, if you select a letter from a book at random you get a different distribution than 1/26. If you took as your symbols the individual bits that would encode those characters in ascii you would get something closer to equal probable symbols. Another choice for symbols would be the words in the book.
keithlfrost 4y ago
This is a very sensible confusion. The forms of macroscopic averaging functions which are useful and valid cannot be made up arbitrarily, but are determined by the microscopic physical laws of the system. There is a reason that the law of increase of entropy is the second law of classical thermodynamics, with conservation of energy being the first law. To state it explicitly: energy is a globally conserved quantity, which can be freely exchanged among the interacting microscopic parts of systems. So we can bring a test system (called a thermometer) into interaction with our system under study, (indirectly) observe the average energy per degree of freedom of the thermometer, and call that observation the temperature of the system under study. Similarly, it is a known physical phenomenon that a gas confined to a container will exert a steady average outward force per normal unit area on the walls of the container; we have ways to measure this force, and we call it pressure. And so on, and on: every useful macroscopic averaging function is a relatively stable, measurable quantity which is determined by the physics of the systems under study. If we discovered some new measurement technique tomorrow which enabled us to measure the "quintessence" of physical systems, and this measurement was stable and reproducible, and could be meaningfully aggregated from the microscopic parts of the system and measured on the macroscopic scale, our definition of entropy would change, to account for "quintessence".
jl2718 4y ago
A specific string of bits has zero entropy. It may be a sample from a distribution which has some entropy. Unless you are selecting a random single bit from that string, in which the entropy of that selection process is -p1log p1 - p0log p0.
lambdatronics 4y ago
>I don't know what Entropy is supposed to mean on the level of individual states/configurations. The entropy is a property of a probability distribution, not of a state. Entropy is defined as H = -sum(p_i log(p_i)). A 'state' implicitly defines a probability distribution: uniform probability over all the microstates compatible with the state description.[0] In the case of a microstate, the entropy of the probability distribution over microstates consistent with that state is zero - there's only one compatible state, so p_i = 0 for all other states, and log(p_i) = 0 for the compatible state. In the case of a macrostate, the entropy of the probability distribution over microstates consistent with the macrostate works out to -sum((1/N) log(1/N)) = log(N), where N is the number of consistent microstates. That's the Boltzmann entropy. Sometimes people will write about the entropy of a 'state' in such a way that it sounds like they're talking about the entropy of a microstate -- but what they're probably talking about is "the entropy of the macrostate that this microstate belongs to." It's sloppy to talk like that, because "the" corresponding macrostate isn't unique. There are many sets of macrostates that could contain a microstate, depending on what properties of the microstates one considers 'macro.' (Ex: 10100101 is a member of both "symmetric bit strings of length 8" and "bit strings of length 8 that average to 1/2". The entropy of "symmetric bit strings of length 8" is 4 bits, whereas the entropy of "bit strings of length 8 that average to 1/2" is ~6.1 bits. And of course, the entropy of "the bit string of length 8 that is exactly 10100101" is zero.)
esalman 4y ago
High entropy = less predictable system. Low entropy = highly predictable system.

oakwhiz 4y ago

>There could be more than one possibility This seems to connect with the idea behind Chaitin's incompleteness theorem. Making specific statements about the reducible complexity of something is not always possible.

pfortuny 4y ago

It is also because “the system” is intrinsic to the notion: as you say, any configuration of bits is equally likely; this only takes into account the “system of the bits “. The moment your system is “the bits and their mean value” everything changes, as there are systems with a single possible configuration. That is what happens when he starts the first example: “a system of particles INSIDE A VOLUME. The volume is what makes the entropy larger or smaller. The particles in a different volume (or just by themselves) have a different entropy.

jbay808 4y ago

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a thermometer in it and measured X. You don't know anything more than the average temperature. It could be in any state consistent with that temperature. If you know more about the system, it has less entropy. If you know it down to the exact microstate, it has zero entropy.

andi999 4y ago
Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.
- oldsecondhand 4y ago
  I think he is confusing the usage of entropy in physics and computer science. In computer science entropy is conditional probability and depends on what we know about a system.
  - kgwgk 4y ago
    As it does in physics! "which parameters [thermodynamic variables] you know" ~ "which parameters [thermodynamic variables] you hold fixed" (or know in average, like the energy for a system in a heat bath where the temperature is fixed) https://bayes.wustl.edu/etj/articles/theory.1.pdf https://bayes.wustl.edu/etj/articles/theory.1.pdf http://nicf.net/articles/thermodynamics-statistical-mechanics/ http://nicf.net/articles/thermodynamics-statistical-mechanic...
    - oldsecondhand 4y ago
      You can observe the movement of molecules beyond macro properties like temperature.

d--b 4y ago

Not a physicist either, and I don't claim to understand entropy that well either but maybe it would help to consider that entropy may not be a universal variable of systems in the universe. I think you should rather consider it as a mathematical construct that applies to some systems where the microscopic quantities are well defined, and where the 'averaging' that we can observe is also well defined. So if you look at thermodynamics, entropy is well defined, but you may be totally right, that what we call "microscopic states" in a gas can be broken down further in elementary particles, that may or may not behave in quantic ways, and what not, and counting the micro-states considering the elementary particles is a whole different game. But it doesn't really matter. What matters is that at the scale we're at and with the microscopic/macroscopic relation that's defined, entropy works. The calculations that give some numbers to entropy show that it looks like entropy cannot decrease. They call it a universal principle of thermodynamics, because there is nothing (to my understanding), that explains it microscopically. And it works for a variety of situation in physics, such that it seems that it's a universal property of nature. But it's mostly mathematical. It seems to say that "given a system we know everything about, there is no way to go to a system that has some unknown things to us". Anyways. I mostly wrote this to see if I could articulate it to myself, hopefully it helps you as well.

monkeycantype 4y ago
Thanks for writing that. From the perspective you’ve articulated I sometimes wonder whether the idea of the heat death of the universe is a matter of perspective, it only applies to the matter and properties of the universe that we consider significant, are we living within the heat deaths of past forms of the universe in which physical interactions we have overlooked dominated?
lambdatronics 4y ago
>It seems to say that "given a system we know everything about, there is no way to go to a system that has some unknown things to us". That's backwards: information is the negative of entropy. The 2nd law says that entropy never decreases, so information never increases (it can only be preserved or lost).

mathgenius 4y ago

Have you looked at Huffman coding? I'd recommend the (free) book by David MacKay, it is secretly the "hackers guide to thermodynamics". http://www.inference.org.uk/mackay/itila/book.html http://www.inference.org.uk/mackay/itila/book.html

chermi 4y ago
This book is phenomenal and I highly second this recommendation.

contravariant 4y ago

Yeah the author is conflating low entropy with a low number of microstates, which is consistent with the thermodynamic assumption that maximal entropy means a uniform distribution of microstates, but is confusing. The purest mathematical justification for why low entropy means a low number of microstates probably comes from the fact that (classical) physical systems are a dynamical systems that preserve the measure induced by the standard metric of the phase space. The measure theoretic definition of entropy then implies the entropy of a partition of the phase space (i.e. a set of macrostates) is indeed the average of the logarithm of the number of microstates. So indeed if W is the number of microstates in the current macrostate then entropy = log(W) (on average). And using the typical set you can show that the probability that the average of log(W) over n samples is within 'epsilon' of the 'exact' entropy goes to 1 as n goes to infinity. This is the mathematical justification for the second law of thermodynamics. The trick is that all of this is true no matter how you partition phase-space. Though that does mean that what is and isn't a high entropy state depends on your perspective.

oh_my_goodness 4y ago
>The trick is that all of this is true no matter how you partition phase-space. Though that does mean that what is and isn't a high entropy state depends on your perspective. That seems to be correct. >Yeah the author is conflating low entropy with a low number of microstates Since entropy is found by counting micro-states (for example your third paragraph), that should be ok. What am I missing?
- contravariant 4y ago
  The equivalence is an important theorem (important enough to be engraved on Boltzmann's gravestone), and if you're switching back and forth in an explanation of what entropy is then you're skipping over some important details that answer what it means for something to be a 'low entropy state'.

tiasbtw 4y ago

I suppose the problem is because entropy is a proxy for the number of states of an abstract configuration space which has the same observed quantity as the concrete object that you take to measure its entropy. So, for example, if you know that your object with mass M and temperature T then to measure its entropy you take all the posible states for an abstract object with mass M and temperature T and the logarithm of that number of states is the definition of the entropy of one object that has mass M and temperature T. So the more you know about the the concrete object the less number of posible states for the abstract model and so the entropy is not a property of the concrete object rather is a property of an abstract model with same fixed global properties.

danbruc 4y ago

First, entropy is a macroscopic property, it makes no sense to talk about the entropy of a single particle. Second, entropy is not a fundamental property, it depends on what the observer cares about. Take the common example of a gas in the corner of a box, in that case we care about the density distribution in the box, a macroscopic property. To make this more concrete, one way to quantify the density distribution could be to tessellate the box with cubes of some size, find the ones with the lowest and highest density, and use the difference between those two densities as a measure of density variation. Once we have decided that all we care about is the density variation in the box, we can go through all possible microscopic states and group them together by their density variation. Finally a low entropy macroscopic state is simply a macroscopic state - a certain density variation - for which there are only a few microscopic states that have the corresponding density variation. On the other hand a high entropy macroscopic state is a macroscopic state for which there are many microscopic states that have the corresponding density variation. You can also call the microscopic states low or high entropy but only with reference to the macroscopic property you use to group them, in themselves microscopic states are not low or high entropy. If you observe a low entropy macroscopic state, then you know a lot about the microscopic state, after all there are only very few. If you observe a high entropy macroscopic state, then you know a lot less about the microscopic state, there are much more possibilities even though they are microscopically indistinguishable. And if there are no limiting constraints on how the microscopic states can evolve, if the evolution is essentially a random walk through all possible microscopic states, then the entropy of the system will increase with high probability as it is much more probable to randomly walk into one of the many microscopic states associated with a high entropy macroscopic state than to walk into one of the few microscopic states associated with a low entropy macroscopic state.

Banana699 4y ago
>there are much more possibilities even though they are microscopically indistinguishable. You meant "macroscopically" ?
- danbruc 4y ago
  True, but to late to edit.

ricardobeat 4y ago

To stay within your bits analogy, I imagine an increase in entropy would be the equivalent of each bit becoming base-3, base-4, and so on, hence increasing the number of possible states (and reducing your ability to predict them).

coldtea 4y ago

>Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. Well, there are only 2 states with all 1 or all 0. But there are 2^N states of mixed 1 and 0. Even if you treat the sets of bits as opaque items, and pick one from a bucket, I'd expect getting one of the 2^N - 2 configurations to be a far more likely outcome than one of the 2 remaining. In fact, we could bet on it...

jbay808 4y ago
But there's only one state that is 10010001111110101000.
- coldtea 4y ago
  Sure, but that's irrelevant. There are billions that are similar, and only one that's all 0.
  - jbay808 4y ago
    "similar" is in the eye of the observer! This fact should be especially clear in the context of bit strings. If 10010001111110101000 is my login password, don't be surprised if other permutations fail to grant you access, even if you have the correct number of 1's and 0's.
    - coldtea 4y ago
      >"similar" is in the eye of the observer! Nope, also similar in algorithmic complexity theory (e.g. counting compressibility).
      - jbay808 4y ago
        That's a very good, and deep, point, and I agree that there is something important there. However, algorithmic complexity is still only defined relative to an arbitrary reference computer. If my reference computer happens to, in hardware, XOR its inputs and outputs with the bitstring 10010001111110101000, then (in terms of bits that end up represented on the drive), that will be the one that has the lowest algorithmic complexity, although the algorithm might think that it's outputting all 0's.

oh_my_goodness 4y ago

Here's a concrete example of entropy with just two macro-states, 'broken' and 'unbroken'. If it becomes unclear or unconvincing, can you point out where that happens? It's intended to be ELI5, clear enough to discuss coherently. Question: Why is it when I drop a vase it smashes into a million pieces; however when I then drop the million pieces it does not form a vase? Answer: Stop! Don't drop any more expensive vases. Start with these simpler systems that do repair themselves sometimes when you drop them. Take a coin and align it so the 'heads' side faces up. 'Heads' means 'unbroken'. (The reason for that will become clearer as we do more experiments.) Now drop the coin on the floor. How often is the 'heads' side still facing up? Now if you drop it again, how often does it 'repair' itself so that the 'heads' side is up? (Really do this.) Try the experiment with 2 coins. Align them all heads-up, drop them, then see if your pattern is 'broken'. ('Broken' means not-all-heads-up.) Drop the 'broken' coins again. How often do they 'repair' themselves? ('Repaired' means all heads-up.) ( Don't think about it! Don't solve for it! Do it! ) Try again with 5 coins. How often does a 5-coin system 'break' when you drop it? How often does a broken 5-coin system 'repair itself' when you drop it again? How about 10 coins? How often does a broken pattern of 10 mixed heads/tails repair itself to all heads when you drop it again? Sometimes it does, but you'll have to be very lucky or patient to see it happen. I think from here you can probably see (part of) the answer to your question about the vase. The word people use for this kind of thing is 'entropy'. With enough coins, the 'broken' state is much more probable than the 'repaired' state. The log of a probability is called 'entropy.' https://www.quora.com/Why-is-it-when-I-drop-a-vase-it-smashes-into-a-million-pieces-however-when-I-then-drop-the-million-pieces-it-does-not-form-a-vase/answer/Tom-McNamara-4 https://www.quora.com/Why-is-it-when-I-drop-a-vase-it-smashe...

sixo 4y ago

> I don't know what Entropy is supposed to mean on the level of individual states/configurations. I don't understand what kind of macroscopic "averaging" function we may use to group up those states I find it helpful to think of entropy as a property of not of the system, or any individual state (micro- or macro-), but as a property of the "compression" process that summarizes microstates with a coarser-grained macro-description. Given a choice of compression, classical physics says a system will tend to spent most of their time in the most likely compressed state. Different choices of compressions can lead to different macro descriptions, with different "entropies" and different dynamics among their macroscopic variables. In this light it's not meaningful to think of the entropy of individual states. You could think about the "identity" compression, but you would end up with a description that was exactly as complicated as the full micro-state time-evolution dynamics; you wouldn't end up with any smaller set of variables that could describe the equilibrium of the whole system (really this would not admit an "equilibrium" at all)

deltaonefour 4y ago

Another thing that contributes to the confusion which you have noted but not fully realized is that there are two different concepts that use the same equation and the same word: "entropy". Information entropy and statistical entropy are two different things.

deleted 4y ago
[deleted]

raxxorraxor 4y ago

In computer science a high entropy means a high informational value. So any information that is not the expected value has a high content of information and therefore a high entropy. In thermodynamics the case is a bit to the opposite as a high entropy means a low state of energy and therefore less internal processes within a system or none at all.

foobarqux 4y ago

For information theoretic entropy (I don’t know anything about thermodynamics): The first thing you describe, that if you draw many symbols from your source distribution at random then you see a distribution of symbols that is equal to the probability distribution of the source, is called the asymptotic equipartition principle. I think your confusion comes from two things. First, conflating bits and symbols and second assuming symbols are equiprobable. Take English text where our symbols could be letters of the alphabet. These are not equiprobable, if you select a letter from a book at random you get a different distribution than 1/26. If you took as your symbols the individual bits that would encode those characters in ascii you would get something closer to equal probable symbols. Another choice for symbols would be the words in the book.

keithlfrost 4y ago

This is a very sensible confusion. The forms of macroscopic averaging functions which are useful and valid cannot be made up arbitrarily, but are determined by the microscopic physical laws of the system. There is a reason that the law of increase of entropy is the second law of classical thermodynamics, with conservation of energy being the first law. To state it explicitly: energy is a globally conserved quantity, which can be freely exchanged among the interacting microscopic parts of systems. So we can bring a test system (called a thermometer) into interaction with our system under study, (indirectly) observe the average energy per degree of freedom of the thermometer, and call that observation the temperature of the system under study. Similarly, it is a known physical phenomenon that a gas confined to a container will exert a steady average outward force per normal unit area on the walls of the container; we have ways to measure this force, and we call it pressure. And so on, and on: every useful macroscopic averaging function is a relatively stable, measurable quantity which is determined by the physics of the systems under study. If we discovered some new measurement technique tomorrow which enabled us to measure the "quintessence" of physical systems, and this measurement was stable and reproducible, and could be meaningfully aggregated from the microscopic parts of the system and measured on the macroscopic scale, our definition of entropy would change, to account for "quintessence".

jl2718 4y ago

A specific string of bits has zero entropy. It may be a sample from a distribution which has some entropy. Unless you are selecting a random single bit from that string, in which the entropy of that selection process is -p1log p1 - p0log p0.

lambdatronics 4y ago

>I don't know what Entropy is supposed to mean on the level of individual states/configurations. The entropy is a property of a probability distribution, not of a state. Entropy is defined as H = -sum(p_i log(p_i)). A 'state' implicitly defines a probability distribution: uniform probability over all the microstates compatible with the state description.[0] In the case of a microstate, the entropy of the probability distribution over microstates consistent with that state is zero - there's only one compatible state, so p_i = 0 for all other states, and log(p_i) = 0 for the compatible state. In the case of a macrostate, the entropy of the probability distribution over microstates consistent with the macrostate works out to -sum((1/N) log(1/N)) = log(N), where N is the number of consistent microstates. That's the Boltzmann entropy. Sometimes people will write about the entropy of a 'state' in such a way that it sounds like they're talking about the entropy of a microstate -- but what they're probably talking about is "the entropy of the macrostate that this microstate belongs to." It's sloppy to talk like that, because "the" corresponding macrostate isn't unique. There are many sets of macrostates that could contain a microstate, depending on what properties of the microstates one considers 'macro.' (Ex: 10100101 is a member of both "symmetric bit strings of length 8" and "bit strings of length 8 that average to 1/2". The entropy of "symmetric bit strings of length 8" is 4 bits, whereas the entropy of "bit strings of length 8 that average to 1/2" is ~6.1 bits. And of course, the entropy of "the bit string of length 8 that is exactly 10100101" is zero.)

esalman 4y ago

High entropy = less predictable system. Low entropy = highly predictable system.