3 ms·

What's not touched on in this article and arguably explains the ubiquity of the Ising model: it's the maximum entropy model given a mean and pairwise correlatio

What's not touched on in this article and arguably explains the ubiquity of the Ising model: it's the maximum entropy model given a mean and pairwise correlations

bencw 6y ago
This is a general fact about Gibbs distributions!
edna314 6y ago
Which entropy does “maximum entropy” refer to here? The entropies of the spins? Wouldn’t this entropy depend on temperature and coupling strength since there is a phase transition?
kgwgk 6y ago
That doesn’t make much sense to me. The model is a model (coupled spins in a heat bath). The “solution” (the equilibrium states) can be found using maximum entropy considerations but that’s not unusual. What makes the Ising model popular is that’s simple and interesting.
- andbberger 6y ago
  Information entropy, not statistical mechanical entropy [1], which is a sort of Occam's razor for models. I would not describe the Ising model as simple, not even in the 2d case and especially not the general spin glass (where W_ij can be anything). [1] https://en.wikipedia.org/wiki/Principle_of_maximum_entropy https://en.wikipedia.org/wiki/Principle_of_maximum_entropy
  - kgwgk 6y ago
    That link has a couple of sentences about “maximum entropy models” and no references. From a quick search it seems that this is a terminology used in some areas thought. It’s not yet completely clear to me what does it mean. (Seems to be about the Ising model which is selected from a large set of ‘similar” Ising models; I would think of “Ising model” as the general thing.) Statistical mechanical entropy is information entropy (and related to thermodynamical entropy). From the link: “The principle was first expounded by E. T. Jaynes in two papers in 1957[1][2] where he emphasized a natural correspondence between statistical mechanics and information theory. In particular, Jaynes offered a new and very general rationale why the Gibbsian method of statistical mechanics works. He argued that the entropy of statistical mechanics and the information entropy of information theory are basically the same thing. Consequently, statistical mechanics should be seen just as a particular application of a general tool of logical inference and information theory.” Simplicity is a relative thing, I guess. Let’s say the interestingness/complexity ratio is high.
    - andbberger 6y ago
      Statistical mechanical (Boltzmann) entropy and informational (Shannon) entropy are not the same thing, but they are closely related. Subtleties are important. The Shannon entropy of the distribution of the positions and velocities is precisely the Boltzmann entropy of the same, but there is no statistical mechanical analog to the Shannon entropy of a (non-Boltzmann) distribution, to my knowledge.
      - kgwgk 6y ago
        > Statistical mechanical (Boltzmann) entropy While Boltzmann entropy was first, and an important breakthrough, it has some limitations. Statistical mechanics is actually based on Gibbs/von Neumann entropy, which is equivalent to Shannon entropy. There are many subtleties indeed but I think the relationship between statistical mechanics and information theory may be closer than you think.
  - kgwgk 6y ago
    Btw, I looked at https://arxiv.org/pdf/1509.03808.pdf https://arxiv.org/pdf/1509.03808.pdf and I don’t understand this paragraph at the end of the first page: “To simulate the system, waiting times w_kj are generated for all candidate states k, and the shortest waiting time i = argmin_k w_kj is chosen. We call this shortest waiting time the holding time. A transition is then performed to state i after a delay of length w_ij.” Say there are three states A, B, C and the transition rate to C from A or B is low. Wouldn’t your “simulation” jump endlessly between A and B without ever (or never again) visiting C? I may be missing something.
    - andbberger 6y ago
      Oh god don't read that... The transition rates for each individual state transition are drawn from an exponential distribution with a rate dependent on the energy difference between the states. So, no, C will be visited with some probability. There are many problems with that paper, but validity of the sampling algorithm is not one of them. It is an unbiased sampler and will eventually converge to the sampled distribution (as proved in the paper). How long that convergence will take is another store. It remains unclear whether this approach converges faster than your garden-variety Markov chain and if so when and why.
      - kgwgk 6y ago
        Thanks for your answer. Sorry, I don’t why i did interpret “waiting times” as “expected waiting times” when it was right in the preceding paragraph how they were sampled.
        andbberger 6y ago
        Nothing to apologize for

What's not touched on in this article and arguably explains the ubiquity of the Ising model: it's the maximum entropy model given a mean and pairwise correlations

bencw 6y ago

This is a general fact about Gibbs distributions!

edna314 6y ago

Which entropy does “maximum entropy” refer to here? The entropies of the spins? Wouldn’t this entropy depend on temperature and coupling strength since there is a phase transition?

kgwgk 6y ago

That doesn’t make much sense to me. The model is a model (coupled spins in a heat bath). The “solution” (the equilibrium states) can be found using maximum entropy considerations but that’s not unusual. What makes the Ising model popular is that’s simple and interesting.

andbberger 6y ago
Information entropy, not statistical mechanical entropy [1], which is a sort of Occam's razor for models. I would not describe the Ising model as simple, not even in the 2d case and especially not the general spin glass (where W_ij can be anything). [1] https://en.wikipedia.org/wiki/Principle_of_maximum_entropy https://en.wikipedia.org/wiki/Principle_of_maximum_entropy
- kgwgk 6y ago
  That link has a couple of sentences about “maximum entropy models” and no references. From a quick search it seems that this is a terminology used in some areas thought. It’s not yet completely clear to me what does it mean. (Seems to be about the Ising model which is selected from a large set of ‘similar” Ising models; I would think of “Ising model” as the general thing.) Statistical mechanical entropy is information entropy (and related to thermodynamical entropy). From the link: “The principle was first expounded by E. T. Jaynes in two papers in 1957[1][2] where he emphasized a natural correspondence between statistical mechanics and information theory. In particular, Jaynes offered a new and very general rationale why the Gibbsian method of statistical mechanics works. He argued that the entropy of statistical mechanics and the information entropy of information theory are basically the same thing. Consequently, statistical mechanics should be seen just as a particular application of a general tool of logical inference and information theory.” Simplicity is a relative thing, I guess. Let’s say the interestingness/complexity ratio is high.
  - andbberger 6y ago
    Statistical mechanical (Boltzmann) entropy and informational (Shannon) entropy are not the same thing, but they are closely related. Subtleties are important. The Shannon entropy of the distribution of the positions and velocities is precisely the Boltzmann entropy of the same, but there is no statistical mechanical analog to the Shannon entropy of a (non-Boltzmann) distribution, to my knowledge.
    - kgwgk 6y ago
      > Statistical mechanical (Boltzmann) entropy While Boltzmann entropy was first, and an important breakthrough, it has some limitations. Statistical mechanics is actually based on Gibbs/von Neumann entropy, which is equivalent to Shannon entropy. There are many subtleties indeed but I think the relationship between statistical mechanics and information theory may be closer than you think.
- kgwgk 6y ago
  Btw, I looked at https://arxiv.org/pdf/1509.03808.pdf https://arxiv.org/pdf/1509.03808.pdf and I don’t understand this paragraph at the end of the first page: “To simulate the system, waiting times w_kj are generated for all candidate states k, and the shortest waiting time i = argmin_k w_kj is chosen. We call this shortest waiting time the holding time. A transition is then performed to state i after a delay of length w_ij.” Say there are three states A, B, C and the transition rate to C from A or B is low. Wouldn’t your “simulation” jump endlessly between A and B without ever (or never again) visiting C? I may be missing something.
  - andbberger 6y ago
    Oh god don't read that... The transition rates for each individual state transition are drawn from an exponential distribution with a rate dependent on the energy difference between the states. So, no, C will be visited with some probability. There are many problems with that paper, but validity of the sampling algorithm is not one of them. It is an unbiased sampler and will eventually converge to the sampled distribution (as proved in the paper). How long that convergence will take is another store. It remains unclear whether this approach converges faster than your garden-variety Markov chain and if so when and why.
    - kgwgk 6y ago
      Thanks for your answer. Sorry, I don’t why i did interpret “waiting times” as “expected waiting times” when it was right in the preceding paragraph how they were sampled.
      - andbberger 6y ago
        Nothing to apologize for