4 ms·

I've found that reading more into these 'accidental' similarities is a lot more useful than not reading into it at all. For what it's worth there is a paramete

I've found that reading more into these 'accidental' similarities is a lot more useful than not reading into it at all. For what it's worth there is a parameter in front that functions as a kind of temperature, except there's more than one parameter so it's not an exact translation. You can still reason about it somewhat (or split it up into a single temperature times a convex combination of hamiltonians).

versteegen 3y ago
Sure, there are fruitful parallels. But actually, energy is much 'realer' or more fundamental here than temperature. BTW, those λ_j Lagrangian multipliers can be negative, so don't map onto inverse temperature well. They're more like interaction weights in a Hopfield network. But there's one setting in which the energy E(x) = Σ_j λ_j H_j(x) really does become a physical (physics-esque) energy: if you sample from the distribution using Hamiltonian Monte Carlo (an MCMC algorithm). Then E(x) is the potential energy of a sample/particle at position x, which is also given a kinetic energy K with the sume E+K held constant as the particle moves about to explore the distribution. It's a cool algorithm but after having used it I'm still not sure how useful it is.
- shiandow 3y ago
  HMC allows you to jump quite a large distance by obeying the 'equations of motion' of the 'Hamiltonian'. This helps give more diverse samples, making them less correlated thereby increasing number the effective number of samples. It is indeed quite cool how it works.
  - versteegen 3y ago
    Right, but I mean that HMC has parameters to tune and multiples the computation needed per sample produced so it's not clear to me for which problems it's significantly beneficial and with how much tuning. I'd guess it's best where you otherwise don't have a good proposal distribution for Metropolis-Hastings leading to either many rejects or very slow mixing.
    - shiandow 3y ago
      The NUTS sampler limits how much tuning you need. As far as I know it's currently one of the best algorithms for sampling the kind of posterior probability distributions you get with Bayesian methods.

versteegen 3y ago

Sure, there are fruitful parallels. But actually, energy is much 'realer' or more fundamental here than temperature. BTW, those λ_j Lagrangian multipliers can be negative, so don't map onto inverse temperature well. They're more like interaction weights in a Hopfield network. But there's one setting in which the energy E(x) = Σ_j λ_j H_j(x) really does become a physical (physics-esque) energy: if you sample from the distribution using Hamiltonian Monte Carlo (an MCMC algorithm). Then E(x) is the potential energy of a sample/particle at position x, which is also given a kinetic energy K with the sume E+K held constant as the particle moves about to explore the distribution. It's a cool algorithm but after having used it I'm still not sure how useful it is.

shiandow 3y ago
HMC allows you to jump quite a large distance by obeying the 'equations of motion' of the 'Hamiltonian'. This helps give more diverse samples, making them less correlated thereby increasing number the effective number of samples. It is indeed quite cool how it works.
- versteegen 3y ago
  Right, but I mean that HMC has parameters to tune and multiples the computation needed per sample produced so it's not clear to me for which problems it's significantly beneficial and with how much tuning. I'd guess it's best where you otherwise don't have a good proposal distribution for Metropolis-Hastings leading to either many rejects or very slow mixing.
  - shiandow 3y ago
    The NUTS sampler limits how much tuning you need. As far as I know it's currently one of the best algorithms for sampling the kind of posterior probability distributions you get with Bayesian methods.