3 ms·

Yes, but... > some kind of energy The negenergy (negative energy) here is just the (unnormalised, hence the 1/Z term) log-probability mass or density. Like th

Yes, but... > some kind of energy The negenergy (negative energy) here is just the (unnormalised, hence the 1/Z term) log-probability mass or density. Like the logit outputs of classifier. You shouldn't read anything more into the 'energy'. There is no temperature parameter in these minimax models (although adding one to smooth the model during sampling or learning can be useful, to interpolate between pure noise and the learnt model).

shiandow 2y ago
I've found that reading more into these 'accidental' similarities is a lot more useful than not reading into it at all. For what it's worth there is a parameter in front that functions as a kind of temperature, except there's more than one parameter so it's not an exact translation. You can still reason about it somewhat (or split it up into a single temperature times a convex combination of hamiltonians).
- versteegen 2y ago
  Sure, there are fruitful parallels. But actually, energy is much 'realer' or more fundamental here than temperature. BTW, those λ_j Lagrangian multipliers can be negative, so don't map onto inverse temperature well. They're more like interaction weights in a Hopfield network. But there's one setting in which the energy E(x) = Σ_j λ_j H_j(x) really does become a physical (physics-esque) energy: if you sample from the distribution using Hamiltonian Monte Carlo (an MCMC algorithm). Then E(x) is the potential energy of a sample/particle at position x, which is also given a kinetic energy K with the sume E+K held constant as the particle moves about to explore the distribution. It's a cool algorithm but after having used it I'm still not sure how useful it is.
  - shiandow 2y ago
    HMC allows you to jump quite a large distance by obeying the 'equations of motion' of the 'Hamiltonian'. This helps give more diverse samples, making them less correlated thereby increasing number the effective number of samples. It is indeed quite cool how it works.
    - versteegen 2y ago
      Right, but I mean that HMC has parameters to tune and multiples the computation needed per sample produced so it's not clear to me for which problems it's significantly beneficial and with how much tuning. I'd guess it's best where you otherwise don't have a good proposal distribution for Metropolis-Hastings leading to either many rejects or very slow mixing.
      - shiandow 2y ago
        The NUTS sampler limits how much tuning you need. As far as I know it's currently one of the best algorithms for sampling the kind of posterior probability distributions you get with Bayesian methods.

shiandow 2y ago

I've found that reading more into these 'accidental' similarities is a lot more useful than not reading into it at all. For what it's worth there is a parameter in front that functions as a kind of temperature, except there's more than one parameter so it's not an exact translation. You can still reason about it somewhat (or split it up into a single temperature times a convex combination of hamiltonians).

versteegen 2y ago
Sure, there are fruitful parallels. But actually, energy is much 'realer' or more fundamental here than temperature. BTW, those λ_j Lagrangian multipliers can be negative, so don't map onto inverse temperature well. They're more like interaction weights in a Hopfield network. But there's one setting in which the energy E(x) = Σ_j λ_j H_j(x) really does become a physical (physics-esque) energy: if you sample from the distribution using Hamiltonian Monte Carlo (an MCMC algorithm). Then E(x) is the potential energy of a sample/particle at position x, which is also given a kinetic energy K with the sume E+K held constant as the particle moves about to explore the distribution. It's a cool algorithm but after having used it I'm still not sure how useful it is.
- shiandow 2y ago
  HMC allows you to jump quite a large distance by obeying the 'equations of motion' of the 'Hamiltonian'. This helps give more diverse samples, making them less correlated thereby increasing number the effective number of samples. It is indeed quite cool how it works.
  - versteegen 2y ago
    Right, but I mean that HMC has parameters to tune and multiples the computation needed per sample produced so it's not clear to me for which problems it's significantly beneficial and with how much tuning. I'd guess it's best where you otherwise don't have a good proposal distribution for Metropolis-Hastings leading to either many rejects or very slow mixing.
    - shiandow 2y ago
      The NUTS sampler limits how much tuning you need. As far as I know it's currently one of the best algorithms for sampling the kind of posterior probability distributions you get with Bayesian methods.