3 ms·
Physics is also great for machine learning, though the approaches can be rather unintuitive. For example message passing and belief propagation in trees/graphs
by Onavo 2y ago
Physics is also great for machine learning, though the approaches can be rather unintuitive. For example message passing and belief propagation in trees/graphs (Bayesian networks, Markov random fields etc.) for modeling latent variables are usually taught using the window/rainy weather marginal probability analogy and involves splitting out a bayesian/statistical equation into subcomponents via the marginalization chain rule. For physicists however, they tend to teach it using Ising models and magnetic spin, which is a totally different analogy.
A lot of the newer generative ML models are also using differential equations/Boltzmann distribution based approaches (state space models, "energy based" models) where the statistical formulations are cribbed wholesale from statistical physics/mechanics and then plugged into a neural network and autodiff system.
The best example is probably the Metropolis-Hastings algorithm which was invented by nuke people.
https://web.archive.org/web/20150603234436/http://flynnmichael.com/2015/06/01/my-favorite-algorithm-metropolis-hastings/ https://web.archive.org/web/20150603234436/http://flynnmicha...
- whimsicalism 2y agoWhat modern ML uses those techniques? my understanding is ebm is quite rare
- Onavo 2y agoResearch mostly, lots of these techniques are used for speech synthesis and occasionally image models (not LLMs)
- hangsi 2y agoThe common method for choosing the next output token for an LLM is sampling from a Boltzmann distribution. If you have seen the term "temperature" in the context of language models, that is a direct link to the statistical gas mechanics.
- whimsicalism 2y agoi don't find the connection between softmax and boltzmann really all that deep tbh (compared to say, the connection between field theory/ising models and EBM)
- scarmig 2y agoAnother is the Nakano-Amari-Hopfield model, which is based on Ising. Hopfield himself was a trained physicist.
- janalsncm 2y agoOne that many people may be familiar with is Stable Diffusion, which is used in many AI image generators today. There is an analogy between random noise -> image and a random distribution of gas particles -> concentrated volume of particles. https://arxiv.org/abs/1503.03585 https://arxiv.org/abs/1503.03585