4 ms·
From the abstract of the article they linked: Energy-Based Models (EBMs) capture dependencies between variables by associating a scalar energy to each configur
by nerdponx 8y ago
From the abstract of the article they linked:
Energy-Based Models (EBMs) capture dependencies between variables by associating a scalar energy to each configuration of the variables. Inference consists in clamping the value of observed variables and finding configurations of the remaining variables that minimize the energy. Learning consists in finding an energy function in which observed configurations of the variables are given lower energies than unobserved ones. The EBM approach provides a common theoretical framework for many learning models, including traditional discriminative and generative approaches, as well as graph-transformer networks, conditional random fields, maximum margin Markov networks, and several manifold learning methods.
Probabilistic models must be properly normalized, which sometimes requires evaluating intractable integrals over the space of all possible variable configurations. Since EBMs have no requirement for proper normalization, this problem is naturally circumvented. EBMs can be viewed as a form of non-probabilistic factor graphs, and they provide considerably more flexibility in the design of architectures and training criteria than probabilistic approaches.
Seems like a really interesting unification of the wide variety of techniques out there in statistics and machine learning, analogous to the "everything is a computation graph, as long as it's differentiable" revolution. I like it when this kind of thing has its day. Would be interesting to see how well it works non-robotics problems.
- deleted 8y ago[deleted]
- BucketSort 8y agoSee Yann's tutorial on EBM: http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf We actually ran into them when doing research in our startup. It is a really powerful perspective.
- lucidrains 8y agoThank you for this!
- eli_gottlieb 8y ago>Probabilistic models must be properly normalized, which sometimes requires evaluating intractable integrals over the space of all possible variable configurations. Since EBMs have no requirement for proper normalization, this problem is naturally circumvented. EBMs can be viewed as a form of non-probabilistic factor graphs, and they provide considerably more flexibility in the design of architectures and training criteria than probabilistic approaches. Can someone explain to me what the major difference is between energy-based models and variational Bayes approximations, which throw out calculating the normalization constant and switch to maximizing the log joint probability of the data and model?