5 ms·

I studied computer vision in grad school when these were some of the state of the art methods, like the Mumford-Shah segmentation model and various inpainting m

by mlthoughts2018 8y ago

I studied computer vision in grad school when these were some of the state of the art methods, like the Mumford-Shah segmentation model and various inpainting models that set up the problem basically as a calculus of variations problem over the appropriate Sobolev space of image intensity functions and/or discontinuity-permitting boundary functions. I’ve worked as a practitioner in computer vision as well and I think I disagree with your “under taught” comment. In practice, the variational methods are not competitive at all with deep learning models, and even very naive transfer learning can apply better inpainting or segmentation than the older variational methods. It is already merely a footnote in the history of computer vision. In 10 years it won’t even merit being part of literature reviews. So for many applied domains of ML, I think spending roughly zero time on these variational methods in schooling is a good idea. You can always come back to them later, but you probably won’t need to. A different application domain, variational Bayesian inference, might be a case where there is more reason to care. Although, things like ADVI are more about one very fixed approach to the problem (minimizing KL divergence to find a best approximating distribution from a class with known differentiability properties), and only in novel research would knowledge of general variational techniques give much value.

ur-whale 8y ago
>In practice, the variational methods are not competitive at all with deep learning models I really don't see why these two things need be pitted against each others. What you're describing (Euler-lagrange applied to computer vision problems) is a direct application of the calculus of variation to the image itself. It's rather naive because it uses an extremely weak representation of the content of the image (i.e. a bunch of pixels which essentially means working with a piecewise-constant basis of the functional space), and as you noted, it is easily beaten by modern techniques. However: Deep neural nets are nothing but another, much more powerful "basis" of functional space (I put "basis" in quote because they're not linear. OTOH, they're generative). A deep net is a finite set of real numbers that maps to a point in functional space, and any arbitrary continuous maps from R^n to R^p can be approximated by deep nets given enough weights and neurons. So, if deep nets are dense in functional space, and "training" a deep net is nothing but solving a minimization problem in functional space, then I don't see why the "calculus of variation" ==> "PDE" via Euler-Lagrange route can't bee applied to training deep nets instead of backprop-based minimizers. Now, if you're claiming that backprop works better than solving the PDE induced by Euler-Lagrange on the functional minimization problem of training a deep-net, I'm very happy to believe you, but I've yet to see literature demonstrating this.
- mlthoughts2018 8y ago
  The variational representations used in image processing are much more complicated than you suggest. In many cases, they are developed with semantic models that inform grammars of what the image can be composed of, and directly model things like occlusion, cusps, edges, blur, etc., when postulating a joint functional space that can describe the “energy” of an image, which then minimizing gives not only pixel values but also parametric curves for boundaries and sometimes also labels. It’s essentially a way to encode semantics as prior distributions over possible images, then variational methods “optimize” for the posterior parameters given some data. I’m trying to say that that whole exercise listed above, conceptually, is basically a failed idea compared with solving the same problems using deep learning to automatically discover what that manifold looks like in parameter space and transform it directly to the output space of practical interest (segmentation boundaries, etc.).
  - ur-whale 8y ago
    >The variational representations used in image processing are much more complicated than you suggest Fair enough, but my main point stands: has anyone tried to apply Euler-Lagrange and a PDE solver to the problem of training a deep net or is it obvious that backprop-based methods will work better?
    - enriquto 8y ago
      not exactly the same thing, but here you have a "Deep Neural Networks motivated by Partial Differential Equations", that may pique your interest https://arxiv.org/abs/1804.04272 https://arxiv.org/abs/1804.04272

ur-whale 8y ago

>In practice, the variational methods are not competitive at all with deep learning models I really don't see why these two things need be pitted against each others. What you're describing (Euler-lagrange applied to computer vision problems) is a direct application of the calculus of variation to the image itself. It's rather naive because it uses an extremely weak representation of the content of the image (i.e. a bunch of pixels which essentially means working with a piecewise-constant basis of the functional space), and as you noted, it is easily beaten by modern techniques. However: Deep neural nets are nothing but another, much more powerful "basis" of functional space (I put "basis" in quote because they're not linear. OTOH, they're generative). A deep net is a finite set of real numbers that maps to a point in functional space, and any arbitrary continuous maps from R^n to R^p can be approximated by deep nets given enough weights and neurons. So, if deep nets are dense in functional space, and "training" a deep net is nothing but solving a minimization problem in functional space, then I don't see why the "calculus of variation" ==> "PDE" via Euler-Lagrange route can't bee applied to training deep nets instead of backprop-based minimizers. Now, if you're claiming that backprop works better than solving the PDE induced by Euler-Lagrange on the functional minimization problem of training a deep-net, I'm very happy to believe you, but I've yet to see literature demonstrating this.

mlthoughts2018 8y ago
The variational representations used in image processing are much more complicated than you suggest. In many cases, they are developed with semantic models that inform grammars of what the image can be composed of, and directly model things like occlusion, cusps, edges, blur, etc., when postulating a joint functional space that can describe the “energy” of an image, which then minimizing gives not only pixel values but also parametric curves for boundaries and sometimes also labels. It’s essentially a way to encode semantics as prior distributions over possible images, then variational methods “optimize” for the posterior parameters given some data. I’m trying to say that that whole exercise listed above, conceptually, is basically a failed idea compared with solving the same problems using deep learning to automatically discover what that manifold looks like in parameter space and transform it directly to the output space of practical interest (segmentation boundaries, etc.).
- ur-whale 8y ago
  >The variational representations used in image processing are much more complicated than you suggest Fair enough, but my main point stands: has anyone tried to apply Euler-Lagrange and a PDE solver to the problem of training a deep net or is it obvious that backprop-based methods will work better?
  - enriquto 8y ago
    not exactly the same thing, but here you have a "Deep Neural Networks motivated by Partial Differential Equations", that may pique your interest https://arxiv.org/abs/1804.04272 https://arxiv.org/abs/1804.04272