6 ms·

Geometric Understanding of Deep Learning (2018)

164 points by yoquan 8y ago

yoquan 8y ago
I'm reading this and just realized one author is the famous Field medalist, Shing-Tung Yau :-)
- soVeryTired 8y ago
  Sorta like when David Mumford got interested in computer vision, I guess.
  - jesuslop 8y ago
    you' kiddin?
    - soVeryTired 8y ago
      Nope: http://www.dam.brown.edu/people/mumford/vision/introvision.html http://www.dam.brown.edu/people/mumford/vision/introvision.h... Tim Gowers is interested in automated theorem proving too!
      - jesuslop 8y ago
        Surprising. Voeovdski also.
        jesuslop 8y ago
        Also positioned as not anti proof assistants I meant.
jesuslop 8y ago
Possibly less sophisticatedly I think of them as a sandwich of affine maps and nonlinear isotropies (as those giving irregular rings in tree trunks). The affinities are represented nicely in GL(n+1) with a homogenous coordinates trick related to neuron biases. A question would be if there's something interesting to say about the interactions of the affinities and isotropies in group theoretic terms (which I dunno).
- MAXPOOL 8y ago
  ReLU is very simple in this regard. In the plain form it's just affine transformation followed by 'a viewport'. The mapping trough multiple layers is alternating affine transformations and windows into data. Learning is combination of squeezing and rotating data so that it can be seen or unseen from the window and rotating window frames to do the same. Any extra stuff, like batch normalization between the layers can again introduce more complex nonlinearity.
  - jesuslop 8y ago
    Really agree with the extra simplicity in the ReLU case. Liwen Zhang, Gregory Naitzat and Lek-Heng Lim showed last year that "the family of such neural networks is equivalent to the family of tropical rational maps", where rational functions are quotient of polynomials and "tropical" is in the context of Tropical Geometry, where instead of the typical "plus, times" ring one uses a "min, plus" ring that has somewhat unexpected applications. For instance, with its module theory one can calculate minimum costs paths in graphs just as one does for reachability starting with the adjacency matrix of a graph over a boolean semiring. arXiv:1805.07091v1
heinrichf 8y ago
(uploaded to the arXiv in May 2018)
- sctb 8y ago
  Thanks! We've updated the headline.
lovelearning 8y ago
[An OT question as somebody not familiar with the academic world] Two of the authors are in a Chinese university. Two of them in different departments in a US university. In general, how does this kind of intercontinental collaboration start, and how do they progress? How are roles defined when multiple people are involved in a theoretical paper like this? Are there some tools that help with collaborative paper writing?
- rhema 8y ago
  Lots of ways. Typically people meet through conferences, where high-impact work is published. It's also not uncommon for researchers to cold-email each other based on mutual interest or friend-of-friend connection. Things like Google Scholar can be used to get emails about when your papers get cited, so this can spur some discussion / collaboration as well. Honestly, you could email an author on this paper and they would probably tell you directly.
- auntienomen 8y ago
  Yau is the connection. He's a professor at Harvard, but he's also been a major force in the development of Chinese mathematical academic institutions. He's the director of several of them.
  - randcraw 8y ago
    And Gu got his PhD in CS at Harvard in computational geometry. Yau teaches differential geometry there.
throwawaymath 8y ago
> ...we show that the fundamental principle attributing to the success is the manifold structure in data... > Then we show for any deep neural network with fixed architecture, there exists a manifold that cannot be learned by the network. I'd venture a guess that you can extend this result to show that, for any deep neural network with fixed architecture, there exists an adversarial manifold it must be vulnerable to. In other words not only is there a manifold the neural network cannot learn, but there is also a manifold it will learn, but incorrectly.
cs702 8y ago
Wow. As far as I know, this is the first time anyone reputable[a] has claimed to show (!) that the "manifold hypothesis" is the fundamental principle that makes deep learning work, as has long been believed: "In this work, we give a geometric view to understand deep learning: we show that the fundamental principle attributing to the success is the manifold structure in data, namely natural high dimensional data concentrates close to a low-dimensional manifold, deep learning learns the manifold and the probability distribution on it." Moreover, the authors also claim to have come up with a way of measuring how hard it is for any deep neural net (of fixed size) to learn a parametric representation of a particular lower-dimensional manifold embedded in some higher-dimensional space: "We further introduce the concepts of rectified linear complexity for deep neural network measuring its learning capability, rectified linear complexity of an embedding manifold describing the difficulty to be learned. Then we show for any deep neural network with fixed architecture, there exists a manifold that cannot be learned by the network." Finally, the authors also propose a novel way to control the probability distribution in the latent space. I'm curious to see how their method compares and relates to recent work, e.g., with discrete and continuous normalizing flows: "...we propose to apply optimal mass transportation theory to control the probability distribution in the latent space." This is not going to be a light read... -- [a] One of the authors, Shing-Tung Yau, is a Fields medalist: https://news.ycombinator.com/item?id=18987219 https://news.ycombinator.com/item?id=18987219
- siavosh 8y ago
  I've been out of academia for a long time, but isn't the notion that 'natural' high dimensional data lies on low dimensional manifolds the basic premise of all machine learning techniques?
  - xtacy 8y ago
    Yes, the authors mention it in the introduction as a "well accepted manifold assumption."
  - cs702 8y ago
    Yes, of course, this has long been a widely held assumption. That's why we call it the "manifold hypothesis." As far as I know, no one has been able to show -- with mathematical rigor -- that this is why deep learning works so well on so many challenging perceptual ("cognitive") tasks. That seems significant to me.
crimsonalucard 8y ago
Anybody know what I should study in order to understand this research paper?
- dragqueen 8y ago
  Looks like you need to know a tiny little bit about manifolds, measure theory/probability theory, topology and more importantly have the requisite "math maturity". For very easy intro to manifolds and measure you could probably take a look into [0] A Visual Introduction to Differential Forms and Calculus on Manifolds by Fortney and [1] The Lebesgue Integral for Undergraduates by Johnston. [0] https://www.amazon.com/Visual-Introduction-Differential-Calculus-Manifolds/dp/3319969919/ref=sr_1_1?s=books&ie=UTF8&qid=1548353414&sr=1-1&keywords=fortney+manifolds https://www.amazon.com/Visual-Introduction-Differential-Calc... [1] https://www.amazon.com/Lebesgue-Integral-Undergraduates-MAA-Textbooks/dp/1939512077/ref=sr_1_6?s=books&ie=UTF8&qid=1548353478&sr=1-6&keywords=lebesgue+integration https://www.amazon.com/Lebesgue-Integral-Undergraduates-MAA-...
  - mlevental 8y ago
    yea despite the purported firepower of the authors this is not a dense paper
    - AnimalMuppet 8y ago
      Maybe that shows most clearly the firepower of the authors...
      - improbable22 8y ago
        I'd reject it. I'm honestly struggling to find anything at all to chew on here...
twic 8y ago
I know next to nothing about deep learning. But this geometric interpretation really reminds of of the way self-organising maps work. Is there a real connection there, or is that superficial?
- FakeComments 8y ago
  I may be wrong, having just heard of self-organizing maps... But they do seem related, in that it’s arguing the data can always be (mostly) accurately represented by a map projecting onto a lower-dimension manifold. The recent paper on topological covers along with the self-organizing maps wiki seem to indicate that nothing should be harmed by doing this in a discrete setting. Essentially, a self-organizing map projecting to the R^n slices, and then learning the shape of the manifold generated by that atlas of projections.
ttflee 8y ago
Another interesting paper on optimal transportation and GAN: https://arxiv.org/abs/1710.05488 https://arxiv.org/abs/1710.05488

164 points by yoquan 8y ago

yoquan 8y ago

I'm reading this and just realized one author is the famous Field medalist, Shing-Tung Yau :-)

soVeryTired 8y ago
Sorta like when David Mumford got interested in computer vision, I guess.
- jesuslop 8y ago
  you' kiddin?
  - soVeryTired 8y ago
    Nope: http://www.dam.brown.edu/people/mumford/vision/introvision.html http://www.dam.brown.edu/people/mumford/vision/introvision.h... Tim Gowers is interested in automated theorem proving too!
    - jesuslop 8y ago
      Surprising. Voeovdski also.
      - jesuslop 8y ago
        Also positioned as not anti proof assistants I meant.

jesuslop 8y ago

Possibly less sophisticatedly I think of them as a sandwich of affine maps and nonlinear isotropies (as those giving irregular rings in tree trunks). The affinities are represented nicely in GL(n+1) with a homogenous coordinates trick related to neuron biases. A question would be if there's something interesting to say about the interactions of the affinities and isotropies in group theoretic terms (which I dunno).

MAXPOOL 8y ago
ReLU is very simple in this regard. In the plain form it's just affine transformation followed by 'a viewport'. The mapping trough multiple layers is alternating affine transformations and windows into data. Learning is combination of squeezing and rotating data so that it can be seen or unseen from the window and rotating window frames to do the same. Any extra stuff, like batch normalization between the layers can again introduce more complex nonlinearity.
- jesuslop 8y ago
  Really agree with the extra simplicity in the ReLU case. Liwen Zhang, Gregory Naitzat and Lek-Heng Lim showed last year that "the family of such neural networks is equivalent to the family of tropical rational maps", where rational functions are quotient of polynomials and "tropical" is in the context of Tropical Geometry, where instead of the typical "plus, times" ring one uses a "min, plus" ring that has somewhat unexpected applications. For instance, with its module theory one can calculate minimum costs paths in graphs just as one does for reachability starting with the adjacency matrix of a graph over a boolean semiring. arXiv:1805.07091v1

heinrichf 8y ago

(uploaded to the arXiv in May 2018)

sctb 8y ago
Thanks! We've updated the headline.

lovelearning 8y ago

[An OT question as somebody not familiar with the academic world] Two of the authors are in a Chinese university. Two of them in different departments in a US university. In general, how does this kind of intercontinental collaboration start, and how do they progress? How are roles defined when multiple people are involved in a theoretical paper like this? Are there some tools that help with collaborative paper writing?

rhema 8y ago
Lots of ways. Typically people meet through conferences, where high-impact work is published. It's also not uncommon for researchers to cold-email each other based on mutual interest or friend-of-friend connection. Things like Google Scholar can be used to get emails about when your papers get cited, so this can spur some discussion / collaboration as well. Honestly, you could email an author on this paper and they would probably tell you directly.
auntienomen 8y ago
Yau is the connection. He's a professor at Harvard, but he's also been a major force in the development of Chinese mathematical academic institutions. He's the director of several of them.
- randcraw 8y ago
  And Gu got his PhD in CS at Harvard in computational geometry. Yau teaches differential geometry there.

throwawaymath 8y ago

> ...we show that the fundamental principle attributing to the success is the manifold structure in data... > Then we show for any deep neural network with fixed architecture, there exists a manifold that cannot be learned by the network. I'd venture a guess that you can extend this result to show that, for any deep neural network with fixed architecture, there exists an adversarial manifold it must be vulnerable to. In other words not only is there a manifold the neural network cannot learn, but there is also a manifold it will learn, but incorrectly.

cs702 8y ago

Wow. As far as I know, this is the first time anyone reputable[a] has claimed to show (!) that the "manifold hypothesis" is the fundamental principle that makes deep learning work, as has long been believed: "In this work, we give a geometric view to understand deep learning: we show that the fundamental principle attributing to the success is the manifold structure in data, namely natural high dimensional data concentrates close to a low-dimensional manifold, deep learning learns the manifold and the probability distribution on it." Moreover, the authors also claim to have come up with a way of measuring how hard it is for any deep neural net (of fixed size) to learn a parametric representation of a particular lower-dimensional manifold embedded in some higher-dimensional space: "We further introduce the concepts of rectified linear complexity for deep neural network measuring its learning capability, rectified linear complexity of an embedding manifold describing the difficulty to be learned. Then we show for any deep neural network with fixed architecture, there exists a manifold that cannot be learned by the network." Finally, the authors also propose a novel way to control the probability distribution in the latent space. I'm curious to see how their method compares and relates to recent work, e.g., with discrete and continuous normalizing flows: "...we propose to apply optimal mass transportation theory to control the probability distribution in the latent space." This is not going to be a light read... -- [a] One of the authors, Shing-Tung Yau, is a Fields medalist: https://news.ycombinator.com/item?id=18987219 https://news.ycombinator.com/item?id=18987219

siavosh 8y ago
I've been out of academia for a long time, but isn't the notion that 'natural' high dimensional data lies on low dimensional manifolds the basic premise of all machine learning techniques?
- xtacy 8y ago
  Yes, the authors mention it in the introduction as a "well accepted manifold assumption."
- cs702 8y ago
  Yes, of course, this has long been a widely held assumption. That's why we call it the "manifold hypothesis." As far as I know, no one has been able to show -- with mathematical rigor -- that this is why deep learning works so well on so many challenging perceptual ("cognitive") tasks. That seems significant to me.

crimsonalucard 8y ago

Anybody know what I should study in order to understand this research paper?

dragqueen 8y ago
Looks like you need to know a tiny little bit about manifolds, measure theory/probability theory, topology and more importantly have the requisite "math maturity". For very easy intro to manifolds and measure you could probably take a look into [0] A Visual Introduction to Differential Forms and Calculus on Manifolds by Fortney and [1] The Lebesgue Integral for Undergraduates by Johnston. [0] https://www.amazon.com/Visual-Introduction-Differential-Calculus-Manifolds/dp/3319969919/ref=sr_1_1?s=books&ie=UTF8&qid=1548353414&sr=1-1&keywords=fortney+manifolds https://www.amazon.com/Visual-Introduction-Differential-Calc... [1] https://www.amazon.com/Lebesgue-Integral-Undergraduates-MAA-Textbooks/dp/1939512077/ref=sr_1_6?s=books&ie=UTF8&qid=1548353478&sr=1-6&keywords=lebesgue+integration https://www.amazon.com/Lebesgue-Integral-Undergraduates-MAA-...
- mlevental 8y ago
  yea despite the purported firepower of the authors this is not a dense paper
  - AnimalMuppet 8y ago
    Maybe that shows most clearly the firepower of the authors...
    - improbable22 8y ago
      I'd reject it. I'm honestly struggling to find anything at all to chew on here...

twic 8y ago

I know next to nothing about deep learning. But this geometric interpretation really reminds of of the way self-organising maps work. Is there a real connection there, or is that superficial?

FakeComments 8y ago
I may be wrong, having just heard of self-organizing maps... But they do seem related, in that it’s arguing the data can always be (mostly) accurately represented by a map projecting onto a lower-dimension manifold. The recent paper on topological covers along with the self-organizing maps wiki seem to indicate that nothing should be harmed by doing this in a discrete setting. Essentially, a self-organizing map projecting to the R^n slices, and then learning the shape of the manifold generated by that atlas of projections.

ttflee 8y ago

Another interesting paper on optimal transportation and GAN: https://arxiv.org/abs/1710.05488 https://arxiv.org/abs/1710.05488