7 ms·

Stanford Stats 385: Theories of Deep Learning

217 points by capocannoniere 9y ago

eduren 9y ago
Does anybody know if they plan on releasing the lecture videos? I couldn't find them on the site and this looks very interesting.
- walrus 9y ago
  They are being recorded and posted to YouTube as unlisted videos. You can find links to some of them on Twitter[1] and ResearchGate[2]. It looks like highlights of the lectures are being posted to Twitter and the full lectures are being posted to ResearchGate. [1] https://twitter.com/stats385 https://twitter.com/stats385 [2] https://www.researchgate.net/project/Theories-of-Deep-Learning https://www.researchgate.net/project/Theories-of-Deep-Learni...
  - rayuela 9y ago
    Oh those full lectures on researchgate are awsome. Thanks for sharing!
tequila_shot 9y ago
Can general public attend these sessions?
- jaytaylor 9y ago
  Technically, no, because you have to be enrolled as a student (not just a student of life ;). However, Quora seems to think you may be able to sneak in, at least sometimes: https://www.quora.com/What-does-Stanford-do-about-non-Stanford-students-sitting-in-on-classes https://www.quora.com/What-does-Stanford-do-about-non-Stanfo...
kleiba 9y ago
What a privilege to study at that college.
- kleiba 9y ago
  From the downvotes, I suppose people thought I was being sarcastic. Quite the contrary, I was really in awe about what a cool lecture (series) this is! If you're into that kind of topic, of course, YMMV. But if you are, you will agree that this is an excellent class -- it underlines (not surprisingly) what an outstanding place to study Stanford is.
  - inputcoffee 9y ago
    I didn't down-vote you but I suspect people reacted negatively because the implication is you have to stop learning after college. They may feel you can always learn it at any time. You can take deep learning and math courses online etc.
    - exelius 9y ago
      Around here, he was likely downvoted by all the Stanford grads who thought he was being sarcastic. Or by the dropout hackers who make just as much as their peers without having gone to college. Who knows :)
    - cgmg 9y ago
      > I didn't down-vote you but I suspect people reacted negatively because the implication is you have to stop learning after college. How on earth does their comment imply that?
      - inputcoffee 9y ago
        I misread it as "to study that at college" instead of "study at that college"! I now have no idea why it was down-voted.
- cerebrum 9y ago
pwaivers 9y ago
The pictures on the first page look exactly like what I expect graduate students and professors of Stats to look like.
inputcoffee 9y ago
I feel like asking: did they solve the problem? Let me see if I can state the problem: Neural Networks are non-linear because of their activation functions. You need a differentiable function in order to take the derivative so you can back-prop the error, more or less. The consequence of the non-linearity is that you can't do some kind of short hand calculation to figure out what the network will do. You have to crank through the network to see the result. There is no economy of thought. That is to say, there is no theory. I am excited that they are working on it, but I would love to have a summary or overview of how they approach what I consider to be the basic problem.
- alexnewman 9y ago
  Is that the only problem? Also why make a network of a certain length. Why not farther. Depth and it’s relation to the degree of the estimated polynomial is well undertood. What’s less understood is why not so deep
  - inputcoffee 9y ago
    I didn't mean to say that it is the only problem. It is more like the "meta problem" as far as theory is concerned. There are many parameters (layers, nodes, connectivity, drop off, number of trials etc). Right now they are guessed at by trial and error. The thing preventing you from saying, "Oh, just reduce the layers by 1, and add more nodes" is the non-linearity.
- bitL 9y ago
  Why is the lack of theory a problem? At some point we have to accept some problems are "out of our league" and use whatever is available even without fully understanding it. We can't even understand simple specializations in computer vision, yet we expect to understand a more general method? It's not like 15 years ago there weren't math theorems proven by enumerating them on a computer. I understand it breaks psyche and pride of some scientists, so what? Universe can't be expected to fit into humanity's collective brain.
  - 9y ago
AlexCoventry 9y ago
I'm a bit surprised that Soatto's and Tishby's papers aren't on the reading list (https://stats385.github.io/readings https://stats385.github.io/readings). I think they have some of the most interesting theories, at the moment, about why Deep Learning works. https://arxiv.org/abs/1706.01350 https://arxiv.org/abs/1706.01350 https://openreview.net/pdf?id=ry_WPG-A- https://openreview.net/pdf?id=ry_WPG-A-
- trashtoss 9y ago
  Is anyone looking into/using algebraic topology and sheaves to analyze or interpret these deep networks?

Does anybody know if they plan on releasing the lecture videos? I couldn't find them on the site and this looks very interesting.

walrus 9y ago
They are being recorded and posted to YouTube as unlisted videos. You can find links to some of them on Twitter[1] and ResearchGate[2]. It looks like highlights of the lectures are being posted to Twitter and the full lectures are being posted to ResearchGate. [1] https://twitter.com/stats385 https://twitter.com/stats385 [2] https://www.researchgate.net/project/Theories-of-Deep-Learning https://www.researchgate.net/project/Theories-of-Deep-Learni...
- rayuela 9y ago
  Oh those full lectures on researchgate are awsome. Thanks for sharing!

tequila_shot 9y ago

Can general public attend these sessions?

jaytaylor 9y ago
Technically, no, because you have to be enrolled as a student (not just a student of life ;). However, Quora seems to think you may be able to sneak in, at least sometimes: https://www.quora.com/What-does-Stanford-do-about-non-Stanford-students-sitting-in-on-classes https://www.quora.com/What-does-Stanford-do-about-non-Stanfo...

kleiba 9y ago

What a privilege to study at that college.

kleiba 9y ago
From the downvotes, I suppose people thought I was being sarcastic. Quite the contrary, I was really in awe about what a cool lecture (series) this is! If you're into that kind of topic, of course, YMMV. But if you are, you will agree that this is an excellent class -- it underlines (not surprisingly) what an outstanding place to study Stanford is.
- inputcoffee 9y ago
  I didn't down-vote you but I suspect people reacted negatively because the implication is you have to stop learning after college. They may feel you can always learn it at any time. You can take deep learning and math courses online etc.
  - exelius 9y ago
    Around here, he was likely downvoted by all the Stanford grads who thought he was being sarcastic. Or by the dropout hackers who make just as much as their peers without having gone to college. Who knows :)
  - cgmg 9y ago
    > I didn't down-vote you but I suspect people reacted negatively because the implication is you have to stop learning after college. How on earth does their comment imply that?
    - inputcoffee 9y ago
      I misread it as "to study that at college" instead of "study at that college"! I now have no idea why it was down-voted.
cerebrum 9y ago

pwaivers 9y ago

The pictures on the first page look exactly like what I expect graduate students and professors of Stats to look like.

inputcoffee 9y ago

I feel like asking: did they solve the problem? Let me see if I can state the problem: Neural Networks are non-linear because of their activation functions. You need a differentiable function in order to take the derivative so you can back-prop the error, more or less. The consequence of the non-linearity is that you can't do some kind of short hand calculation to figure out what the network will do. You have to crank through the network to see the result. There is no economy of thought. That is to say, there is no theory. I am excited that they are working on it, but I would love to have a summary or overview of how they approach what I consider to be the basic problem.

alexnewman 9y ago
Is that the only problem? Also why make a network of a certain length. Why not farther. Depth and it’s relation to the degree of the estimated polynomial is well undertood. What’s less understood is why not so deep
- inputcoffee 9y ago
  I didn't mean to say that it is the only problem. It is more like the "meta problem" as far as theory is concerned. There are many parameters (layers, nodes, connectivity, drop off, number of trials etc). Right now they are guessed at by trial and error. The thing preventing you from saying, "Oh, just reduce the layers by 1, and add more nodes" is the non-linearity.
bitL 9y ago
Why is the lack of theory a problem? At some point we have to accept some problems are "out of our league" and use whatever is available even without fully understanding it. We can't even understand simple specializations in computer vision, yet we expect to understand a more general method? It's not like 15 years ago there weren't math theorems proven by enumerating them on a computer. I understand it breaks psyche and pride of some scientists, so what? Universe can't be expected to fit into humanity's collective brain.
- 9y ago

AlexCoventry 9y ago

I'm a bit surprised that Soatto's and Tishby's papers aren't on the reading list (https://stats385.github.io/readings https://stats385.github.io/readings). I think they have some of the most interesting theories, at the moment, about why Deep Learning works. https://arxiv.org/abs/1706.01350 https://arxiv.org/abs/1706.01350 https://openreview.net/pdf?id=ry_WPG-A- https://openreview.net/pdf?id=ry_WPG-A-

trashtoss 9y ago
Is anyone looking into/using algebraic topology and sheaves to analyze or interpret these deep networks?