6 ms·

Capsule Networks Tutorial [video]

140 points by isp 9y ago

isp 9y ago
I found this video to be much easier-to-follow than previous posts focusing on intuition, e.g., https://news.ycombinator.com/item?id=15690121 https://news.ycombinator.com/item?id=15690121
ntenenz 9y ago
When Hinton approves, you know you've done well... https://www.reddit.com/r/MachineLearning/comments/7ew7ba/d_capsule_networks_capsnets_tutorial/dq8yc9p/ https://www.reddit.com/r/MachineLearning/comments/7ew7ba/d_c...
- isp 9y ago
  For anyone who hasn't watched the video: this comment is on topic and certainly relevant, because Hinton himself invented Capsule Networks. https://www.wired.com/story/googles-ai-wizard-unveils-a-new-twist-on-neural-networks/ https://www.wired.com/story/googles-ai-wizard-unveils-a-new-...
georgehm 9y ago
The author answers some questions in the comments as well. Worth checking out!
cloverich 9y ago
I think this is the same author that published "Hands-On Machine Learning with Scikit-Learn and TensorFlow...". The quality of the book (thus far) is so high that I immediately started Googling about the author to try and learn more (and did not learn much), assuming he must be well known. I did not learn much, but can at least say the book is fantastic. [1]: https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1491962291 https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-T...
- colmvp 9y ago
  I second that book. Well worth the small price and very accessible to read
  - isp 9y ago
    Yes, same author. He has a picture of his book at the end of the video (21:40). With these recommendations, and after that video, I am going to buy & read his book.
- chillee 9y ago
  He used to be the PM of youtube video classification too.
- j_s 9y ago
  Too bad O'Reilly no longer sells books -- Cyber Monday 50% off is no more.
tw1010 9y ago
I predict capsule networks will not nearly have as big of an impact on the ML community as many think it will. Why? Because the main reason they exist is to address performance issues in really advanced, cutting-edge, models. But that is not what drives upvotes here and on reddit. The failure of capsule networks to pick up steam and the continued popularity of GANs, I think, is a signal that the main reason ML is still trendy and in vogue is because the subject, AI, tickles the imagination of engineers, but is still, five years after ML started to become popular, not as big in actual practical engineering systems as what the outside public might think it is.
- Mmrnmhrm 9y ago
  I agree, it still needs a strong use case. IMHO, new network architectures are generally overhyped. Even AlexNet performs wonderfully well in most problems once you add nice initialization and batch normalization.
- chillee 9y ago
  I think you have a misconception of what capsule networks are. They are not intended to address "performance issues in really advanced models", they are intended as another paradigm in deep learning that Geoff Hinton thinks has a lot of promise. I also don't know what you mean by "the failure of capsule networks to pick up steam". The paper literally came out a month ago. It's too early to say whether it'll "pick up steam" or not. I also don't understand what you mean by "the continued popularity of GANs" showing anything.
- dnautics 9y ago
  The specific performance issue they are designed to address is training set size dependence. That's not exactly trivial.
Mmrnmhrm 9y ago
Nice video, however instead of riding the hype train of arxiv, could we wait until peer review analyzes the paper? If someone other than Hinton presented a YADLA (Yet Another Deep Learning Architecture) that does not achieve state of the art level of performance in the basic datasets, it would not be very well received.
- sja 9y ago
  I think it's important to note that the paper was accepted for NIPS 2017, and isn't just some random paper pushed on arXiv.
- mindcrime 9y ago
  Wait for what exactly? Talking about this? Implementing and testing it? Experimenting with it, trying to replicate results, trying to extend the ideas? Etc? I'd argue that all of this is peer-review, if not in the traditional / formal sense. And CS (especially ML/AI) seems to be moving in this direction over the past few years and that's not necessarily a bad thing. Also, keep in mind that peer-review or not, if you look at this from a Bayesian point-of-view, the prior on this work being important / meaningful is going to be pretty high for a lot of people - just because it is Hinton. And that's a reasonable position given his past work.
  - Mmrnmhrm 9y ago
    We have double blind peer review precisely to avoid the bias that you express in your second sentence.
    - mindcrime 9y ago
      I feel like you're missing my point. Bayesian reasoning of that nature is totally reasonable and isn't something to be avoided just for the sake of avoiding it. What is is, is useful as a guide for where to direct energy and focus. And what it is is faster than sitting around playing with your pud waiting for review for a journal submission. Again, what's going on now is a form of peer-review. Double blind? No, but that's not really relevant in this context anyway. ML is really more of an empirical field in this day and age and people are going to read pre-prints on ArXiv, and use various Bayesian weighting schemes to decide what to direct time and energy towards. This process complements, not replaces, the kind of formal peer review you're demanding. There will still be plenty of room, and time, for that stuff, but there's no real reason to wait for all that to happen before starting to look into something.
mycat 9y ago
How does it compare with, Spiking Neural Network? Both use vectors (but in different way) to encode more information
dnautics 9y ago
Two impressions: 1. when I saw the original Hinton proposal of capsule networks, I thought it was kind of halfway to a hofstadter-style cognitive machine from his work "conceptual slippages". Now understanding it more, I am more confident in my assessment. 2. I think that implementations are going to be hamstrung by the clunky nature of tensorflow's architecture... Did anyone else feel this?
- halflings 9y ago
  What is clunky exactly about tensorflow's architecture?
  - dnautics 9y ago
    for starters, the problem that you have to separate the definition of the computational graph from the actual execution of the function (there's separate declarative and imperative stages). This problem is not insurmountable. Something like this would be really cool: https://www.youtube.com/watch?v=ijI0BLf-AH0 https://www.youtube.com/watch?v=ijI0BLf-AH0

140 points by isp 9y ago

isp 9y ago

I found this video to be much easier-to-follow than previous posts focusing on intuition, e.g., https://news.ycombinator.com/item?id=15690121 https://news.ycombinator.com/item?id=15690121

ntenenz 9y ago

When Hinton approves, you know you've done well... https://www.reddit.com/r/MachineLearning/comments/7ew7ba/d_capsule_networks_capsnets_tutorial/dq8yc9p/ https://www.reddit.com/r/MachineLearning/comments/7ew7ba/d_c...

isp 9y ago
For anyone who hasn't watched the video: this comment is on topic and certainly relevant, because Hinton himself invented Capsule Networks. https://www.wired.com/story/googles-ai-wizard-unveils-a-new-twist-on-neural-networks/ https://www.wired.com/story/googles-ai-wizard-unveils-a-new-...

georgehm 9y ago

The author answers some questions in the comments as well. Worth checking out!

cloverich 9y ago

I think this is the same author that published "Hands-On Machine Learning with Scikit-Learn and TensorFlow...". The quality of the book (thus far) is so high that I immediately started Googling about the author to try and learn more (and did not learn much), assuming he must be well known. I did not learn much, but can at least say the book is fantastic. [1]: https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1491962291 https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-T...

colmvp 9y ago
I second that book. Well worth the small price and very accessible to read
- isp 9y ago
  Yes, same author. He has a picture of his book at the end of the video (21:40). With these recommendations, and after that video, I am going to buy & read his book.
chillee 9y ago
He used to be the PM of youtube video classification too.
j_s 9y ago
Too bad O'Reilly no longer sells books -- Cyber Monday 50% off is no more.

tw1010 9y ago

I predict capsule networks will not nearly have as big of an impact on the ML community as many think it will. Why? Because the main reason they exist is to address performance issues in really advanced, cutting-edge, models. But that is not what drives upvotes here and on reddit. The failure of capsule networks to pick up steam and the continued popularity of GANs, I think, is a signal that the main reason ML is still trendy and in vogue is because the subject, AI, tickles the imagination of engineers, but is still, five years after ML started to become popular, not as big in actual practical engineering systems as what the outside public might think it is.

Mmrnmhrm 9y ago
I agree, it still needs a strong use case. IMHO, new network architectures are generally overhyped. Even AlexNet performs wonderfully well in most problems once you add nice initialization and batch normalization.
chillee 9y ago
I think you have a misconception of what capsule networks are. They are not intended to address "performance issues in really advanced models", they are intended as another paradigm in deep learning that Geoff Hinton thinks has a lot of promise. I also don't know what you mean by "the failure of capsule networks to pick up steam". The paper literally came out a month ago. It's too early to say whether it'll "pick up steam" or not. I also don't understand what you mean by "the continued popularity of GANs" showing anything.
dnautics 9y ago
The specific performance issue they are designed to address is training set size dependence. That's not exactly trivial.

Mmrnmhrm 9y ago

Nice video, however instead of riding the hype train of arxiv, could we wait until peer review analyzes the paper? If someone other than Hinton presented a YADLA (Yet Another Deep Learning Architecture) that does not achieve state of the art level of performance in the basic datasets, it would not be very well received.

sja 9y ago
I think it's important to note that the paper was accepted for NIPS 2017, and isn't just some random paper pushed on arXiv.
mindcrime 9y ago
Wait for what exactly? Talking about this? Implementing and testing it? Experimenting with it, trying to replicate results, trying to extend the ideas? Etc? I'd argue that all of this is peer-review, if not in the traditional / formal sense. And CS (especially ML/AI) seems to be moving in this direction over the past few years and that's not necessarily a bad thing. Also, keep in mind that peer-review or not, if you look at this from a Bayesian point-of-view, the prior on this work being important / meaningful is going to be pretty high for a lot of people - just because it is Hinton. And that's a reasonable position given his past work.
- Mmrnmhrm 9y ago
  We have double blind peer review precisely to avoid the bias that you express in your second sentence.
  - mindcrime 9y ago
    I feel like you're missing my point. Bayesian reasoning of that nature is totally reasonable and isn't something to be avoided just for the sake of avoiding it. What is is, is useful as a guide for where to direct energy and focus. And what it is is faster than sitting around playing with your pud waiting for review for a journal submission. Again, what's going on now is a form of peer-review. Double blind? No, but that's not really relevant in this context anyway. ML is really more of an empirical field in this day and age and people are going to read pre-prints on ArXiv, and use various Bayesian weighting schemes to decide what to direct time and energy towards. This process complements, not replaces, the kind of formal peer review you're demanding. There will still be plenty of room, and time, for that stuff, but there's no real reason to wait for all that to happen before starting to look into something.

mycat 9y ago

How does it compare with, Spiking Neural Network? Both use vectors (but in different way) to encode more information

dnautics 9y ago

Two impressions: 1. when I saw the original Hinton proposal of capsule networks, I thought it was kind of halfway to a hofstadter-style cognitive machine from his work "conceptual slippages". Now understanding it more, I am more confident in my assessment. 2. I think that implementations are going to be hamstrung by the clunky nature of tensorflow's architecture... Did anyone else feel this?

halflings 9y ago
What is clunky exactly about tensorflow's architecture?
- dnautics 9y ago
  for starters, the problem that you have to separate the definition of the computational graph from the actual execution of the function (there's separate declarative and imperative stages). This problem is not insurmountable. Something like this would be really cool: https://www.youtube.com/watch?v=ijI0BLf-AH0 https://www.youtube.com/watch?v=ijI0BLf-AH0