4 ms·

Stacked Capsule Autoencoders

46 points by _Microft 7y ago

p1esk 7y ago
So it’s been almost two years since the capsules paper, and they still have nothing but mnist results? A bit underwhelming...
- tw1010 7y ago
  Has anything actually happened in the last two years? Genuinely curious.
  - p1esk 7y ago
    As a comparison, look at the transformer paper (also 2017), and the progress it has triggered.
  - ivalm 7y ago
    I think there are two big things: 1. Attention mechanism (transformers). Really moved forward NLP. It allows long distance dependencies without the need for sequential processing (like in RNNs). Generally scales much better (see BERT/GPT-2/XL-Net/etc) 2. Separable convolutions. Significantly sped-up the standard CV networks. (mobile net, but now everything, really).
- SubiculumCode 7y ago
  https://arxiv.org/abs/1906.06818 https://arxiv.org/abs/1906.06818 says June 2019. To which paper do you refer?
  - muizelaar 7y ago
    "Matrix capsules with EM routing"
- MAXPOOL 7y ago
  Capsule networks are attempt to get past the limits of current approach to deep learning. It took 20 years until the the current paradigm in deep learning started to generate results. You start with toy problems and explore. Hinton's "What is wrong with convolution neural nets?" is still true to the state of the art in convnets. Gradually turning features into objects like we do in deep learning is limiting. Distinguishing feature integration from object detection like humans do and how to do it is an open question. It can take another 10-15 years.
  - p1esk 7y ago
    Or maybe it’s just another overcomplicated model that’s ultimately a dead end, like, say, RBMs, which were also pushed hard by Hinton (before DL breakthrough in 2012).
ilaksh 7y ago
Why do people assume that the assumptions in DL are correct for more general intelligence? It's not remotely like a real biological NN. It's seems like an arbitrary gross approximation that everyone is starting with just because it is the prevailing worldview. Why do some Chinese focus on SNNs? I suspect they just have a different worldview because they are exposed to a different information stream in their language. But SNNs are still a gross approximation. Also, why assume it needs to look like an NN at all?

46 points by _Microft 7y ago

So it’s been almost two years since the capsules paper, and they still have nothing but mnist results? A bit underwhelming...

tw1010 7y ago
Has anything actually happened in the last two years? Genuinely curious.
- p1esk 7y ago
  As a comparison, look at the transformer paper (also 2017), and the progress it has triggered.
- ivalm 7y ago
  I think there are two big things: 1. Attention mechanism (transformers). Really moved forward NLP. It allows long distance dependencies without the need for sequential processing (like in RNNs). Generally scales much better (see BERT/GPT-2/XL-Net/etc) 2. Separable convolutions. Significantly sped-up the standard CV networks. (mobile net, but now everything, really).
SubiculumCode 7y ago
https://arxiv.org/abs/1906.06818 https://arxiv.org/abs/1906.06818 says June 2019. To which paper do you refer?
- muizelaar 7y ago
  "Matrix capsules with EM routing"
MAXPOOL 7y ago
Capsule networks are attempt to get past the limits of current approach to deep learning. It took 20 years until the the current paradigm in deep learning started to generate results. You start with toy problems and explore. Hinton's "What is wrong with convolution neural nets?" is still true to the state of the art in convnets. Gradually turning features into objects like we do in deep learning is limiting. Distinguishing feature integration from object detection like humans do and how to do it is an open question. It can take another 10-15 years.
- p1esk 7y ago
  Or maybe it’s just another overcomplicated model that’s ultimately a dead end, like, say, RBMs, which were also pushed hard by Hinton (before DL breakthrough in 2012).

ilaksh 7y ago

Why do people assume that the assumptions in DL are correct for more general intelligence? It's not remotely like a real biological NN. It's seems like an arbitrary gross approximation that everyone is starting with just because it is the prevailing worldview. Why do some Chinese focus on SNNs? I suspect they just have a different worldview because they are exposed to a different information stream in their language. But SNNs are still a gross approximation. Also, why assume it needs to look like an NN at all?