3 ms·

1. Both Hinton’s capsules papers have been released at the same time (Oct 2017). You can see the first comment on OpenReview page for the EM paper is dated Nov

by p1esk 7y ago

1. Both Hinton’s capsules papers have been released at the same time (Oct 2017). You can see the first comment on OpenReview page for the EM paper is dated Nov 2017. From what I remember, the two papers appear very similar with the main difference in how the routing is implemented. 2. You cite a convnet result from 2011 (!). Don’t you think a modern convnet would do vastly better on this task? 3. Could input size play a role? Did you try feeding 96x96 inputs to the models you’re comparing against, to see if they also benefit from it? 4. I’m a bit confused as to why other implementations failed to reproduce Hinton’s results given that he open sourced their code (link in the first OpenReview comment). 5. Ok, Imagenet is too slow, how about Cifar-10? What would it take to reach, say, 95%? That would be equivalent to a well trained Resnet-18. If you can show such result, I personally would become more interested, because I worked quite a bit with Cifar-10, but not with Norb. I think you might be onto something, but it’s still not clear that capsules approach is scalable and ultimately superior to plain convnets.

fheinsen 7y ago
I’m surprised you did not comment on the fact that my version of EM routing also achieves SOTA on another domain, natural language. Same code. Here are the answers to your questions: 1. The final, published version is stamped “ICLR 2018,” so I used that year. 2. I don’t know if a conventional CNN can do this with 10x fewer parameters, while also learning to do a form of “reverse graphics” without explicitly optimizing for it. (I wouldn’t know how to get a CNN to do that without explicitly making it a training objective.) 3. IIRC, the convnet model from 2011 accepts 96x96 images. As to why Hinton et al. downsample images to 9x smaller, I suspect (but don’t know for sure) they had no choice to conserve memory and computation using their version of EM routing. I was able to reduce memory and computation with my variant of EM routing (by between one and two orders of magnitude) by setting the first routing layer to accept a variable number of inputs, without regard to location in image. 4. Me too. But you asked me about work other than Hinton’s, and that’s all I could find! 5. CIFAR10 is on the to-do list (work permitting!) :-)
- p1esk 7y ago
  How does a regular convnet do on another domain? Learning to do “reverse graphics” is only useful if you can show it is the reason behind performance improvement, compared to a plain convnet. Until we have cifar-10 results it’s not clear. What I’m saying is - no one has yet demonstrated a clear superiority of any capsules based model to the best available plain convnet. Even on cifar-10. Looking forward to your results!
  - fheinsen 7y ago
    > How does a regular convnet do on another domain? As far as I know, regular convnets have failed to outperform query-key-value self-attention models (i.e., transformers based on Vaswani et al.'s work) on pretty much every sequence task, including natural language tasks. > Learning to do “reverse graphics” is only useful if you can show it is the reason behind performance improvement. I would strongly disagree. Building systems that can learn "reverse graphics" on their own has long been a goal of computer vision. It seems a prerequisite for building machines that can build internal representations of the state of the physical world around them. Hinton et al.'s 2018 paper has a summary of recent efforts on this front on the "Related Work" section. > What I’m saying is - no one has yet demonstrated a clear superiority of any capsules based model to the best available plain convnet. No one is saying otherwise. :-) Convnets are still the right tool for most production systems in visual recognition today. That said, I don't think a convnet can achieve 99.1% accuracy on smallNORB with only 272K parameters, after training from scratch without using any additional data or metadata of any kind -- like the model using my routing algorithm. If you think you can do that with a convnet, do it and put it up online (I'd love to see it :-)
    - p1esk 7y ago
      You’re comparing sentence classification done using transformer embeddings to older results which use inferior embeddings. How do regular convnets do when you feed them transformer embeddings? Re learning reverse graphics - ok, maybe it is indeed the main feature of your work. I’d need to look into that, because from skimming your paper it’s not immediately clear what’s going on there. Re convnet accuracy on Norb - I’m willing to make that effort for cifar-10 as soon as you have the results.

by p1esk 7y ago

fheinsen 7y ago

I’m surprised you did not comment on the fact that my version of EM routing also achieves SOTA on another domain, natural language. Same code. Here are the answers to your questions: 1. The final, published version is stamped “ICLR 2018,” so I used that year. 2. I don’t know if a conventional CNN can do this with 10x fewer parameters, while also learning to do a form of “reverse graphics” without explicitly optimizing for it. (I wouldn’t know how to get a CNN to do that without explicitly making it a training objective.) 3. IIRC, the convnet model from 2011 accepts 96x96 images. As to why Hinton et al. downsample images to 9x smaller, I suspect (but don’t know for sure) they had no choice to conserve memory and computation using their version of EM routing. I was able to reduce memory and computation with my variant of EM routing (by between one and two orders of magnitude) by setting the first routing layer to accept a variable number of inputs, without regard to location in image. 4. Me too. But you asked me about work other than Hinton’s, and that’s all I could find! 5. CIFAR10 is on the to-do list (work permitting!) :-)

p1esk 7y ago
How does a regular convnet do on another domain? Learning to do “reverse graphics” is only useful if you can show it is the reason behind performance improvement, compared to a plain convnet. Until we have cifar-10 results it’s not clear. What I’m saying is - no one has yet demonstrated a clear superiority of any capsules based model to the best available plain convnet. Even on cifar-10. Looking forward to your results!
- fheinsen 7y ago
  > How does a regular convnet do on another domain? As far as I know, regular convnets have failed to outperform query-key-value self-attention models (i.e., transformers based on Vaswani et al.'s work) on pretty much every sequence task, including natural language tasks. > Learning to do “reverse graphics” is only useful if you can show it is the reason behind performance improvement. I would strongly disagree. Building systems that can learn "reverse graphics" on their own has long been a goal of computer vision. It seems a prerequisite for building machines that can build internal representations of the state of the physical world around them. Hinton et al.'s 2018 paper has a summary of recent efforts on this front on the "Related Work" section. > What I’m saying is - no one has yet demonstrated a clear superiority of any capsules based model to the best available plain convnet. No one is saying otherwise. :-) Convnets are still the right tool for most production systems in visual recognition today. That said, I don't think a convnet can achieve 99.1% accuracy on smallNORB with only 272K parameters, after training from scratch without using any additional data or metadata of any kind -- like the model using my routing algorithm. If you think you can do that with a convnet, do it and put it up online (I'd love to see it :-)
  - p1esk 7y ago
    You’re comparing sentence classification done using transformer embeddings to older results which use inferior embeddings. How do regular convnets do when you feed them transformer embeddings? Re learning reverse graphics - ok, maybe it is indeed the main feature of your work. I’d need to look into that, because from skimming your paper it’s not immediately clear what’s going on there. Re convnet accuracy on Norb - I’m willing to make that effort for cifar-10 as soon as you have the results.