30 ms·
A New Twist on Neural Networks
- flabbyrabbit 9y agoDirect link: https://www.wired.com/story/googles-ai-wizard-unveils-a-new-twist-on-neural-networks/ https://www.wired.com/story/googles-ai-wizard-unveils-a-new-...
- AlphaWeaver 9y agoCan the article link please be changed to this non AMP link?
- program_whiz 9y agoI believe that the kernels learned by a deep net (especially the detailed ones) are basically what this guy is talking about (a small nnet that recognizes basically one feature). I suppose you could sample a large number of capsules, but that would be equivalent to just making a bigger deep net.
- malmsteen 9y agoIt's probably more than that otherwise that guy wouldn't waste his time.
- ccozan 9y agoyes, is about feeding the 3D context too. It means to recognize a feature once, and then give a spatial translation, is able to say, yes is the same feature, just turned 30deg right, 40deg up, for example, without having to train the model with a _picture_ of an object taken from all sides and perspectives. Humans use binocular vision [1], but AI can be programmed to do more. This is practically introducing AI to the real world: an object is more than the picture of it. [1] https://en.wikipedia.org/wiki/Binocular_vision https://en.wikipedia.org/wiki/Binocular_vision
- visarga 9y agoThere is dynamic routing between capsules, compared to static routing in normal deep nets. The routing itself is learned.
- Nokinside 9y agoSmall correction: "this guy" is "the guy" behind deep learning revolution.
- carapace 9y ago>Abstract. In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks. http://people.idsia.ch/~juergen/deep-learning-overview.html http://people.idsia.ch/~juergen/deep-learning-overview.html
- drdebug 9y agoSo same concept as face detection by Viola Jones ? Look at smaller features and a superset/composition of them ?
- bitL 9y agoCNNs do the same (hierarchy of features, combining lower, simpler features to more complex ones on each level). This is a bit different concept.
- _joel 9y agoPlease can we not use AMP links, but the direct URL, thanks
- lightbyte 9y agoWhy? The AMP site for this specific article is significantly cleaner and easier to read. Edit: whoops apparently I'm not allowed to question this.
- zeusk 9y agoOn Desktop, it is horrible. There are no margins, and zoom is messed up on Chrome/Firefox.
- lightbyte 9y agoDesktop is what I was referring to, its significantly better in my opinion. Just straight text full width, no wasted space and junk all over the screen.
- zeusk 9y agoSurely you don't use a default non-100% zoom, it is crazy how Google who also happens to make a browser can't get such trivial details right. Also, the zero margins are not helpful on a large display. These alone make the experience grotesque and awful for me; DOM-distiller is so much better than AMP in terms of readability and layout.
- yorwba 9y agoYou can get the same experience by just requesting the AMP version from Wired directly (just append `/amp` to the original URL), no reason to go through Google's cache for that.
- ztjio 9y agoI sincerely hope you have no authority over design of any software or web page that any human actually has to read. There is extensive research proving that text wider than about 60-70 characters is increasingly difficult to follow and understand. There is further research showing the benefit of margins for visual understanding as well. Basically what you get with this AMP link goes against all known facts about how the human eyes & brain best receives textual information, unless you just happen to have a huge zoom or tiny screen.
- chrisa 9y agoHere's a youtube video that explains the new capsule idea: https://www.youtube.com/watch?v=VKoLGnq15RM https://www.youtube.com/watch?v=VKoLGnq15RM (I'll admit that I don't fully understand it yet), but I think the major thing that capsules tries to fix is that a CNN only looks at a small window of the image at a time. Since the capsules aggregate more information, it can learn more general features. Also, he notes that the paper was done on the MNIST data set (small images), and may not generalize to larger images, but the initial results are promising.
- smhx 9y agoThe article does not mention the first and second authors of the research work, which is an atrocious thing to do. The paper was authored by Sara Sabour, Nicholas Frosst, Geoffrey E Hinton in that order.
- wyattk 9y agoI agree but it's unsurprising. Hinton has a bit of "star power" in the ML community. He's done a lot of pioneering work on neural networks, and stated that he's been working on this capsule concept for years. I get why the focus is on him but they could have at least mentioned the others.
- nabla9 9y agoCapsules are Hinton's idea. His speech "What is Wrong With Convolutional Neural Nets?" was 2014 and he was already working with capsules. http://techtv.mit.edu/collections/bcs/videos/30698-what-s-wrong-with-convolutional-nets http://techtv.mit.edu/collections/bcs/videos/30698-what-s-wr... The credit from the paper should go to all researchers of course, but Hinton is main driving force behind the research.
- faitswulff 9y agoI'd say the first and second authors still warrant a mention, at least.
- dmix 9y agoThe article was updated to include the two other authors names. There is an update at the end of the article mentioning this.
- justicezyx 9y agoI would argue that ideas are cheap until they are proven correct, relatively, compared to the effort to proven them, in some cases, also to apply them into real world scenarios. I see the society give unproportionally large credit to the so-called "leaders", because they pioneered certain "ideas". That's wrong. Histories bury too many talented individuals who are not recognized, because of some pioneers are frantically sucked all the attentions the society can give. In professional research community, authors are listed because they are deemed a critical contributor to the work. Plain and simple.
- dr_zoidberg 9y agoHinton explains the concept of capsules in this video: https://www.youtube.com/watch?v=rTawFwUvnLE https://www.youtube.com/watch?v=rTawFwUvnLE Which is a lot better than reading someone tell you about this new idea that is called "capsules" but doesn't go into detail. The only thing is that, when this presentation was given, it seems they hadn't worked much more than MNIST (so the new thing now would be the toys-recognition net). Better source, with date: http://techtv.mit.edu/collections/bcs/videos/30698-what-s-wrong-with-convolutional-nets http://techtv.mit.edu/collections/bcs/videos/30698-what-s-wr... (December 2014, for the lazy).
- olewhalehunter 9y agoat a first glance this looks like it's going back to old 80s research on neural representations, back then this kind of stuff was known as Parallel Distributed Processing https://mitpress.mit.edu/books/parallel-distributed-processing https://mitpress.mit.edu/books/parallel-distributed-processi...
- grandalf 9y agoIndeed. A blast from the past. Oddly the capsule approach is how I naively thought image recognition worked until I learned more about it.
- deleted 9y ago[deleted]
- pmikesell 9y agoThank you for the link to the video - this was excellent. His key critique of convnets is that (paraphrasing) all of these pooling layer (max, avg, whatever) are really a very poor way of attempting to solve what is actually a routing problem.
- thomasahle 9y agoApparently, one of the main changes since the talk, is that then he used a "probabilistic unit" next to his "property vector". Now the length of the property vector indicates the probability.
- yolorn123 9y agowhat's the difference between concatinating the layer and adding more layers?
- robthebrew 9y agoI think the key is not looking at how well humans perform, but how badly they make mistakes. For example, over the halloween period we interpret 2 flashing LEDs as scary cat eyes. We might not do the same taken out of the temporal context. How we "fail" is a possible indicator to how we succeed.
- wavefunction 9y agoI'm not sure I follow your example of LEDs as cat eyes and Halloween. Do you mean a Halloween decoration involving LEDs that a human interprets as a representation of cat eyes? That's not really a mistake. Or a human mistaking flashing LEDs as cat eyes? In which case I can't see how the mistake would be limited to the Halloween period.
- robthebrew 9y agoI mean we decide that the LEDs are supposed to be cat's eyes because everyone has halloween decorations up, whereas another time of year, we might conclude (correctly) that they are just flashing LEDs
- 2K17 9y agoI thought Hassabis was in charge there?
- visarga 9y agoDeep Mind, which keeps separate kitchen from Google.
- thallukrish 9y agoI guess the key is our ability to extrapolate, imagine things from a single picture. When a child is shown a proper image of a cat and it sees a cartoon cat, it has done that extrapolation of cat's body contours. Or rather I would say it is some sort of meta data that is learnt out of each experience like the way we model in OO - class and object instances. We somehow are able to abstract the class out of a image even if it is just a single image and I feel it is the meta data that gets refined over time rather than the storing pixels of actual images.
- empath75 9y agoEventually they’re going to start connecting specialized neural networks together into a neural network of neural networks and that’s where the real magic is going to happen.
- folksinger 9y agoEventually Jesus will come back to Earth and that is where the real magic is going to happen.
- tw1010 9y agoThere it is, the obligatory "inception neural networks will lead the singularity" comment.
- jonbronson 9y agoThe author writes "Human children don’t need such explicit and extensive training to learn to recognize a household pet." This claim seems dubious. Study's have shown humans can react to visual stimuli in as little as 1-3ms. If a child observes a cat in the room for only 10 seconds, that's already between 3,000 to 10,000 samples from various perspectives. While our human experience may describe this as a single viewing 'instance', our neurons are actually getting an extensive, continuous training. Is this accounted for in the literature?
- jayd16 9y agoSo we should be using 10 second videos instead of images to train AI?
- dr_zoidberg 9y agoI'm pretty sure that was the reasoning to build the ImageNet (about a million labeled images) in the first place. But labeling images is expensive, and there are hints there's more at play with human cognition. If you see a black cat and a white cat, and someone tells you there are striped colored cats, you can imagine it. And if you were to come across it, you'd instantly recognize it as a cat. Neurals nets can't do that. You can also see a lynx and recognize it as "some kind of cat". Again, neural nets are not there yet. Which is why there are people researching to find new, better algorithms that better mimic what we recognize as intelligence.
- yorwba 9y agoAre you sure about your examples of things neural nets can't do? I think GANs might be able to "imagine" striped cats, provided they have been trained on enough images to capture the space of black/white/striped objects. And a lynx being classified as a cat doesn't seem so outlandish. It has to be classified as something and cats are likely the closest in appearance. Of course these are just based on my intuition of what neural nets are capable of, so if you have examples of cases where these specific tasks were attempted unsuccessfully, I'm interested.
- 9y ago
- guskel 9y agoLayman's interpretation of capsules is that they're designed to facilitate inverse graphics. It's like a pixel shader in reverse.
- epmaybe 9y agoI haven't read the research papers yet, but as someone new to machine learning and image recognition... > Hinton’s capsule networks matched the accuracy of the best previous techniques on a standard test of how well software can learn to recognize handwritten digits Is the journalist just saying that capsule networks can perform well on MNIST? Don't most state of the art techniques perform with 99+ accuracy on MNIST?
- yorwba 9y agoYes, their first paper is on MNIST. That they get high accuracy isn't earth-shattering, but since they are doing something very different from other approaches, it's still noteworthy. The real benefit is in the generalization performance: We then tested this network on the affNIST 4 data set, in which each example is an MNIST digit with a random small affine transformation. Our models were never trained with affine transformations other than translation and any natural transformation seen in the standard MNIST. An under-trained CapsNet with early stopping which achieved 99.23% accuracy on the expanded MNIST test set achieved 79% accuracy on the affnist test set. A traditional convolutional model with a similar number of parameters which achieved similar accuracy (99.22%) on the expanded mnist test set but only achieved 66% on the affnist test set.
- amelius 9y agoYes that seems like a good first test for generalization. Did they publish these images somewhere?
- c3534l 9y agoIn Machine Learning terms, current neural network techniques are a local minima. Capsule networks might not perform as well now, but may offer a way forward which will outperform them in the long term as they're developed and refined. Being able to match MNIST results, rather than do worse, is a good reason to work on them.
- rdlecler1 9y agoIt’s interesting that they mention that Geof’s Inspiration is coming from biology. I think there is a lot more mining we could do in this area. We don’t have to capture the implementation details, just the salient ingredients that make intelligence work in biological organisms.
- tgtweak 9y agoOh... Just that then. :)
- HammadB 9y agoThis paper https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1692705/pdf/10670021.pdf https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1692705/pdf/106... I read in a neuroscience class once makes the argument that their are various "levels and loops" or abstraction layers to the way the brain functions. Kind of similar line of reasoning.
- bawana 9y agois there a 'toy' example where one could compare a 'regular' NN as compared to a 'capsule' NN? Code??
- amrrs 9y agoThat's what they've done with MNIST dataset. https://github.com/naturomics/CapsNet-Tensorflow https://github.com/naturomics/CapsNet-Tensorflow Tensorflow Implementation of CapsNet.
- adamnemecek 9y ago“AI wizard” wired you are killing me.
- c3534l 9y agoWizard is an established and well respected title within computer science.
- indescions_2017 9y agoCongrats to Hinton, et al on publishing. Should see more info at NIPS 2017 in December. Quite admirable, embarking on a late-career "Year Zero" course correction, all in the name of advancing the field ;) How does the human brain handle "invariance"? Not just of the spatial variety. But transformational, temporal, conceptual, and auditory invariance as well? Some background on "columns" from bio-inspired computational neuroscience startup Numeta: Why Does the Neocortex Have Layers and Columns, A Theory of Learning the 3D Structure of the World https://www.biorxiv.org/content/biorxiv/early/2017/07/12/162263.full.pdf https://www.biorxiv.org/content/biorxiv/early/2017/07/12/162...
- fori1to10 9y agoIs there a link to the original papers?
- pgodzin 9y ago> To teach a computer to recognize a cat from many angles, for example, could require thousands of photos covering a variety of perspectives. Human children don’t need such explicit and extensive training to learn to recognize a household pet. Human children see their pet from a million different viewpoints every day
- danharaj 9y agoSure, but they're not being explicitly trained to do anything. It just happens because that's what children do. Also, you don't have to label their cat experiences to identify them from the other millions of experiences they have in a day. You don't need to, they just figure it out without even realizing it. That's pretty great.
- fma 9y agoYes, and toddlers (my 14 month old...) can see a cat in a picture, and recognize a cat in real life, even if it's different color.
- pgodzin 9y agoSure, but at some point you do label "This is our cat Tabby. He lives here." Now every time they see a cat in the house, it is implicitly labelled. I'm definitely disagreeing with "extensive" more than "explicit" in the original quote, but I think it's silly to differentiate explicit vs implicit when talking about a human vs computer
- yters 9y agoFodor's the mind doesn't work like that is a great book explaining the shortcomings of both connectionist and modular models of cognition. He basically says neither should work, nor combinations thereof. Never seen anything more than dismissal of his work.
- mempko 9y agoWow, this should reignite the Chomsky vs Norvig debate. This is the kind of science Chomsky wants.
- asadlionpk 9y agoThis hackernoon article is what cleared the concept of capsules for me: https://hackernoon.com/what-is-a-capsnet-or-capsule-network-2bfbe48769cc https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...
- mlboss 9y agoSiraj Raval's implementation of Capsule Network using tensorflow. video: https://www.youtube.com/watch?v=VKoLGnq15RM https://www.youtube.com/watch?v=VKoLGnq15RM code: https://github.com/llSourcell/capsule_networks https://github.com/llSourcell/capsule_networks
- jacinabox 9y agoIf you consider it, a convolutional neural network is applicable to any type of picture, including those that are not pictures of 3D scenes, such as seismic data. So, in order to handle pictures of 3D scenes well, you are going to have to take extra assumptions about the data. This Geoffrey Hinton does, by taking the assumption that a scene consists of objects, with associated pose parameters.
- crishoj 9y agoMods, please de-AMP the link: https://www.wired.com/story/googles-ai-wizard-unveils-a-new-twist-on-neural-networks/ https://www.wired.com/story/googles-ai-wizard-unveils-a-new-...
- deleted 9y ago[deleted]