4 ms·
This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". T
by nl 12y ago
This whole slide deck is worth reading. A couple of highlights:
Pg 26, quote: "Anything humans can do in 0.1 sec, the
right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true.
Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rate of around 16%. The 2014 system had 24 layers and an error rate of 6.66%. Note that trained humans have an error rate of around 5%[1].
Page 50-57 talk about the miracle that is Word2Vec, and what is possible with that.
Page 60-70 talks about paragraph embedding. I haven't seen this published before.
Page 70-73 extends word/paragraph embedding for translation. I've seen a slide deck showing this works before, but I need to read the new paper cited there.
Page 74+ talks about cross-modal embeddings, especially the caption generation stuff. HN has had a few things on that over the past month or so.
[1] http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ http://karpathy.github.io/2014/09/02/what-i-learned-from-com...
- waterlesscloud 12y ago"Anything humans can do in 0.1 sec, the right big 10-layer network can do too" That's an interesting direction from which to view things. Then AI progress can be measured by increasing that timeframe. Though I suspect there are some pretty gigantic discontinuities in there. The things humans can do in 3-4 seconds are qualitatively different from what they can do in less than 1 second, for example. Still, it's a useful perspective to keep in mind.
- crazypyro 12y agoThere's also the issue that we don't even fully understand what the brain even does, so how can we claim that anything in this incomplete set of operations is possible with a computer...
- sanxiyn 12y ago> Page 60-70 talks about paragraph embedding. I haven't seen this published before. Paragraph embedding was published this year as "Distributed Representations of Sentences and Documents". http://arxiv.org/abs/1405.4053 http://arxiv.org/abs/1405.4053
- nl 12y agoThanks. It's interesting how the state of the art is outpacing publishing. From a quick scan that appears quite similar to the approach in papers like "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" (2011)[1]. Edit: I see they cite this paper too. [1] http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf
- agibsonccc 12y agoThere's also better embeddings than word2vec now: http://nlp.stanford.edu/projects/glove/ http://nlp.stanford.edu/projects/glove/
- nl 12y agoThe characterisation of Glove as better than Word2Vec is controversial. I'm on mobile now, but one of the word2vec authors had a Google doc going through the claims, and pointing out that similar performance was possible from word2vec by changing the parameters word2vec is used with. Edit: a link about this. https://news.ycombinator.com/item?id=8660624 https://news.ycombinator.com/item?id=8660624
- agibsonccc 12y agoSpeaking from personal experience. I get paid to do deep learning. One of skymind's biggest app areas is text. That being said: I will be benchmarking deeplearning4j's glove with word2vec here soon. Any machine learning algorithm is better when you tune it. I personally like glove due to having less knobs. The mechanics involving document statistics being part of the gradient update is also interesting. I've also messed quite a bit with the distributed representations. I'm not partial to any particular implementation. I'll use what works. That being said, I'm not armchair. I'll be backing this up with my own data as well.
- agibsonccc 12y ago
- jberryman 12y ago> Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. Actually, I'm not sure we can do those things in quite 0.1 sec. All I know about this is from around minute 9 from http://www.radiolab.org/story/267176-never-quite-now/ http://www.radiolab.org/story/267176-never-quite-now/ . One guest on the show even estimates thinking the simplest thought to be on the order of 0.25-0.5 sec.
- rasz_pl 12y agomaybe concious thought, most of the stuff in the brain is on autopilot and happens behind our backs, conciousness gets the results and just builds narrative (often rationalizing the results to make us feel better)
- xfs 12y ago"Anything humans can do in 0.1 sec, the right big 10-layer network can do too" Actually this was argued by Connectionists in 1980s. It is called Feldman's 100-step rule: The critical resource that is most obvious is time. Neurons whose basic computational speed is a few milliseconds must be made to account for complex behaviors which are carried out in a few hundred milliseconds (Posner, 1978). This means that entire complex behaviors are carried out in less than a hundred time steps. Current AI and simulation programs require millions of time steps. Feldman, J. A., & Ballard, D. H. (1982). Connectionist models and their properties. Cognitive Science, 6, p. 206.
- Strilanc 12y agoWhelp, Page 54 blew my mind right out the window. > E(hotter) - E(hot) + E(big) ≈ E(bigger) > E(Rome) - E(Italy) + E(Germany) ≈ E(Berlin) These things are linearly separable?!
- abrichr 12y agoDeep neural networks essentially transform input data into a vector space where the data is "easier" to model. So while the input vectors may not be linearly separable in the input space, the network learns how to transform the input vectors into a space where they are.
- nl 12y agoYes. Hence 'miraculous' Word2vec.
- fuelfive 12y agoThese are great highlights, although your second point from [1] is a bit misleading. Humans have a 5% error rate because the average person can't differentiate between 150 different breeds of dogs and other similar nonsense.