8 ms·
The Unreasonable Effectiveness of Recurrent Neural Networks (2015)
- thinkr42 8y agoIt is remarkable how frequently “unreasonable effectiveness” is thrown around.
- nerdponx 8y agoIt's a deliberate riff on "The Unreasonable Effectiveness of Mathematics in the Natural Sciences", in the same vein of declaring something "considered harmful."
- tomrod 8y agoI must be missing something -- I have never considered the "unreasonable effectiveness" and "considered harmful" to be equivalent?
- adrianhel 8y ago"The unreasonable effectiveness of ___" and "___ considered harmful" titles are copied a lot. They are not equivalent in any way.
- jasode 8y ago>I have never considered the "unreasonable effectiveness" and "considered harmful" to be equivalent? I didn't downvote you but as an fyi... When gp wrote "in the same vein of", he wasn't saying the phrases meant the same thing. He was pointing out that they are the same classification of rhetorical device which some call a "snowclone". (See examples.[1]) The "unreasonable effectiveness of <x>" is re-used as a text template similar to "<x> considered harmful". For another timely example, the current #1 story on HN is ""Debugging Emacs, or How I Learned to Stop Worrying and Love DTrace"". I think many readers will recognize it as a snowclone of the Kubrick film title "Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb" The snowclone is: "<X> or: How I Learned to Stop Worrying and Love <Y>".[2] [1] https://en.wikipedia.org/wiki/Snowclone#Notable_examples https://en.wikipedia.org/wiki/Snowclone#Notable_examples [2] More examples of previous HN submissions that used that Strangelove snowclone: https://hn.algolia.com/?query=or%20%22how%20i%20learned%20to%20love%22&sort=byPopularity&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=or%20%22how%20i%20learned%20to...)
- tomrod 8y agoAh, that makes sense. Thanks -- and thanks for teaching me about "snowclone".
- buboard 8y agofor the sake of completeness, both are good reads https://homepages.cwi.nl/~storm/teaching/reader/Dijkstra68.pdf https://homepages.cwi.nl/~storm/teaching/reader/Dijkstra68.p... https://www.maths.ed.ac.uk/~v1ranick/papers/wigner.pdf https://www.maths.ed.ac.uk/~v1ranick/papers/wigner.pdf
- thinkr42 8y agoThat is something I hadn’t thought of before
- swagasaurus-rex 8y agoPerhaps it's the educated equivalent of "You'll be amazed at this one amazing trick!"
- backpropaganda 8y agoWe embrace that as well: https://arxiv.org/abs/1404.5997 https://arxiv.org/abs/1404.5997
- gwern 8y agoIf this were written today, Karpathy would have to call it "The Unreasonable Effectiveness of Convolutions". Since 2015, convolutions, causal or dilated convolutions, and especially convolutions with attention like the Transformer, have made remarkable inroads onto RNN territory and are now SOTA for most (all?) sequence-related tasks. Apparently RNNs just don't make very good use of that recurrency & hidden memory, and the non-locality & easy optimization of convolutions allow for much better performance through faster training & bigger models. Who knew?
- phowon 8y agoCan you elaborate on how Transformers are "convolutions with attention"?
- epberry 8y agoI found this to be a great read on what I think the parent is referring to: https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html#transformer https://lilianweng.github.io/lil-log/2018/06/24/attention-at...
- phowon 8y agoBut no part of that Transformer section makes any reference to convolutions.
- gwern 8y agoSelf-attention uses 1x1 convolutions, and while the original Transformer is fully-connected-only (I think?) the latest & greatest uses convolutions (https://arxiv.org/abs/1901.11117 https://arxiv.org/abs/1901.11117), and even if you're using the original, it's very often being used in conjunction with convolutions elsewhere. So you could argue that the real title would have to be 'The Unreasonable Effectiveness of Attention', but given all the other stuff convolutions have done without attention like WaveNet, I think it's currently fairer to go with convolutions than attention. But we'll see how it goes over the next 4 years...
- tanilama 8y agoBut RNNs are losing its edge over sequence modeling against fully attentional techniques like Transformer, at least it seems to be the consensus right now, Transformer, when done right, offers better performance.
- bjourne 8y agoThe cool part of RNNs are how simple they are. https://gist.github.com/karpathy/d4dee566867f8291f086 https://gist.github.com/karpathy/d4dee566867f8291f086 These 112 lines of Python is all it takes to very realistic Shakespeare prose (or Trump tweets if that's your thing). The results are much better than when using HMMs.
- jhinra 8y agoI loved this article; I used Karpathy's tutorial to imitate a local burger reviewer: http://brian.copeland.bz/2018/12/16/the-burger-hunter/ http://brian.copeland.bz/2018/12/16/the-burger-hunter/
- minimaxir 8y agoI made textgenrnn (https://github.com/minimaxir/textgenrnn https://github.com/minimaxir/textgenrnn) as a higher-level approach to creating char-rnns, solving some issues such as the cold start problem (textgenrnn doesn't need a seed) and incorporating a few newer discoveries since 2015 such as Attention and CuDNN speedups. There are other strategies for working with text to solve generic classification problems (e.g. BERT), but for text generation, LSTMs still can't be beat even though it still has issues with longer-term dependencies.
- deleted 8y ago[deleted]
- alkonaut 8y agoI worked with recurrent nets for dynamic system prediction for an undergrad thesis in ‘02 and we really struggled to train them on the CPUs we had. They had probably fewer than 100 parameters.
- minimaxir 8y agoIn the GPU space, the new CuDNN RNNs are almost as fast as running MLPs on GPUs.
- CoreSet 8y agoFun to read this thread and see all the projects it spawned. It's a testament to how compelling the post and example are that they simply demand to be tinkered with. I used the RNN to train a George RR Martin-sounding Twitter account. Really fun for a while - and drove home the "Unreasonable Effectiveness" part.