6 ms·
The Unreasonable Syntactic Expressivity of RNNs
- GreenHeuristics 6y agoWhat's unreasonable about it?
- ve55 6y agoIt would be better read in a way such as 'unreasonably surprising'
- marton78 6y agoIt's an hommage to https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness... The title has become a meme just like "... considered harmful".
- kubanczyk 6y agoActually, it's a homage to "The Unreasonable Effectiveness of Recurrent Neural Networks" which is a homage to "The Unreasonable Effectiveness of Mathematics in the Natural Sciences".
- Nowado 6y agoMeme chains definitely are a progress compared to eponyms, but personally I hoped we would make a slightly bigger jump this time.
- neolog 6y agoPlease add an RSS feed to your blog.
- pedro1976 6y agoAs an intermediate solution you can use this feed [0] which is the result of rss-proxy [1]. [0] https://rssproxy.migor.org/api/feed?url=https%3A%2F%2Fnlp.stanford.edu%2F%2F~johnhew%2F&rule=best&output=ATOM https://rssproxy.migor.org/api/feed?url=https%3A%2F%2Fnlp.st... [1] https://github.com/damoeb/rss-proxy https://github.com/damoeb/rss-proxy
- adamnemecek 6y agoThe ad nauseam repetition of memes like "considered harmful" or "unreasonable..." is too much.
- FeepingCreature 6y agoI think you mean "Unreasonable repetition of 'unreasonable' and 'considered harmful' considered harmful." After all, self-referentiality is all you need.
- dunefox 6y agoI actually like this kind of title, so... `"Unreasonable repetition of 'unreasonable' and 'considered harmful' considered harmful." considered harmful.`
- thesz 6y agoNo word about convolutional sequence modeling: https://openreview.net/pdf?id=rk8wKk-R- https://openreview.net/pdf?id=rk8wKk-R- (PDF) It is even more unreasonably effective, outperforming already unreasonable effective recursive neural networks on reasonable spectacular number of tasks.
- Der_Einzige 6y agoWhen will this be reviewed? Says it's still under review at ICLR...
- eli_gottlieb 6y agoLooks like it ended up as an ICLR 2018 workshop paper. https://www.iclr.cc/Conferences/2018/Schedule?showEvent=511 https://www.iclr.cc/Conferences/2018/Schedule?showEvent=511
- jerome-jh 6y agoI read it as: RNN's (well actually FSTM's) can simulate a type of FSM's with a memory efficiency not thought possible.
- kubanczyk 6y agoDon't you think that the plural nominatives without an apostrophe are less ambiguous? I mean RNNs or FSTMs. It's the OED's and CMoS's recommendation: https://english.stackexchange.com/a/56010/15831 https://english.stackexchange.com/a/56010/15831
- 7373737373 6y agoI'm surprised that genetic algorithms on resource-bounded general purpose virtual machines have not surpassed neural nets in their problem solving ability. Is the computational advantage of neural nets on GPUs so dramatic?
- gmadsen 6y agoyes, extremely. parallelism like that isn't possible with GA
- king_magic 6y agoGAs can be way, way, way more compute intensive as your population size increases. That said, I use GAs for optimization and find them to be an incredibly useful tool for various optimization tasks - just generally not for learning weights of NNs, where a GPU + backprop generally reigns supreme (or at least works well enough/much faster than a GA).
- pchiusano 6y agoYou might be interested in "compact GA" which only requires keeping the % of 1s (or 0s) at each bit position. I suspect that with some more engineering and attention from people doing ML stuff, GA-style algorithms can be made just as memory and space efficient as gradient methods, while giving better results and being more widely applicable. Here's a post on this: http://pchiusano.github.io/2020-10-12/learning-without-a-gradient2.html http://pchiusano.github.io/2020-10-12/learning-without-a-gra...
- cpgxiii 6y agoIt comes down to gradients. When you have useful gradients to work with backprop on a GPU is vastly faster and more directed than GAs. One could uncharitably say that all of the impressive results from NNs in the last decade are the result of this ability to throw large amounts of data into highly parallel training, rather than new theory about the expressiveness and capabilities of NNs. When you don't have or can't use a gradient, GAs become the go-to tool, for optimizations (if only because you don't have much in the way of other options).
- 6y ago
- albertzeyer 6y agoThis is quite interesting. In the beginning, there is this statement (assumption): > when processing natural language, we can bound the maximum number of things we’ll need to store on the stack Why is that? Is it because there is a limit N and all natural sentences ever are less than N words, and thus you also have a bound on the stack depth? But that's not strictly true, just highly unlikely, right? In principle, for every sentence of length N, I can easily add a word, and have length N+1. In a similar way, I can also always make the stack deeper. This discussion reminds me in general about whether RNNs are turing complete or not. There is a famous paper stating that they are (On the computational power of neural nets, Siegelmann & Sontag, 1992). However, that uses infinite precision rational numbers for weights and activations. Once you have finite precision (standard floats), it has finite memory. Then stack depths are bounded of course, and also it is not Turing complete anymore. Some interesting discussion about this is also here (own question, and also has some own answer, among others): https://stackoverflow.com/questions/2990277/how-useful-is-turing-completeness-are-neural-nets-turing-complete https://stackoverflow.com/questions/2990277/how-useful-is-tu...
- jhardy54 6y ago> Once you have finite precision (standard floats), it has finite memory. Then stack depths are bounded of course, and also it is not Turing complete anymore. My understanding is that Turing machines assume infinite memory -- could you help me understand how this differs? I'm guessing that I'm not understanding your point, but if "finite memory" is a disqualifier then Turing machines aren't Turing-complete.
- albertzeyer 6y agoYes, for Turing machines (TM), or a machine which is as powerful as a Turing machine, you need infinite memory. A Turing machine always infinite memory (the tape is infinite). This is an important property of a TM. Once you have a finite tape only, it is strictly less powerful. That means, for any model / machine with finite memory, it is not as powerful as a Turing machine, i.e. not Turing complete. So, (standard) RNNs with finite precision (standard floats) have finite memory, and thus are not Turing complete. This paper (On the computational power of neural nets) shows that RNNs with infinite precision are Turing complete. But in practice, we never model RNNs that way. There are other ways we can add external (conceptually infinite) memory to a RNN. E.g. see neural turing machines (NTM) and differentiable neural computer (DNC).