4 ms·
I've thought a bit about RNNs, and I can see an obvious problem: Fixed amount of memory. Is there any chance someone's come up with an RNN that has dynamic amo
by tormeh 11y ago
I've thought a bit about RNNs, and I can see an obvious problem: Fixed amount of memory.
Is there any chance someone's come up with an RNN that has dynamic amounts of memory?
- varelse 11y agoThere's a huge degree of data re-use in the weights. This should be exploited. Second, one could envision paging the hidden units back to system memory on a coprocessor-based implementation (GPUs/FPGAs/not Xeon Phi, gag me). 256 GB servers are effectively peanuts these days relative to developer salaries and university grants (datapoint: my grad school work system was ~$100K in 1990 dollars) so unless you're trying to create the first strong AI, I don't think this is a serious constraint. Good luck with that no matter what Stephen Hawking, Elon Musk, and Nick Bostrom harp on about: we have no idea what the error function for strong AI ought to be and even if we did, it's over a MW using current technology to achieve the estimated FLOPS of a human cerebrum.
- tormeh 11y agoI meant that the state vector has constant size and just setting it at the maximum available might give you problems with training.
- varelse 11y agoNothing you can't work around if you're willing to roll your own code. That said, I agree 100% if you're dependent on someone else's framework...
- tormeh 11y agoI meant that the state vector has constant size and just setting it at the maximum available might give you problems with training.
- exgrv 11y agoThere is this paper by Joulin & Mikolov: Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (http://arxiv.org/abs/1503.01007 http://arxiv.org/abs/1503.01007). In this case, the memory of the RNN is an ensemble of differentiable stacks.
- p1esk 11y agoLook up Neural Turing Machines: connecting neural networks to external banks of memory.