3 ms·
If you're interested in this you may want to take a look at SDM content-addressable memory, which uses neural networks as address encoders/decoders: https://en
by bra-ket 10y ago
If you're interested in this you may want to take a look at SDM content-addressable memory, which uses neural networks as address encoders/decoders: https://en.wikipedia.org/wiki/Sparse_distributed_memory https://en.wikipedia.org/wiki/Sparse_distributed_memory
it was developed by Pentti Kanerva at NASA in the 80s
For related memory-augmented models see: https://en.wikipedia.org/wiki/Deep_learning#Networks_with_separate_memory_structures https://en.wikipedia.org/wiki/Deep_learning#Networks_with_se...
There was also "Reasoning, Attention, Memory (RAM)" NIPS Workshop 2015 organized by Jason Weston on this topic: http://www.thespermwhale.com/jaseweston/ram/ http://www.thespermwhale.com/jaseweston/ram/
There is disproportionate amount of work on training learning models while ignoring memory mechanics. For some reason this research is pursued by very few machine learning/AI labs, mostly at Google DeepMind/Brain, FB, and Numenta, and may be Tom Mitchell's 'never-ending learning' project at CMU.
- ericjang 10y agoI think part of the reason why memory / attention mechanisms haven't caught on quite yet is that they add a layer of complexity to reasoning about what DNNs can/cannot do, which is already challenging for researchers. It takes awhile to prove that these are scalable enough for a "killer app". Also, industry applications of deep learning are at least 1-2 years behind academia, so I don't expect to see any use of differentiable memory mechanisms in production anytime soon. Anyway, the workshop reading list looks amazing. Thanks for this!