3 ms·

It appears they are touting 'memory' as the key new feature, but I know at least in the deep learning NLP world there already exists models with 'memory', like

It appears they are touting 'memory' as the key new feature, but I know at least in the deep learning NLP world there already exists models with 'memory', like LSTMs or RNNs with dynamic memory or 'attention.' I can't imagine this model is too radically different than the others. Maybe I just feel a bit uneasy with a claim such as: > We hope DNCs provide a new metaphor for cognitive science and neuroscience.

modeless 10y ago
The "memory" in a typical RNN is akin to a human's short term working memory. It only holds a few things and forgets old things quickly as new things come in. This new memory can hold a large number of things and stores them for an unlimited amount of time, more like a human's long term memory or a computer's RAM. It's a big difference, and the implementation is completely different too.
- orthoganol 10y ago
  I was not referring to typical RNNs, but LSTMs or RNNs with 'attention'. They are designed to overcome vanishing/ exploding gradient problems and hold arbitrary memory lengths.
  - vintermann 10y ago
    They can technically be as long as you want them, but in practice there are still severe constraints. LSTMs alleviate the gradient problems, but you still get real trouble with long-term dependencies. Alex Graves and some others in DeepMind have focused a lot in the past year or so on developing practical differentiable data structures, so that the LSTM can read and write to an external memory (and save its precious internal state for more immediate needs) yet still be trainable via backpropagation.
  - modeless 10y ago
    The memory provided by an attention architecture is immutable and has only one addressing mode. The memory in this paper is mutable (hence "Neural Turing Machine") and has several different addressing modes.
empath75 10y ago
Basically they're using differentiable memory allocation, which means that training can not just change 'what' is stored in memory but 'where' it's stored.
deleted 10y ago
[deleted]
praccu 10y ago
LSTMs are very different. Think of it this way. LSTMs store information about the current problem you're solving. DNCs store information about the world. LSTMs are designed to capture long range dependencies, e.g., "this word at the start of the sentence interacts with this word at the end of the sentence." DNCs are designed to incorporate outside information, e.g., "i happen to know (from background knowledge) that these two people in this sentence are married"

modeless 10y ago

The "memory" in a typical RNN is akin to a human's short term working memory. It only holds a few things and forgets old things quickly as new things come in. This new memory can hold a large number of things and stores them for an unlimited amount of time, more like a human's long term memory or a computer's RAM. It's a big difference, and the implementation is completely different too.

orthoganol 10y ago
I was not referring to typical RNNs, but LSTMs or RNNs with 'attention'. They are designed to overcome vanishing/ exploding gradient problems and hold arbitrary memory lengths.
- vintermann 10y ago
  They can technically be as long as you want them, but in practice there are still severe constraints. LSTMs alleviate the gradient problems, but you still get real trouble with long-term dependencies. Alex Graves and some others in DeepMind have focused a lot in the past year or so on developing practical differentiable data structures, so that the LSTM can read and write to an external memory (and save its precious internal state for more immediate needs) yet still be trainable via backpropagation.
- modeless 10y ago
  The memory provided by an attention architecture is immutable and has only one addressing mode. The memory in this paper is mutable (hence "Neural Turing Machine") and has several different addressing modes.

empath75 10y ago

Basically they're using differentiable memory allocation, which means that training can not just change 'what' is stored in memory but 'where' it's stored.

deleted 10y ago

[deleted]

praccu 10y ago

LSTMs are very different. Think of it this way. LSTMs store information about the current problem you're solving. DNCs store information about the world. LSTMs are designed to capture long range dependencies, e.g., "this word at the start of the sentence interacts with this word at the end of the sentence." DNCs are designed to incorporate outside information, e.g., "i happen to know (from background knowledge) that these two people in this sentence are married"