4 ms·

Can someone knowledgeable in the field elaborate on the differences between this and other NN approaches that also use additional memory structures (LSTM, etc)?

crazypyro 11y ago
They address this in the paper under section 3, "Related work". From the paper (there's more on other things besides LSTM, but here's the LSTM specific mention): Hochreiter and Schmidhuber [20] introduced the Long Short Term Memory network (LSTM) architecture. While this model was orginally developed to address the vanishing and exploding gradient problems, LSTM is also able to learn simple context-free and context-sensitive grammars [17, 37]. This is possible because its hidden units can choose through a multiplicative gating mechanism to be either linear or non-linear. The linear units allow the network to potentially count (one can easily add and subtract constants) and store a finite amount of information for a long period of time. These mechanisms are also used in the Gated Recurrent Unit network [7, 9]. In our work we investigate the use of a similar mechanism in a context where the memory is unbounded and structured. As opposed to previous work, we do not need to “erase” our memory to store a new unit. Edit: They also provide a table in the paper showing improvements over LSTM for certain types of algorithms.
andrewtbham 11y ago
I will hazard a guess as to why this might be better than LSTM. LSTM maintains a vector that represents the long short term memory, but once information is cleared from the vector, there is no way to get it back... whereas a stack you can keep going further down the stack if needed. http://colah.github.io/posts/2015-08-Understanding-LSTMs/ http://colah.github.io/posts/2015-08-Understanding-LSTMs/
- sxyuan 11y ago
  One related work that came to mind was Neural Turing Machines, which augments the LSTM network with a (fixed size) memory [1]. I did see a brief reference to the Graves et al. paper in the paper linked to in the original post, but I'm also wondering if someone more knowledgeable could elaborate on the difference. My own guess would be that the models are similar (both using NNs with external memory, trained through backprop), but this has a simpler memory structure and so could be trained more easily. It seems like the choice of regular RNNs vs. an LSTM network doesn't matter much since the Graves paper also compared LSTM vs. feedforward controllers and the results were about the same, with the feedforward controllers actually learning a bit faster. The key of both works seems to be in making a continuous system that can be trained to use external memory through backprop. [1] http://arxiv.org/abs/1410.5401 http://arxiv.org/abs/1410.5401
eli_gottlieb 11y ago
From reading the page and these comments... A feedforward neural network can be viewed as a "continuous generalization" of a fixed-size circuit, which makes it able to learn any function within the circuit complexity of the network. A deep neural network biases towards circuits that are deeper than they are wide, towards functions composed out of smaller functions. Since many useful functions are such (and for other theoretical reasons, such as deep networks having "funnels" in their free-energy landscapes), deep learning works well. A recurrent neural network goes from circuits to deterministic finite-state automata. LSTM basically turns the whole thing into a small, fixed-size almost sorta-kinda pushdown automaton, and Neural Turing Machines generalize to a sorta-kinda linear-bounded automaton. This stack approach makes things much more like a real pushdown automaton. So a feedforward network with a stack should be able to "learn any PDA", and a recurrent network with an unlimited-size stack should be able to learn more-or-less any Turing machine (though it's biased towards Turing machines that "look like" PDAs with funny state transitions). Kinda. waves hands
robryk 11y ago
A difference between the two is that LSTM has a constant amount of memory, defined at training time. In this model, the stacks can grow arbitrarily large. Also, using a finite state machine and two stacks one can simulate any Turing machine, so one can represent arbitrary computations in this model (assuming we structure the input in such a way that there is time to do the computation in question).

crazypyro 11y ago

They address this in the paper under section 3, "Related work". From the paper (there's more on other things besides LSTM, but here's the LSTM specific mention): Hochreiter and Schmidhuber [20] introduced the Long Short Term Memory network (LSTM) architecture. While this model was orginally developed to address the vanishing and exploding gradient problems, LSTM is also able to learn simple context-free and context-sensitive grammars [17, 37]. This is possible because its hidden units can choose through a multiplicative gating mechanism to be either linear or non-linear. The linear units allow the network to potentially count (one can easily add and subtract constants) and store a finite amount of information for a long period of time. These mechanisms are also used in the Gated Recurrent Unit network [7, 9]. In our work we investigate the use of a similar mechanism in a context where the memory is unbounded and structured. As opposed to previous work, we do not need to “erase” our memory to store a new unit. Edit: They also provide a table in the paper showing improvements over LSTM for certain types of algorithms.

andrewtbham 11y ago

I will hazard a guess as to why this might be better than LSTM. LSTM maintains a vector that represents the long short term memory, but once information is cleared from the vector, there is no way to get it back... whereas a stack you can keep going further down the stack if needed. http://colah.github.io/posts/2015-08-Understanding-LSTMs/ http://colah.github.io/posts/2015-08-Understanding-LSTMs/

sxyuan 11y ago
One related work that came to mind was Neural Turing Machines, which augments the LSTM network with a (fixed size) memory [1]. I did see a brief reference to the Graves et al. paper in the paper linked to in the original post, but I'm also wondering if someone more knowledgeable could elaborate on the difference. My own guess would be that the models are similar (both using NNs with external memory, trained through backprop), but this has a simpler memory structure and so could be trained more easily. It seems like the choice of regular RNNs vs. an LSTM network doesn't matter much since the Graves paper also compared LSTM vs. feedforward controllers and the results were about the same, with the feedforward controllers actually learning a bit faster. The key of both works seems to be in making a continuous system that can be trained to use external memory through backprop. [1] http://arxiv.org/abs/1410.5401 http://arxiv.org/abs/1410.5401

eli_gottlieb 11y ago

From reading the page and these comments... A feedforward neural network can be viewed as a "continuous generalization" of a fixed-size circuit, which makes it able to learn any function within the circuit complexity of the network. A deep neural network biases towards circuits that are deeper than they are wide, towards functions composed out of smaller functions. Since many useful functions are such (and for other theoretical reasons, such as deep networks having "funnels" in their free-energy landscapes), deep learning works well. A recurrent neural network goes from circuits to deterministic finite-state automata. LSTM basically turns the whole thing into a small, fixed-size almost sorta-kinda pushdown automaton, and Neural Turing Machines generalize to a sorta-kinda linear-bounded automaton. This stack approach makes things much more like a real pushdown automaton. So a feedforward network with a stack should be able to "learn any PDA", and a recurrent network with an unlimited-size stack should be able to learn more-or-less any Turing machine (though it's biased towards Turing machines that "look like" PDAs with funny state transitions). Kinda. waves hands

robryk 11y ago

A difference between the two is that LSTM has a constant amount of memory, defined at training time. In this model, the stacks can grow arbitrarily large. Also, using a finite state machine and two stacks one can simulate any Turing machine, so one can represent arbitrary computations in this model (assuming we structure the input in such a way that there is time to do the computation in question).