They address this in the paper under section 3, "Related work". From the paper (there's more on other things besides LSTM, but here's the LSTM specific mention

They address this in the paper under section 3, "Related work". From the paper (there's more on other things besides LSTM, but here's the LSTM specific mention): Hochreiter and Schmidhuber [20] introduced the Long Short Term Memory network (LSTM) architecture. While this model was orginally developed to address the vanishing and exploding gradient problems, LSTM is also able to learn simple context-free and context-sensitive grammars [17, 37]. This is possible because its hidden units can choose through a multiplicative gating mechanism to be either linear or non-linear. The linear units allow the network to potentially count (one can easily add and subtract constants) and store a finite amount of information for a long period of time. These mechanisms are also used in the Gated Recurrent Unit network [7, 9]. In our work we investigate the use of a similar mechanism in a context where the memory is unbounded and structured. As opposed to previous work, we do not need to “erase” our memory to store a new unit. Edit: They also provide a table in the paper showing improvements over LSTM for certain types of algorithms.