3 ms·
LSTMs are a recurrent architecture for neural networks, meaning that your output depends both on your current input and your previous output. This is similar to
by probably_wrong 2y ago
LSTMs are a recurrent architecture for neural networks, meaning that your output depends both on your current input and your previous output. This is similar to how language works, as the next word in your sentence must fit both the idea you're trying to convey (your input) and the words you've said up until now (your previous output).
LSTMs where very popular for a while (I think the first good version of Google Translate used them) but they had two critical downsides: their performance went down with longer outputs, and they where a bit annoying to parallelize because computing the output for the 10th word required first computing the output of the previous 9 words - no way to use 10 parallel computers. The first problem was solved with Attention, a scaffolding method that prevented degradation over longer sequences. Eventually someone realized that Attention was doing most of the heavy lifting, built an attention-only network that could be easily parallelized (the Transformer), and LSTMs lost the top place.
Are xLSTMs better? On paper I'd say they could be - they seem to have a solid theory and good results. Will they dethrone Transformers? My guess is no, as it wouldn't be the first time that the "better" technology ends up losing against whatever is popular. Having said that, it is entirely possible that some inherently recurrent tasks like stock price prediction could get a boost from this technology and they may find their place.