3 ms·
But that sounds exactly like the memory in long/short term memory recurrent neural networks (LSTM RNN)
by wodenokoto 5y ago
But that sounds exactly like the memory in long/short term memory recurrent neural networks (LSTM RNN)
- kettleballroll 5y agoThe key difference is that RNNs store the already seen parts of the input to use it when processing future inputs, while self attention allows you to focus on all parts in parallel: information from the last input token can be used on the first token, which in RNNs is not possible (NB: that's one of many differences, but likely the one most relevant in this context)