Cool. The idea of augmenting a transformer with an RNN isn't new, but I'll take a look. See also the threads here, mentioning other methods for scaling up cont

Cool. The idea of augmenting a transformer with an RNN isn't new, but I'll take a look. See also the threads here, mentioning other methods for scaling up context length up to the millions of tokens: https://news.ycombinator.com/item?id=35502187 https://news.ycombinator.com/item?id=35502187