XLNet and Transformer-XL can maintain a memory of the last thousand or so tokens. The results are pretty remarkable. Check out the hallucinated Napoleonic-histo

XLNet and Transformer-XL can maintain a memory of the last thousand or so tokens. The results are pretty remarkable. Check out the hallucinated Napoleonic-history wikipedia article at the bottom of https://arxiv.org/pdf/1901.02860.pdf https://arxiv.org/pdf/1901.02860.pdf