Yep, the number has been expanding. I just picked 512 because Bert uses that. But yeah 50k and beyond unlocks new ways of using context. I can’t wait to see how

Yep, the number has been expanding. I just picked 512 because Bert uses that. But yeah 50k and beyond unlocks new ways of using context. I can’t wait to see how they optimize and improve the reformer model. I expect there to be an improvement like there was for BERT -> Albert