4 ms·
Shortformer: Better Language Modeling Using Shorter Inputs (Paper Explained)
- deeplstm 6y agoModelling long sequences has always been hard for transformer-based models. This paper proposes a super innovative way for the transformer to cache previously processed tokens. And it makes generation 9X faster. This is truly mind-blowing Paper https://arxiv.org/abs/2012.15832 https://arxiv.org/abs/2012.15832 Code https://github.com/ofirpress/shortformer https://github.com/ofirpress/shortformer