It could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at r

It could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime (even better if your attention is subquadratic - again there is a dozen of varieties to pick from), see: "A Length-Extrapolatable Transformer" https://arxiv.org/abs/2212.10554 https://arxiv.org/abs/2212.10554 "Language Is Not All You Need: Aligning Perception with Language Models" https://arxiv.org/abs/2302.14045 https://arxiv.org/abs/2302.14045 Notably, this positional embedding has been implemented by lucidrains in his x-transformers package: https://github.com/lucidrains/x-transformers/blob/main/x_transformers/x_transformers.py#L873 https://github.com/lucidrains/x-transformers/blob/main/x_tra...