Here's some more info on it: https://arxiv.org/pdf/2306.15595.pdf https://arxiv.org/pdf/2306.15595.pdf https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/nt

Here's some more info on it: https://arxiv.org/pdf/2306.15595.pdf https://arxiv.org/pdf/2306.15595.pdf https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar... https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamic... In short, the context is just an array of indexes passed along with the data, which can be changed to floats and encode more sparsely to scale to an arbitrarily small or large context. It does need some tuning of the model to work well though afaik. What's funnier is that Meta came up with it (that paper is theirs) and somehow didn't bother including it in LLama 2.