4 ms·

>The 4k context length is nice, but RoPE makes it irrelevant anyway. Can you elaborate on this?

by msp26 3y ago

philovivero 3y ago
Start searching SuperHOT and RoPE together. 8k-32k context length on regular old Llama models that were originally intended to only have 2k context lengths.
- Der_Einzige 3y ago
  Any trick which is not doing full quadratic attention cripples a models ability to reason "in the middle" more than they already are crippled. Good long context length models are currently a mirage. This is why no one is seriously using GPT-4-32k or Claude-100k in production right now. Edit: even if it's doing full attention like the commentator says, turns out that's not good enough! https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172
  - redox99 3y ago
    This is still doing full quadratic attention.
moffkalast 3y ago
Here's some more info on it: https://arxiv.org/pdf/2306.15595.pdf https://arxiv.org/pdf/2306.15595.pdf https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar... https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamic... In short, the context is just an array of indexes passed along with the data, which can be changed to floats and encode more sparsely to scale to an arbitrarily small or large context. It does need some tuning of the model to work well though afaik. What's funnier is that Meta came up with it (that paper is theirs) and somehow didn't bother including it in LLama 2.
- deleted 3y ago
  [deleted]
- imjonse 3y ago
  It's likely the paper's ideas were still fresh or incompletely validated when LLama 2 had already been training.
ZeroCool2u 3y ago
See: https://arxiv.org/abs/2306.15595 https://arxiv.org/abs/2306.15595 "We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least ∼600× smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure."

>The 4k context length is nice, but RoPE makes it irrelevant anyway. Can you elaborate on this?

by msp26 3y ago

>The 4k context length is nice, but RoPE makes it irrelevant anyway. Can you elaborate on this?

Start searching SuperHOT and RoPE together. 8k-32k context length on regular old Llama models that were originally intended to only have 2k context lengths.

Der_Einzige 3y ago
Any trick which is not doing full quadratic attention cripples a models ability to reason "in the middle" more than they already are crippled. Good long context length models are currently a mirage. This is why no one is seriously using GPT-4-32k or Claude-100k in production right now. Edit: even if it's doing full attention like the commentator says, turns out that's not good enough! https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172
- redox99 3y ago
  This is still doing full quadratic attention.

moffkalast 3y ago

Here's some more info on it: https://arxiv.org/pdf/2306.15595.pdf https://arxiv.org/pdf/2306.15595.pdf https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar... https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamic... In short, the context is just an array of indexes passed along with the data, which can be changed to floats and encode more sparsely to scale to an arbitrarily small or large context. It does need some tuning of the model to work well though afaik. What's funnier is that Meta came up with it (that paper is theirs) and somehow didn't bother including it in LLama 2.

deleted 3y ago
[deleted]
imjonse 3y ago
It's likely the paper's ideas were still fresh or incompletely validated when LLama 2 had already been training.

ZeroCool2u 3y ago

See: https://arxiv.org/abs/2306.15595 https://arxiv.org/abs/2306.15595 "We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least ∼600× smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure."