4 ms·
I had to edit the title a bit to fit the character limits, original: NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any
by ianbutler 3y ago
I had to edit the title a bit to fit the character limits, original:
NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.
---------
This is a claimed improvement over the recent Superhot scaling blog post.