3 ms·
NTK-Aware Scaled RoPE allows models to have extended context without finetuning
- ianbutler 3y agoI had to edit the title a bit to fit the character limits, original: NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. --------- This is a claimed improvement over the recent Superhot scaling blog post.