4 ms·
You must have missed the memo... It's now super easy to extend the context of 2k llama models to 8k, 16k, or even 32k with just a small fine tune and a tweak to
by Tostino 3y ago
You must have missed the memo... It's now super easy to extend the context of 2k llama models to 8k, 16k, or even 32k with just a small fine tune and a tweak to the code.
You still need the memory to be able to go that high, but it's totally doable.
- thomasahle 3y agoWhat memo/paper does this?
- Tostino 3y agohttps://arxiv.org/abs/2306.15595 https://arxiv.org/abs/2306.15595
- imtemplain 3y ago[dead]
- brucethemoose2 3y agoI saw the SuperHOT LORAs as well. But I assumed full training would give better perplexity for large contexts, and perhaps this method would be more effective at 16K+ with an 8K model to start with.
- Tostino 3y agoPossibly, but the perplexity has shown to decrease while fine-tuning a 2048 model on larger context sizes for outputs within it's original context limit...so, more research needed.