3 ms·
I saw the SuperHOT LORAs as well. But I assumed full training would give better perplexity for large contexts, and perhaps this method would be more effective
by brucethemoose2 3y ago
I saw the SuperHOT LORAs as well.
But I assumed full training would give better perplexity for large contexts, and perhaps this method would be more effective at 16K+ with an 8K model to start with.
- Tostino 3y agoPossibly, but the perplexity has shown to decrease while fine-tuning a 2048 model on larger context sizes for outputs within it's original context limit...so, more research needed.