3 ms·
The context extending methods still hurt perplexity/quality some. The longer the base model is, the more effective the context extending finetunes/post training
by brucethemoose2 3y ago
The context extending methods still hurt perplexity/quality some. The longer the base model is, the more effective the context extending finetunes/post training tricks will be.
- moffkalast 3y agoSure it does, it's not magic. But the alternative is to start dropping out text out of context entirely, which is arguably far worse. As someone else mentioned, this is probably more due to Llama 2 being already in training when this was figured out and it's not fully accepted yet, but I wouldn't be surprised if there was LLama 3 with out of the box dynamically scaled context at some point.