3 ms·
Retrieval Meets Long Context Large Language Models
- Jimmc414 3y agoThe main findings are: Retrieval-augmentation can significantly boost the performance of both 4K short context LLMs and 16K/32K long context LLMs. The 4K context LLMs with simple retrieval-augmentation can perform comparable to 16K long context LLMs, while being more efficient at inference. After context window extension and retrieval-augmentation, the best model LLaMA2-70B-32k-ret can outperform GPT-3.5-turbo-16k and Davinci003 in terms of average score on a suit of downstream tasks with informative queries.
- arxiv_papers 3y ago[dead]