3 ms·
The main findings are: Retrieval-augmentation can significantly boost the performance of both 4K short context LLMs and 16K/32K long context LLMs. The 4K cont
by Jimmc414 3y ago
The main findings are:
Retrieval-augmentation can significantly boost the performance of both 4K short context LLMs and 16K/32K long context LLMs.
The 4K context LLMs with simple retrieval-augmentation can perform comparable to 16K long context LLMs, while being more efficient at inference.
After context window extension and retrieval-augmentation, the best model LLaMA2-70B-32k-ret can outperform GPT-3.5-turbo-16k and Davinci003 in terms of average score on a suit of downstream tasks with informative queries.