2 ms·
Are there any solid benchmarks regarding increasing context length on a model, leading to decreased reliability? I have heard that this is the case, but is the
by consumer451 3y ago
Are there any solid benchmarks regarding increasing context length on a model, leading to decreased reliability?
I have heard that this is the case, but is there some data?
- unraveller 3y agohttps://github.com/TIGER-AI-Lab/LongICLBench https://github.com/TIGER-AI-Lab/LongICLBench is 1 month old, might as well be an eon in this space. There are other benchmarks coming out. https://evalplus.github.io/repoqa.html https://evalplus.github.io/repoqa.html https://news.ycombinator.com/item?id=40261442 https://news.ycombinator.com/item?id=40261442 There was also a post on HN that I can't find right now claiming to have solved the reliability problem in long context using a demo of a disney privacy policy page, deleting some and rearranging all document references perfectly after finding the needle in the haystack