3 ms·
There are “needle in the haystack” benchmarks for long context performance. It would be good to see those.
by d4rkp4ttern 1y ago
There are “needle in the haystack” benchmarks for long context performance. It would be good to see those.
- throwuxiytayq 1y agoThese aren’t really indicative of real world performance. Retrieving a single fact is pretty much the simplest possible task for a long context model. Real world use cases require considering many facts at the same time while ignoring others, all the while avoiding the overall performance degradation that current models seem susceptible to when the context is sufficiently full.
- d4rkp4ttern 1y agoI agree, retrieving a single fact is necessary but not sufficient.