4 ms·
Those benchmarks are about long context. The question isn't about long context. It's about recall, or the ability to fetch and repeat parts of the input context
by cs702 3y ago
Those benchmarks are about long context. The question isn't about long context. It's about recall, or the ability to fetch and repeat parts of the input context. There's nothing about recall on section 4.3.
- refulgentis 3y agoDo I have the wrong paper open again? :) See "Long-Context QA tasks in Scrolls", I don't want to copy and paste the whole thing, I'll elide the words in between: "...long-context open-book question answering (QA), we use a simple prompt {CONTEXT} Q: {QUESTION} A" n.b. It's literally the same eval in both papers :) I know, it's buried and non-obvious, it took me 20 minutes so far today to double check it 3x, even after reading it yesterday.