3 ms·
Section 4.3 addresses this, runs 3 benchmarks, tl;dr: 7B roundly beats LLamA 2 7B, almost same performance as LLamA 2 7B-L, which got an extra 500K of training
by refulgentis 3y ago
Section 4.3 addresses this, runs 3 benchmarks, tl;dr: 7B roundly beats LLamA 2 7B, almost same performance as LLamA 2 7B-L, which got an extra 500K of training tokens specifically at long context length.
* section 4.3, in both the paper you linked, and the paper we're commenting on. why did I notice that? it took me several minutes to understand "how the paper changed": it was 2 papers all along, I just switched tabs without realizing. And it's only Tuesday.
- cs702 3y agoThose benchmarks are about long context. The question isn't about long context. It's about recall, or the ability to fetch and repeat parts of the input context. There's nothing about recall on section 4.3.
- refulgentis 3y agoDo I have the wrong paper open again? :) See "Long-Context QA tasks in Scrolls", I don't want to copy and paste the whole thing, I'll elide the words in between: "...long-context open-book question answering (QA), we use a simple prompt {CONTEXT} Q: {QUESTION} A" n.b. It's literally the same eval in both papers :) I know, it's buried and non-obvious, it took me 20 minutes so far today to double check it 3x, even after reading it yesterday.