3 ms·
The two issues I see with this work: * There's no mention of model performance on recall tasks. Models with attention do well on recall tasks, but models witho
by cs702 3y ago
The two issues I see with this work:
* There's no mention of model performance on recall tasks. Models with attention do well on recall tasks, but models without it, like this one, tend to do poorly.[a] What is the performance of this model on recall tasks?
* As others here have pointed out, the github link is dead. In addition, the pretrained weights don't seem to be available on HF or anywhere else. It's hard to check any claims if we have neither code nor weights!
---
[a] https://arxiv.org/abs/2402.01032 https://arxiv.org/abs/2402.01032
- phh 3y ago> * There's no mention of model performance on recall tasks. Models with attention do well on recall tasks, but models without it, like this one, tend to do poorly.[a] What is the performance of this model on recall tasks? For what it's worth, RWKV's (another "transformer-less") website on that matter mention that yes it's bad on recall, but for the vast majority of tasks you can just ask the question *before* the content, and it'll handle the task just fine. (I'm just reporting, I haven't spent time trying myself)
- refulgentis 3y agoSection 4.3 addresses this, runs 3 benchmarks, tl;dr: 7B roundly beats LLamA 2 7B, almost same performance as LLamA 2 7B-L, which got an extra 500K of training tokens specifically at long context length. * section 4.3, in both the paper you linked, and the paper we're commenting on. why did I notice that? it took me several minutes to understand "how the paper changed": it was 2 papers all along, I just switched tabs without realizing. And it's only Tuesday.
- cs702 3y agoThose benchmarks are about long context. The question isn't about long context. It's about recall, or the ability to fetch and repeat parts of the input context. There's nothing about recall on section 4.3.
- refulgentis 3y agoDo I have the wrong paper open again? :) See "Long-Context QA tasks in Scrolls", I don't want to copy and paste the whole thing, I'll elide the words in between: "...long-context open-book question answering (QA), we use a simple prompt {CONTEXT} Q: {QUESTION} A" n.b. It's literally the same eval in both papers :) I know, it's buried and non-obvious, it took me 20 minutes so far today to double check it 3x, even after reading it yesterday.
- euclaise 3y agoThis one does have attention, it's just chunked into segments of 4096
- cs702 3y agoYes, but the claim is about "unlimited context length." I doubt attention over each segment can be as good at recall as attention over the full input context.