4 ms·
This one does have attention, it's just chunked into segments of 4096
by euclaise 2y ago
This one does have attention, it's just chunked into segments of 4096
- cs702 2y agoYes, but the claim is about "unlimited context length." I doubt attention over each segment can be as good at recall as attention over the full input context.