3 ms·
It remains to be seen just how effective longer contexts are because if the attention vectors don't ever learn to pick up specific items from further back in th
by woeirua 3y ago
It remains to be seen just how effective longer contexts are because if the attention vectors don't ever learn to pick up specific items from further back in the text then having more tokens doesn't really matter.
Given that the conventional cost of training attention layers grows quadratically with the number of tokens I think Anthropic is doing some kind of approximation here. Not clear at all that you would get the same results as vanilla attention.
- ttul 3y agoThey did mention that the inference time to answer a question about the book was something like 22 seconds, so perhaps they are indeed still using self-attention.