4 ms·
I think the author is confusing the attention mechanism with attention in the general sense. He refers to a phenomenon called “attention drift”, but neither Goo
by ozgung 1y ago
I think the author is confusing the attention mechanism with attention in the general sense. He refers to a phenomenon called “attention drift”, but neither Google nor LLM searches return any relevant references to that term. In fact, ChatGPT only cited this same blog post.
- eric-burel 1y agoI think LLM vendors papers entertain such confusions, the lack of peer reviewing tends to favour grandiose but barely defined wording as such. At least author seems to have checked a few papers that do some empirical validation (which is still different from establishing any kind of theory of prompt engineering, there is no other way than cutting the model up to do that).