Have you tried asking GPT-4? I got a pretty good explanation of attantion from it. The 'key' and 'value' description in the attention mechanism probably doesn'

Have you tried asking GPT-4? I got a pretty good explanation of attantion from it. The 'key' and 'value' description in the attention mechanism probably doesn't make much sense to you because at best it's a strained metaphor, rather than an actual principle by which attention operates. If you take a look at the maths, you can interchange the role of 'keys' and 'values' and get something mathematically identical.