The token embedding is what ultimately gets nudged around by the heads though, right? The key vector just relates to the context size, not the token embedding s

The token embedding is what ultimately gets nudged around by the heads though, right? The key vector just relates to the context size, not the token embedding size, afaik.