4 ms·
Does it make sense to talk about the index of a token? The model never sees such an index does it? I thought the tokens are converted to learned embeddings at t
by markisus 2y ago
Does it make sense to talk about the index of a token? The model never sees such an index does it? I thought the tokens are converted to learned embeddings at the first layer.
- fi-le 2y agoThis is a great point, I think I might have been wrong actually. It doesn't really make sense that one row of the embedding matrix is treated differently than another...
- markisus 2y agoIndeed. Maybe the learned circuit does something like the following. For each token's feature vector, compute a representation of positions where it appears in the sentence. This could be made possible by the positional embeddings. Token Features 0 => list[1, 5, 6, 10] Token Features 1 => list[7, 8] ... These "list features" would be invariant to Caesar cipher. So then the LLM could pass these list features to a learned Caesar cipher decoder unit to spit out the decoded text. It's still unexplained, however, why the Byzantine Music Notation would trigger the this circuit while other Caesar cipher's wouldn't.