Indeed. Maybe the learned circuit does something like the following. For each token's feature vector, compute a representation of positions where it appears in

Indeed. Maybe the learned circuit does something like the following. For each token's feature vector, compute a representation of positions where it appears in the sentence. This could be made possible by the positional embeddings. Token Features 0 => list[1, 5, 6, 10] Token Features 1 => list[7, 8] ... These "list features" would be invariant to Caesar cipher. So then the LLM could pass these list features to a learned Caesar cipher decoder unit to spit out the decoded text. It's still unexplained, however, why the Byzantine Music Notation would trigger the this circuit while other Caesar cipher's wouldn't.