Yeah, a solution to the counting letters problem would probably be encoding the letters into the large embeddings by probably adding a really small loss and a s

Yeah, a solution to the counting letters problem would probably be encoding the letters into the large embeddings by probably adding a really small loss and a small classification model on the embeddings so that it would learn to encode the letters into the embeddings (but they probably don't care that much, they would rather scale the model).