There are at least two reasons for transformers poor performance on that prompt: - Transformers view the word as tokens, not words or characters. - The positi

There are at least two reasons for transformers poor performance on that prompt: - Transformers view the word as tokens, not words or characters. - The positional encoding might be holding them back. See this recent paper discussed here: Transformers Can Do Arithmetic with the Right Embeddings [1] [1] https://news.ycombinator.com/item?id=40497379 https://news.ycombinator.com/item?id=40497379