I frequently see people surprised about the kinds of "simple" mistakes LLMs make when given tasks that involve letters, syllables, word lengths, rhyming, etc. T

I frequently see people surprised about the kinds of "simple" mistakes LLMs make when given tasks that involve letters, syllables, word lengths, rhyming, etc. They don't do well at character oriented tasks which seem trivial to us, like "write me a sentence with only 5 letter words". All of these problems stem from the fact that LLMs don't "see" the actual letters/characters in the text they are consuming and producing. They are only dealing in tokens, each of which usually blends together multiple letters. The fact that they can sometimes or partially succeed at character-oriented tasks is the actual surprise. They are presumably using meta-knowledge about specific words. For example, they may have learned by rote that "cat" has 3 letters. Or they just know as a fact that "moon" and "tune" rhyme, without having any sense of what it means to pronounce them.