Just to follow on from what some others are saying, it may be because of tokens. These are the 'words' it sees in the poem: https://i.imgur.com/EzffHiZ.png htt

Just to follow on from what some others are saying, it may be because of tokens. These are the 'words' it sees in the poem: https://i.imgur.com/EzffHiZ.png https://i.imgur.com/EzffHiZ.png To be able to answer the question correctly, it essentially needs to memorize how long each of the tokens in its vocabulary are. One token seems to range from 1 character to 5 characters normally, but I'm sure some longer tokens exist, too. Judging by how often it fails at tasks like this, it seems likely that the model isn't aware and is just blindly guessing (as it always does).