Something that I’m intrigued by with tokenization is that there are obviously overlapping tokenizations for the same text - SolidGoldMagikarp can also be repres

Something that I’m intrigued by with tokenization is that there are obviously overlapping tokenizations for the same text - SolidGoldMagikarp can also be represented as Solid+Gold+Mag+ikarp or S+o+lid+Go+l+d+M+agi+Ka+r+p or a bunch of other representations. Now at training time, I guess these tokens were matched maximally - the greediest token was always chosen. So the LLM was trained on datasets where whenever SolidGoldMagikarp showed up, it used the full token. But when SolidGoldPikachu appears it gets tokenized as Solid+Gold+P+ik+achu. So when an LLM is predicting tokens and for some reason it decides it wants to suggest more things in the vein of FlappyOrangePikachu WetGreenCharmander FluffySilverSnorlax It seems like it’s going to output tokens much more hesitantly, gradually building a plausible adjective/color/Pokémon combination. If it actually did output SolidGoldMagikarp Token by token, doesn’t that mean it would miss any embedding that that full token has? It would only see it as a random adjective/color/Pokémon combination. Now maybe choosing a glitch token is a bad idea here because the problem with that token is that it lacks any further associations in the LLM model. But the same applies to like programming language keyword tokens. If it has a token for xmlHttpRequest doesn’t that mean the LLM might just throw together a variable name like that because the individual pieces make sense, without realizing ‘Oh hey! I know that word!’